A Multi-Timescale Cooperative Scheduling Method for Flexible Load in Power Distribution System Considering Dynamic Transformer Rating

Abstract

With the large-scale integration of new energy, electric vehicles, and other new loads, disorderly electricity consumption has led to surging peak loads and heightened overload risks for distribution transformers. Particularly in aging, high-density urban areas constrained by the cost and space limitations of upgrading distribution facilities, there is an urgent need to tap into the flexible load control potential of existing power distribution systems to ensure system safety. This paper proposes a multi-timescale cooperative scheduling framework for flexible loads in distribution systems, deeply integrating the dynamic load capacity of transformers with the dispatchable characteristics of a flexible load. First, a day-ahead scheduling layer based on multi-agent reinforcement learning is constructed to optimize electricity plans and smooth peak–valley loads in the distribution system. Second, a dynamic transformer-rating model for distribution transformers is established to uncover their dynamic load capabilities under varying environmental conditions. Finally, an intraday scheduling layer for flexible loads is developed. It dynamically matches the regulation demands of distribution transformers and flexible loads via real-time optimization of consumption strategies to address electricity price fluctuations and user behavior randomness. Case study results demonstrate that the methods described in this paper effectively reduce power load fluctuations, ensuring the safe and stable operation of distribution and power supply systems.

1. Introduction

In recent years, the large-scale integration of distributed renewable energy and electric vehicles (EVs) has significantly amplified uncertainties on both the generation and demand sides of power systems [1,2,3]. This integration continuously widens the peak-to-valley load difference, posing severe challenges for the safe and stable operation of distribution networks [4,5,6,7,8,9]. The issue is particularly pronounced in aging, high-density urban areas constrained by limited physical space and prohibitive infrastructure-upgrading costs [10,11,12]. Traditional distribution systems struggle to manage the power fluctuations caused by these strong uncertainties, frequently resulting in transformer overloads. When distribution transformers operate under prolonged overload conditions, internal heat gradually accumulates, causing sudden hotspot temperature spikes. This thermal stress accelerates insulation aging, reduces equipment lifespan, and increases operational risks [13,14,15,16]. This challenge stems primarily from disorderly user behavior—such as uncoordinated EV charging overlapping with evening peak loads—and the inherent inability of land-scarce urban grids to rapidly adapt to new load characteristics.

To mitigate these risks without requiring massive capital investments, it is crucial to leverage the regulatory potential of flexible loads to smooth peak–valley fluctuations. Flexible loads, characterized by their responsive agility, serve as vital resources in modern power systems [17,18]. They provide value through active peak-shifting and avoidance mechanisms. As a highly scalable flexible resource, EVs represent a significant regulatory asset [19]. Because EV charging possesses inherent shiftability and transferability, intelligent management of EV clusters can redistribute charging demands across time and multiple adjacent distribution substations, effectively preventing localized transformer overloads [20,21,22,23,24,25,26,27,28].

While static mixed-integer programming and single-agent reinforcement learning have been employed to schedule EV charging, these approaches often struggle in modern grids [29,30,31,32,33]. Static optimization suffers from the curse of dimensionality when scaling to large EV fleets, and single-agent models fail to capture the highly competitive, non-stationary dynamics between independent charging stations vying for grid capacity [34,35]. In contrast, Multi-Agent Reinforcement Learning (MARL) provides a robust framework for handling distributed decision-making under uncertainty. Recent advancements demonstrate that MARL can effectively minimize charging costs while smoothing load profiles through Centralized Training with Decentralized Execution (CTDE) architectures [36,37]. However, a critical limitation in the current MARL literature is the reliance on static transformer capacity constraints. Treating transformer limits as rigid, worst-case seasonal boundaries forces conservative scheduling, leaving substantial operational headroom untapped. Dynamic Transformer Rating (DTR) leverages real-time ambient conditions to safely evaluate the true thermal capacity of transformers; yet, integrating DTR into model-free MARL introduces severe non-differentiable boundaries that can destabilize policy convergence.

To bridge this gap, this paper proposes a Multi-Agent Deep Deterministic Policy Gradient (MADDPG) scheduling framework intrinsically coupled with DTR constraints. The primary advantage of our approach lies in its decentralized execution phase, which eliminates communication bottlenecks and safely unlocks hidden transformer capacity without necessitating hardware upgrades. Conversely, the main trade-off is the high computational and memory overhead required during the initial centralized training phase.

Therefore, we propose a multi-timescale flexible load coordination strategy that actively embeds DTR evaluation into the multi-agent decision-making process. While our previous preliminary work [29] established the static distribution network environment and basic mathematical constraints for EV modeling, it lacked adaptive capacity mechanisms. The core theoretical novelty of this manuscript is its integration of dynamic thermal boundaries into a CTDE-based potential game, ensuring that local policy updates mathematically converge to a Nash Equilibrium that aligns with global grid safety. The main contributions are as follows:

(1)

We fully consider the characteristics of the power distribution system and electric vehicles in adjacent substations. We established physical models for EV charging status and charging station status and constructed a day-ahead scheduling layer based on multi-agent reinforcement learning to optimize power consumption plans and smooth peak–valley loads in the power distribution system.

(2)

We explored the dynamic load capacity of distribution transformers under varying environmental conditions and developed an evaluation model for the dynamic load capacity of distribution transformers.

(3)

We constructed an intraday scheduling layer for flexible loads. By dynamically optimizing electricity consumption strategies in real time to address price fluctuations and user behavior randomness, it is possible to achieve dynamic matching between distribution transformers and flexible load regulation demands.

The remainder of this paper is organized as follows. Section 2 introduces the day-ahead flexible load dispatch model governed by the MADDPG algorithm, outlining the observation, action, and reward structures. Section 3 details the intraday DTR integration, formulating the thermal and physical constraints acting as the real-time clipping boundary. Section 4 presents a comprehensive case study on a modified IEEE 33-bus system, verifying the algorithm against baseline methods through quantitative analysis. Finally, Section 5 concludes the study and discusses future implications.

2. Flexible Load Dispatch Model for a Distribution System in the Day-Ahead Stage

Flexible loads represented by electric vehicles involve multiple entities, each with distinct objectives. Users seek lower charging costs and reduced queue times, while charging stations aim to maintain stable load levels to prevent distribution transformer overloads. Furthermore, EV charging behavior is deeply coupled with electricity prices and charging stations. To effectively address diverse EV charging patterns and coordinate the objectives of EVs and charging stations, this paper constructs a day-ahead flexible load scheduling model for power distribution systems based on multi-agent reinforcement learning. As shown in the Figure 1, the reinforcement learning framework primarily includes the agent, reward, action, observation state, and environment.

In the figure above, Agent represents the decision-making entity, primarily referring to the electric vehicle agent and charging station agent in this paper. Action reflects the Agent’s behavior, representing the decision-making actions of electric vehicles and charging stations in different scenarios. Observation state denotes a subset of environmental information from the environment, serving as the basis for each entity’s decisions. Reward reflects environmental feedback, guiding agents to optimize and accumulate benefits. When scheduling strategies effectively reduce EV charging costs and smooth out peak–valley differences at charging stations, each Agent’s reward increases. Environment encompasses factors like charging station distribution. Through distributed architecture and collaborative learning mechanisms, multi-agent reinforcement learning can reflect the behavioral decisions of diverse agents like flexible loads. It characterizes the dynamic coupling relationships between factors such as electric vehicles and charging stations [16]. Consequently, it effectively enhances EV charging efficiency and reduces charging costs while mitigating load fluctuations in the distribution system. The relevant variables and their meanings in this paper are shown in the

Table 1. Corresponding meanings of model variables in this paper.

Variables	Description
N	Total number of accessible charging stations within the local distribution zone.
i, j	Indices for nodes in the distribution network.
t	Time step index.
r, x	Branch resistance and reactance, respectively.
${\mu }_1,$ ${\mu }_2$	Statistical mean times for morning (08:30) and evening (18:00) charging peaks.
δ	Grid loss weighting coefficient.
V	Node voltage magnitude.
I	Branch current flowing between adjacent nodes.
γ	Discount factor for the MARL algorithm.
$P_{i,j},$ $Q_{i,j}$	Active and reactive power flow from node $i$ to node $j$.

.

2.1. Observation State Space Model

Flexible loads represented by electric vehicles are deeply coupled with users’ daily habits, with peak activity concentrated between 8:00 AM and 10:00 AM and 5:00 PM and 8:00 PM. Therefore, this paper employs a double-hybrid normal distribution to model electric vehicle charging behavior, with the primary model shown below.

$$\begin{array}{c}{t}_a=f_a(t)=p_a{\displaystyle \frac{1}{\sigma \sqrt{2\pi }}}\exp \left(-{\displaystyle \frac{{\left(t-{\mu }_1\right)}^2}{2{\sigma }^2}}\right)\\ +\left(1-p_a\right){\displaystyle \frac{1}{\sigma \sqrt{2\pi }}}\exp \left(-{\displaystyle \frac{{\left(t-{\mu }_2\right)}^2}{2{\sigma }^2}}\right)\end{array}$$

(1)

where $t_a$ represents the exact arrival time of the EV at the charging station. The parameters ${\mu }_1$ and ${\mu }_2$ denote the statistical temporal peaks of user charging behavior; specifically, ${\mu }_1$ represents the morning commute arrival peak, while ${\mu }_2$ captures the evening return peak (centered near 18:00). The variable $\sigma$ denotes the standard deviation characterizing arrival stochasticity, and $\omega$ is the mixture weighting coefficient. The probability distribution model for the initial State of Charge (SOC) of electric vehicles upon entering a charging station can be formalized as follows:

$$f(t)={\displaystyle \frac{1}{\sigma \sqrt{2\pi }}}\exp \left(-{\displaystyle \frac{{(t-\mu )}^2}{2{\sigma }^2}}\right)$$

(2)

If the charging time of an electric vehicle at a charging station is T, then the time the electric vehicle leaves the charging station is $t_b=t_a+T$. Therefore, the Observation State Space Model [29] in this paper can be formalized as follows:

$$\left\{\begin{array}{l}{S}_i^t=\left\{o_i^t,O_i^t\right\}\hfill \\ o_i^t=\left\{{\mathrm{SOC}}_i^t,p_i^t,t_a,t_b,g_i^c,d_i,q_i,f_i^c,{\omega }_c^t\right\}\hfill \\ O_i^t=\left\{P_{PVi}^t,{\omega }_c^t,P_{i,j}^t,Q_{i,j}^t,V_i^t\right\}\hfill \end{array}\right.$$

(3)

where $o_i^t$ is the electric vehicle observation space, $O_i^t$ is the observation space of the charging station, $g_i^c$ is the charging station number, primarily used for scheduling decisions, $d_i$ represents the distance between electric vehicles and charging stations, $q_i$ represents the current queue length at the charging station, $f_i^c$ indicates whether the electric vehicle is currently charging, and ${\omega }_c^t$ represents the current price of electricity. To ensure safe and stable operation of the charging station, the node voltage constraints are set as follows:

$$V_{\min }\le V_i^t\le V_{\max } \forall i\in L$$

(4)

where $V_{\min }$ and $V_{\max }$ represent the maximum and minimum values of the node voltage.

2.2. Action Space Model

To reflect the complex coupling relationship between electric vehicles and charging stations, this paper employs a Markov game tuple $\{N, S, A, P, R, \gamma \}$ to characterize the behavioral decisions of the agents. $S$ is the joint state space of agents. $A$ is the coordinated action of intelligent agents. $P$ is the state transition probability. $R$ is the cumulative global reward. $\gamma$ represents the discount factor. When the i-th electric vehicle initiates a charging request, its mathematical expression and constraints are as follows:

$$A_i^t=\left\{p_i^t\right\}i=1,2,\cdots ,N$$

(5)

$$-p_{\max }\le {\displaystyle \sum _{i=1}^N{\eta }_i}\cdot p_i^t\le p_{\max }$$

(6)

$$-p_{2\max }\le p_i^t\le p_{2\max }$$

(7)

$${\eta }_i=\left\{\begin{array}{ll}{\eta }_c\hfill & p_i^t\ge 0\hfill \\ 1/{\eta }_d\hfill & p_i^t<0\hfill \end{array}\right.$$

(8)

where $P_{\max }$ is the maximum power of the charging station, $P_{2\max }$ is the maximum power of the charging station’s charging pile, ${\eta }_i$ is the power exchange efficiency, ${\eta }_c$ is charging efficiency, ${\eta }_d$ is the discharge efficiency, and $p_i^t$ is the charging and discharging power of the i-th electric vehicle at time t. When $p_i^t>0$, it indicates that the electric vehicle is in G2V mode. When $p_i^t<0$, it indicates that the electric vehicle is in V2G mode.

2.3. State Transition Model

When an electric vehicle begins charging or discharging, the change in battery capacity can be formalized as the following equation:

$$SO{C}_i^{t+1}=SO{C}_i^t+{\displaystyle \frac{{\eta }_i\cdot p_i^t\cdot \Delta t}{C_i}}$$

(9)

where $C_i$ represents the battery capacity of an electric vehicle. To ensure that the voltage and current at each node of the power grid comply with physical constraints during the charging and discharging process of electric vehicles, this paper sets the following constraints:

$$\left\{\begin{array}{l}{P}_{ij}^t=P_{c,j}^t-P_{PV,j}^t+{\displaystyle \sum _{jk\in L}{P}_{jk}^t}+r_{ij}{\left(I_{ij}^t\right)}^2\hfill \\ Q_{ij}^t=Q_{c,j}^t+{\displaystyle \sum _{jk\in L}{Q}_{jk}^t}+x_{ij}{\left(I_{ij}^t\right)}^2\hfill \\ {\left(I_{ij}^t{V}_i^t\right)}^2={\left(P_{ij}^t\right)}^2+{\left(Q_{ij}^t\right)}^2\hfill \\ {\left(V_j^t\right)}^2={\left(V_i^t\right)}^2-2\left(r_{ij}{P}_{ij}^t+x_{ij}{Q}_{ij}^t\right)+\left(r_{ij}^2+x_{ij}^2\right){\left(I_{ij}^t\right)}^2\hfill \end{array}\right.$$

(10)

where $P_{ij}^t$ is the active power from node i to node j, $Q_{ij}^t$ is the reactive power from node i to node j, $P_{c,j}^t$ is the active power demand of node j at time t, and $Q_{c,j}^t$ is the reactive power demand of node j at time t.

2.4. Flexible Load Scheduling Model Based on Multi-Agent Reinforcement Learning

To reflect the dynamic game-theoretic process between electric vehicles and charging stations, this paper defines the joint strategy space of the game model as $\left\{{\pi }_{EV},{\pi }_{CS}\right\}$. Through continuous iterative training, the model gradually reaches a Nash equilibrium state. At this point, the system state can be formalized as the following equation:

$$\left\{\begin{array}{l}{R}_{EV}\left({\pi }_{EV}^{\ast },{\pi }_{CS}^{\ast }\right)\ge R_{EV}\left({\pi }_{EV},{\pi }_{CS}^{\ast }\right)\hfill \\ R_{CS}\left({\pi }_{EV}^{\ast },{\pi }_{CS}^{\ast }\right)\ge R_{CS}\left({\pi }_{EV}^{\ast },{\pi }_{CS}\right)\hfill \end{array}\right.$$

(11)

where ${\pi }_{EV}$ is an electric vehicle charging strategy, ${\pi }_{CS}$ is the charging station strategy, and ${\pi }_{EV}^{\ast }$ and ${\pi }_{CS}^{\ast }$ represent achieving optimal strategy. To ensure the convergence of the proposed algorithm, which involves multi-objective games among multiple agents, this paper unifies the optimization objectives of electric vehicles and charging stations. A global potential function is constructed, and individual rewards are adjusted through weight coefficients, as shown in the following equation:

$$\left\{\begin{array}{l}{R}_{EV}\left({\pi }_{EV}^{\prime },{\pi }_{CS}\right)-R_{EV}\left({\pi }_{EV},{\pi }_{CS}\right)\hfill \\ =\Phi \left({\pi }_{EV}^{\prime },{\pi }_{CS}\right)-\Phi \left({\pi }_{EV},{\pi }_{CS}\right)\hfill \\ R_{CS}\left({\pi }_{EV},{\pi }_{CS}^{\prime }\right)-R_{CS}\left({\pi }_{EV},{\pi }_{CS}\right)\hfill \\ =\Phi \left({\pi }_{EV},{\pi }_{CS}^{\prime }\right)-\Phi \left({\pi }_{EV},{\pi }_{CS}\right)\hfill \end{array}\right.$$

(12)

$$\Phi \left({\pi }_{CS},{\pi }_{EV}\right)=c{R}_{CS}\left({\pi }_{CS},{\pi }_{EV}\right)+(1-c)R_{EV}\left({\pi }_{CS},{\pi }_{EV}\right)$$

(13)

where $c$ is the weighting coefficient. At this point, when the entire model reaches the Nash equilibrium, the potential function must also be maximized. The primary formula is as follows:

$$\Phi \left({\pi }_{EV}^{\ast },{\pi }_{CS}^{\ast }\right)\ge \Phi \left({\pi }_{EV},{\pi }_{CS}\right)\forall \left({\pi }_{EV},{\pi }_{CS}\right)$$

(14)

The maximum point of the potential function corresponds to the Pareto optimal solution, and the Nash equilibrium point naturally becomes the Pareto optimal strategy combination. To achieve strategy convergence, this paper employs a multi-agent strategy gradient method. Through repeated iterations, model parameters are updated using gradient ascent. The primary mathematical expressions are as follows:

$$\left\{\begin{array}{l}{\pi }_{EV}^{k+1}={\pi }_{EV}^k+\alpha \nabla {\pi }_{EV}{R}_{EV}\left({\pi }_{EV}^k,{\pi }_{CS}^k\right)\hfill \\ {\pi }_{CS}^{k+1}={\pi }_{CS}^k+\alpha \nabla {\pi }_{CS}{R}_{CS}\left({\pi }_{EV}^k,{\pi }_{CS}^k\right)\hfill \end{array}\right.$$

(15)

During peak grid load periods, the charging power of electric vehicles will be moderately reduced. If the target SOC cannot be achieved within the specified timeframe, user demand cannot be met. Therefore, this paper introduces a secondary penalty for such scenarios, expressed primarily as follows:

$$r_1=\left\{\begin{array}{ll}-{\left(\left(0.8-SO{C}_i^t\right)\cdot {\displaystyle \frac{a}{t_i^c+ϵ}}\right)}^2\hfill & SO{C}_i^t<0.8\hfill \\ a{\left(0.8-SO{C}_i^t\right)}^2\hfill & SO{C}_i^t\ge 0.8\hfill \end{array}\right.$$

(16)

where $a$ is the weight adjustment coefficient for SOC, $ϵ$ is the smoothing coefficient, and $t_i^c$ is the remaining driving time for the electric vehicle.

To guide electric vehicles and charging stations toward achieving an optimal equilibrium state, this paper introduces charging cost and unauthorized discharge penalty functions, distance and queueing penalty functions, line loss penalty functions, and load balancing penalty functions. The charging cost and unauthorized discharge penalty function $r_2$ encourages EVs to charge during off-peak hours, while the distance and queueing penalty function $r_3$ encourages EVs to select nearby, idle charging stations. The line loss penalty function $r_4$ reduces grid energy waste, and the load-balancing penalty function $r_5$ suppresses load fluctuations to smooth grid curves. Detailed expressions [29] for these penalty functions are as follows:

$$r_2=-p_i^t\cdot {\omega }_c^t-\mu \cdot \mathbb{I}(p_i^t<0\&SO{C}_i^t<0.8)$$

(17)

$$r_3=-\xi d_i-\delta q_i$$

(18)

$$r_4=-b{\displaystyle \sum _{ij\in L}{r}_{ij}}{\left(I_{ij}^t\right)}^2$$

(19)

$$r_5=-{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N{\left(P_i-\overline{P}\right)}^2}$$

(20)

where $\mu$ serves as a positive scaling coefficient and $\mathbb{I}$ acts as the indicator function. $\xi$ is the distance weighting coefficient, $\delta$ is the queue weight coefficient, $b$ is the grid loss weighting coefficient, and $\overline{P}$ is the average active load. The design of Equation (17) specifically targets grid economic efficiency: it mathematically disincentivizes charging behaviors during peak periods by explicitly penalizing high charging costs. Furthermore, it imposes a severe negative reward through the indicator function whenever an agent attempts an unauthorized Grid-to-Vehicle (G2V) or Vehicle-to-Grid (V2G) power transition that violates the operational constraints, thus strictly enforcing temporal consistency in the charging schedule.

In multi-agent reinforcement learning, each agent must interact not only with the environment but also with other agents. During the training stage, existing research primarily employs the Centralized Training with Decentralized Execution (CTDE) framework. During training, agents optimize based on global information, while during execution stage, they make independent decisions using local information. The multi-agent reinforcement learning equations involving value function decomposition and policy gradients in CTDE are as follows:

$$Q_{tot}(s,a)={\displaystyle \sum _{i=1}^n{\omega }_i}(s)Q_i\left(s,a_i\right)$$

(21)

$$\nabla J\left({\theta }_i\right)=\mathbb{E}\left[{\nabla }_{{\theta }_i}{\pi }_i\left(a_i\mid o_i\right){\nabla }_a{Q}_i\left(S,a_1,\dots ,a_n\right)\right]$$

(22)

where $Q_{tot}(s,a)$ is the global action-value function, $Q_i\left(s,a_i\right)$ is the local Q-value, ${\omega }_i(s)$ is the state-dependent weight, $\nabla J\left({\theta }_i\right)$ is the gradient of the strategy parameters, and ${\nabla }_a{Q}_i$ is the action gradient.

To enhance the training efficiency of multi-agent reinforcement learning, this paper groups charging stations, with each station accommodating a certain number of electric vehicles. Electric vehicles within the same group only need to communicate with their corresponding charging station [29]. At each time step, the charging station collects information from all electric vehicles. This relationship can be formalized as follows:

$${\xi }_k^t={\displaystyle \frac{1}{\left|g_k\right|}}{\displaystyle \sum _{i\in g_k}{o}_i^t}$$

(23)

The charging station generates decisions based on aggregated information ${\xi }_k^t$. The policy update process employs policy gradient methods combined with a Gaussian distribution to address continuous action problems, which can be formalized as follows:

$${\nabla }_{{\varphi }_k}J\left({\varphi }_k\right)=\mathbb{E}\left[{\nabla }_{{\varphi }_k}\log {\pi }_k\left(a_{g_k,t}\mid {\xi }_k^t\right)A_{g_k}\left({\xi }_k^t,a_{g_k,t}\right)\right]$$

(24)

where $A_{gk}$ is the advantage function. During training, multiple agents undergo group-based training. Each charging station updates its model parameters based on group-specific information and feedback. This update process can be formalized as follows:

$${\mathcal{L}}_{TD}^{g_k}={\left(R_{gk,t}+\gamma \max Q_{{\theta }_i^{\prime }}^{g_k}\left({\xi }_k^{t+1},a_{g_k,t+1}\right)-Q_{{\theta }_i^{\prime }}^{g_k}\left({\xi }_k^t,a_{g_k,t}\right)\right)}^2$$

(25)

To reflect the game-theoretic relationships among different groups and ensure that each group’s local optimization direction aligns with the overall global objective, this paper employs a fusion evaluation method for the behavioral decision outcomes of different groups. This fusion evaluation method can be formalized as the following equation:

$$\left\{\begin{array}{c}{y}^t=R^t+\gamma \max {\overline{Q}}_{\varphi }^{tot}\left(S,a_{g1,t},\dots ,a_{g_k,t}\right)\\ L_{TD}(\varphi )=\mathbb{E}\left[{\left(y^t-{\overline{Q}}_{\varphi }^{tot}\left(S,a_{g1,t},\dots ,a_{g_k,t}\right)\right)}^2\right]\end{array}\right.$$

(26)

To achieve simultaneous global and local optimization of model parameters during multiple iterative cycles, this paper constructs consistency constraints for guidance. The loss function is as follows:

$${\mathcal{L}}^{(\mathrm{Cl})}(\varphi ,\omega )=\mathbb{E}\left[\max {\left\{{\overline{Q}}_{\varphi }^{\mathrm{tot} }\left(S,a_{g1,t},\dots ,a_{g_k,t}\right)-Q_{\omega }^{\mathrm{tot} },0\right\}}^2\right]$$

(27)

$${\mathcal{L}}^{(\mathrm{C}2)}(\varphi ,\omega )=\mathbb{E}\left[{\left({\max }_{a^{\ast }}{\overline{Q}}_{\varphi }^{tot}\left(S,a_{g1,t},\dots ,a_{g_k,t}\right)-{\max }_{a^{\ast }}{Q}_{\omega }^{tot}\right)}^2\right]$$

(28)

where $Q_{\omega }^{tot}=T_{\omega }\left(Q_{{\theta }_i^{\prime }}^{g_1},\dots ,Q_{{\theta }_i^{\prime }}^{g_k}\right)$, ${\overline{Q}}_{\varphi }^{\mathrm{tot}}$ denotes the global evaluation result, and $Q_{\omega }^{\mathrm{tot} }$ denotes the results of the local evaluation.

This paper constructs a day-ahead flexible load scheduling model for power distribution systems based on multi-agent reinforcement learning algorithms. By grouping charging stations and electric vehicles and establishing a potential game model, it coordinates global and local optimization directions to achieve optimal electric vehicle scheduling outcomes during the day-ahead stage, laying the groundwork for subsequent stages.

3. Flexible Load Scheduling Model for Distribution System Considering Dynamic Transformer Rating During Intraday Stages

3.1. Multi-Timescale Coupling and Receding Horizon Optimization

The integration of the multi-agent system with the transformer evaluation model forms a dual-layer architecture. The day-ahead MARL layer generates an economically optimal, global baseline trajectory. However, relying purely on this trajectory is risky due to stochastic forecasting errors in renewable output and user arrivals. To mitigate this, the intraday layer operates as a Receding Horizon Optimization (RHO) mechanism, akin to Model Predictive Control. It continuously monitors the real-time DTR, derived dynamically from actual ambient temperatures, treating it as an absolute hard boundary. If the aggregate power demand from the day-ahead schedule threatens to exceed the real-time DTR limit, the intraday algorithm executes a priority-based clipping function. This strictly truncates the instantaneous EV charging power, sacrificing marginal economic optimality to unconditionally guarantee the physical integrity of the distribution transformer.

3.2. Dynamic Transformer-Rating Evaluation Model

In aging, high-density urban areas, constrained by the cost and space limitations of upgrading distribution facilities, timely upgrades to power distribution and consumption systems are challenging [23,24,25]. There is an urgent need to tap into the potential for flexible load control within existing power distribution and consumption systems. The load capacity of distribution transformers is closely tied to their operating environment. Relying solely on rated capacity as the primary reference standard while ignoring the dynamic variations in a distribution transformer’s load capacity prevents the full utilization of the existing power distribution and consumption system’s equipment potential. Therefore, evaluating the Dynamic Transformer Rating (DTR) under varying conditions and optimizing EV charging station scheduling based on DTR constraints are crucial for reducing peak-to-valley differences in distribution transformers and ensuring the safe and stable operation of power distribution systems.

Numerous factors influence DTR fluctuations, among which the hotspot temperature of distribution transformers is the key limiting factor for DTR. This paper evaluates DTR using the hottest spot temperature (HST) calculation method recommended in IEEE std. C57.91-2011 [30]. The hotspot temperature is derived from factors such as ambient temperature and top-layer oil temperature. The formula for calculating top-layer oil temperature is as follows:

$${\theta }_{TO}={\theta }_A+\Delta {\theta }_{TO}$$

(29)

where ${\theta }_{TO}$ is the top-oil temperature of the distribution transformer, ${\theta }_A$ is the ambient temperature, and $\Delta {\theta }_{TO}$ is the temperature rise of the top-oil temperature relative to the ambient temperature. The specific expression for $\Delta {\theta }_{TO}$ is as follows [30]:

$$\Delta {\theta }_{TO}=\left(\Delta {\theta }_{TO,U}-\Delta {\theta }_{TO,i}\right)\left(1-e^{{\displaystyle \frac{24}{{\tau }_{TO}}}}\right)+\Delta {\theta }_{TO,i}$$

(30)

where $\Delta {\theta }_{TO,i}$ is the initial temperature at the current time, ${\tau }_{TO}$ is the oil time constant, and $\Delta {\theta }_{TO,U}$ is steady-state oil temperature. $\Delta {\theta }_{TO,U}$ is expressed as follows:

$$\Delta {\theta }_{TO,U}=\Delta {\theta }_{TO,R}{\left[{\displaystyle \frac{\left(L_U^{2}R+1\right)}{(R+1)}}\right]}^n$$

(31)

where $n$ is the cooling coefficient, which primarily depends on the transformer’s cooling method, and $L_U$ is the transformer load factor. The parameters for $R$ are provided by the equipment manufacturer. Therefore, the HST calculation equation for distribution transformers is as follows:

$${\theta }_H={\theta }_A+\Delta {\theta }_{TO}+\Delta {\theta }_H$$

(32)

$$\Delta {\theta }_H=\left(\Delta {\theta }_{H,U}-\Delta {\theta }_{H,i}\right)\left(1-e^{{\displaystyle \frac{24}{{\tau }_W}}}\right)+\Delta {\theta }_{H,i}$$

(33)

where $\Delta {\theta }_{H,U}=\Delta {\theta }_{H,R}{K}_u^{2m}$, ${\theta }_H$ is HST, $\Delta {\theta }_H$ is the temperature rise of the top-oil temperature for HST, and $\Delta {\theta }_{H,i}$ is the HST at the initial moment.

When HST is excessively high, it accelerates the aging rate of insulating paper in distribution transformers. When distribution transformers operate under prolonged overload, the insulating paper exhibits reduced mechanical properties and deteriorated insulation performance. This paper employs Degree of Polymerization (DP) to assess aging in distribution transformers. Newly manufactured transformers exhibit an initial DP value of $D{P}_0=1000$. After prolonged operation, when $D{P}_f=200$, the insulation paper’s performance significantly deteriorates [31]. At this stage, the insulation paper can no longer guarantee the safe and stable operation of the distribution transformer. Therefore, when tapping into the operational potential of distribution transformers, one should avoid incurring additional life loss. Consequently, this paper constructs the following model to reflect the aging condition of transformers:

$$T_W={\left[{\displaystyle \frac{-R}{E_a}}\cdot \ln {\displaystyle \frac{{\displaystyle \sum e^{-E_a/R{T}_i}}}{n}}\right]}^{-1}$$

(34)

$$t_{life}={\displaystyle \frac{\frac{1}{D{P}_f}-\frac{1}{D{P}_0}}{A\times 24\times 365}}\cdot e^{E_a/R{T}_i}$$

(35)

where $R$ is the molar gas constant, $T_W$ is the equivalent aging temperature, $E_a$ is the empirically derived activation energy, and $A$ is an empirically derived environmental factor. Elevated HST not only accelerates thermal aging of distribution transformer insulation paper but also increases the volume of oil within the transformer. This causes the oil level to rise, compressing the gas in the upper chamber and elevating internal pressure. To prevent issues such as tank bulging and weld seam cracking, this paper establishes the following model to evaluate and limit the internal pressure of distribution transformers [31].

$$P_{int}=0.145\cdot P_0\left({\displaystyle \frac{{\theta }_H}{{\theta }_A}}\right)\left({\displaystyle \frac{V_{l0}}{V_l}}\right)\left[{\displaystyle \frac{S{F}_0+\left(\frac{V_{g0}}{V_{l0}}\right)}{S{F}_i+\left(\frac{V_g}{V_l}\right)}}\right]$$

(36)

where $P_{\mathrm{int}}$ is the final pressure inside the distribution transformer, $P_0$ is the initial environmental pressure, $V_{l0}$ is the oil volume at initial oil temperature, $V_l$ is the oil volume at the current oil temperature, $V_{g0}$ is the initial gas volume, $V_g$ is the current gas volume, $S{F}_i$ is the solubility factor at the final temperature, and $S{F}_0$ is the solubility factor at ambient temperature. Therefore, the DTR assessment methods under different environmental conditions are as follows:

$$K_{TD}=K_{TD(X)}+{\displaystyle {\displaystyle \sum _{n=1}^N}{K}_{t(n,t)}}+{\displaystyle {\displaystyle \sum _{n=1}^N}{K}_{EV(t(n,Y))}^{\prime }}+K_{B(T)}$$

(37)

where $K_{B(T)}$ is the base load, $K_{TD}$ is the DTR, ${\displaystyle \sum _{n=1}^N{K}_{t(n,t)}}$ is the electric vehicle currently being charged, and ${\displaystyle \sum _{n=1}^N{K}_{EV(t(n,Y))}^{\prime }}$ is an electric vehicle that will be charging within the next few hours. During the intraday stage, this paper employs DTR as the constraint for distribution transformer capacity. At this point, the assessment of whether a distribution transformer is overloaded primarily depends on HST, pressure indicators, and thermal aging constraints. If, after superimposing new electric vehicle charging loads, the distribution transformer exhibits excessively high HST or pressure exceeding limits, the scheduling model proposed in this paper will dynamically adjust the charging power of that electric vehicle based on the load curve.

4. Example Analysis

4.1. Test System and Simulation Environment

To validate the effectiveness of the proposed multi-timescale scheduling framework and evaluate its impact on distribution systems, this paper constructed an enhanced IEEE 33-bus active distribution network model. The system incorporates three photovoltaic (PV) power stations connected at nodes 11, 18, and 33, alongside six fast-charging stations located at nodes 8, 15, 21, 25, 28, and 31. The flowchart is shown in the Figure 2.

The day-ahead scheduling layer executes offline centralized training under the CTDE architecture. During this phase, electric vehicle agents and charging station agents interact with the environment, observing local states including arrival time, initial state of charge, queue length, and real-time electricity price. The MADDPG algorithm trains centralized critics and decentralized actors, yielding a baseline charging trajectory that minimizes peak–valley differentials and charging costs. In the intraday stage, the dynamic transformer rating is evaluated continuously for the monitored transformer using real-time ambient conditions. The receding horizon optimization module then compares the aggregated charging load at the monitored transformer against the real-time DTR limit. If the load threatens to exceed the thermal boundary, a priority-based clipping function is triggered to curtail instantaneous EV charging power. The topology structure of the power system is shown in the following figure.

The CTDE framework was operationalized using the MADDPG algorithm. To ensure stable convergence, the hyperparameters were strictly configured. The actor and critic networks feature three hidden layers using ReLU activation. The primary training hyperparameters, which govern the neural network updates are summarized in

Table 2. MADDPG Training Hyperparameters.

Parameter	Value	Description
Actor Learning Rate	1 × 10−4	Controls the step size for updating the actor policy network
Critic Learning Rate	1 × 10−3	Controls the step size for updating the critic value network
Discount Factor	0.95	Determines the importance of future rewards
Target Soft Update Rate	0.01	Smoothing coefficient for updating target networks
Batch Size	1024	Number of experiences sampled per training iteration
Replay Buffer Capacity	106	Maximum number of stored transitions

.

This paper initially focuses on the distribution transformer at Node 28 as the primary research subject due to its high historical load density. Using the assessment methods detailed in Section 3, the daily maximum Dynamic Transformer Rating (DTR) relative to the rated capacity (800 kVA) over a standard week was evaluated, as illustrated in Figure 3.

As shown in Figure 4, the maximum DTR within a week is generally concentrated around 1.2 times the rated capacity. The dynamic load capacity frequently exceeds the conservative rated capacity, particularly during periods of lower ambient temperature, allowing for additional EV integration without incurring equipment damage.

A core claim of integrating DTR is the safe unlocking of operational capacity without exacerbating equipment degradation. Under an uncoordinated charging scenario, severe peak load overlaps drove the distribution transformer’s hotspot temperature (HST) at high temperature. Based on the aging model, this resulted in an equivalent aging acceleration factor greater than 1. Consequently, the daily equivalent aging acceleration factor was maintained below 1.0, yielding a negligible insulation deterioration. This confirms that the framework successfully unlocked 18% to 23% of hidden transformer capacity while incurring zero additional lifespan loss.

4.2. Baseline Algorithm Comparison

To quantify the theoretical superiority of the proposed framework, it was benchmarked against two standard baselines: a centralized Rule-Based Optimization algorithm and a standard single-agent Deep Q-Network (DQN).

The above figure illustrates the reward convergence curves during the training phase. The proposed MADDPG model converges to a higher stable reward threshold in fewer episodes compared to the DQN baseline. The single-agent DQN struggles with the non-stationary environment caused by independent EVs competing for charging capacity, leading to significant oscillations.

Regarding physical grid impact, comparing the uncoordinated charging pattern with the optimized schedule highlights the algorithm’s peak-shaving capability. Figure 5 displays the maximum and minimum loads of the original system versus the system after implementing the MADDPG-coordinated scheduling.

As depicted in Figure 6, the proposed method reduces the peak load by 5.2% compared to the uncoordinated baseline. The original maximum peak valley difference was 0.67, and after optimization using the method proposed in this paper, the peak valley difference was reduced to 0.41.

4.3. Sensitivity Analysis of Reward Coefficients

The stability of the MARL algorithm is highly dependent on the weighting coefficients configured in the potential game reward function. A sensitivity analysis was conducted on the distance penalty coefficient and the grid loss weighting coefficient. The impact of grid loss weighting coefficient varying on system performance is detailed in

Table 3. Impact of Grid Loss Coefficient on System Performance.

Variation of Grid Loss Weighting Coefficient	Station Utilization Rate (%)	Relative Charging Cost Index
−15%	88.5	105
−5%	87.2	102
Base (0%)	85.0	100
+5%	80.5	98
+15%	73.0	92

.

As shown in Table 3, the system is robust to minor variations. However, it exhibits high sensitivity when grid loss weighting coefficient is substantially altered. Artificially inflating the grid loss coefficient by 15% resulted in overly conservative agent behaviors; EVs opted to defer charging entirely rather than incur marginal line losses, dropping the station utilization rate by 12% and leading to unmet user charging demands.

To rigorously evaluate whether the performance improvements stem from the architectural innovations, we conducted systematic ablation studies. The results of the ablation tests are shown in the table below.

Transformer Utilization is defined as the ratio of the actual carried load to the rated capacity, where values exceeding 100% indicate successful exploitation of dynamic thermal headroom enabled by DTR. The full framework achieves 118.5% utilization with zero safety violations. This demonstrates that the 18.5% capacity augmentation is realized without compromising equipment integrity. Conversely, removing the DTR layer constrains utilization to 98.2%. Without DTR, the system cannot implement preemptive power clipping during rapid load aggregations, causing transient HST spikes that exceed safe thresholds. Ablating the CTDE structure yields 112.4% utilization but 11 violations. Independent learners, lacking centralized coordination during training, exhibit transient non-stationarity where agents simultaneously select high-power actions.

4.4. Scalability Validation

As EV adoption accelerates, scheduling algorithms must scale efficiently. The computational scalability of the proposed decentralized execution architecture was tested against a standard Centralized Optimization solver by varying the EV penetration rate from 20% to 50%, as recorded in

Table 4. Performance Comparison of Component Ablation Configurations.

Configuration	Transformer Utilization (%)	Safety Violations
Full Framework	118.5	0
w/o DTR Layer (Static Rating)	98.2	7
w/o CTDE (Independent Learners)	112.4	11
w/o Intraday Layer	115.3	4

and

Table 5. Computational Scalability Across EV Penetration Rates.

EV Penetration Rate (%)	Centralized Optimization Execution Time (s)	Proposed MADDPG Execution Time (s)
20%	12.3	0.5
30%	35.8	0.8
40%	128.3	1.2
50%	351.0	1.5

.

As demonstrated in Table 4, the centralized method experiences exponential growth in execution time. In contrast, the CTDE architecture distributes the computational burden locally to the charging stations. The execution time for the MADDPG model scales linearly, maintaining an average local decision latency of under 2.0 s, confirming its suitability for large-scale real-world deployment.

5. Conclusions

This paper addresses the severe capacity constraints and overload risks threatening distribution networks in aging urban areas due to uncoordinated EV charging. By shifting away from conservative static ratings and centralized computational bottlenecks, we propose a multi-timescale scheduling framework coupling MARL with Dynamic Transformer Rating. The decentralized execution of the MADDPG algorithm successfully bypasses the curse of dimensionality. As quantified in Section 4, this framework reduces the system peak-to-valley load difference from 0.67 to 0.41 while lowering peak loads by 5.2% compared to uncoordinated baselines. The integration of intraday DTR constraints acts as a rigid physical safeguard.

Future research will focus on transitioning the current simulation environment into a hardware-in-the-loop (HIL) testbed, explicitly incorporating the degradation models of EV lithium-ion batteries into the multi-agent reward structure to facilitate deep Vehicle-to-Grid (V2G) integration.