Flexibility Resource Services and Electricity Cost Optimization Oriented Control Strategy of Data Centers Based on Hierarchical Reinforcement Learning

Abstract

As the core of digital infrastructure, the exceptionally rapid development of data centers (DCs) faces serious challenges due to their high electricity costs. Traditional approaches treat computational task scheduling separately from different physical control mechanisms, such as server group management, overlooking the synergistic potential between the two aspects. To address this problem, this paper proposes a computational–physical collaborative optimization model that realizes spatiotemporal task migration on the computational side and adaptive parameter regulation of IT equipment and cooling devices on the physical side. In response to the lack of global coordination in conventional distributed optimization, a two-layer partially observable Markov game (POMG) is constructed to unify global cooperative decision-making and local autonomous control. On this basis, the hierarchical multi-agent deep deterministic policy gradient (H-MADDPG) algorithm is designed by introducing task priority ranking and a variable-dimension action mask mechanism, which effectively handles the discrete–continuous hybrid action space and adapts to the dynamic variation in action dimensions caused by uncertain task arrivals. Comparative experiments with various benchmark schemes are conducted to verify the effectiveness and superiority of the proposed strategy in total cost, power usage effectiveness (PUE), resource utilization, and load balancing.

1. Introduction

1.1. Background and Motivation

With the vigorous development of the digital economy, the application of emerging technologies, such as cloud computing and artificial intelligence (AI), has been expanding continuously. As the physical carrier supporting their operation, DCs have become major energy consumers [1,2]. The power consumption of DCs accounted for approximately 1.5% of the global total power consumption in 2024 [3], reaching 415 terawatt-hours (TWh). By 2030, the power consumption of DCs is projected to more than double [4]. On the one hand, DCs can adjust their power consumption according to real-time electricity price signals to reduce the operating cost. On the other hand, DCs can also act as important flexible resources to participate in demand response programs, helping the power grid maintain safe and stable operation. Therefore, how to regulate the energy load of DCs, unlock their flexibility potential, and improve energy efficiency has become a key concern for DC operators.

1.2. Literature Review

Unlike traditional loads that only have temporal regulation potential [5,6], DCs are characterized by the core attribute of spatiotemporal adjustability [7,8]. By exploiting spatiotemporal electricity price disparities, computational tasks can be migrated among DCs or rescheduled within a single DC, thus cutting energy costs [9]. Ref. [10] established a two-layer model to explore workload migration strategies for urban DCs and discussed the impact of migration costs on the total system cost. Ref. [11] explored the spatiotemporal adjustability potential of DCs in peer-to-peer (P2P) trading. Refs. [12,13] mathematically characterized the temporal flexibility of batch-processing loads through the transferable power matrix and data-driven methods, respectively. Refs. [14,15] proposed a low-carbon-oriented spatiotemporal task migration mechanism. Ref. [16] formulates the bidding strategy for DC aggregators participating in the electricity market.

Moreover, a DC internally comprises IT equipment, cooling systems, and auxiliary infrastructure [17]. Among these components, IT equipment and cooling systems account for more than 80% of the total power consumption of the DC [18,19], and thus their physical parameters can be directly regulated to achieve significant energy-saving effects. In terms of IT equipment regulation, dynamic voltage and frequency scaling (DVFS) technology reduces server power consumption by real-time sensing of system load changes and dynamically adjusting the operating voltage and frequency of chips [20,21,22]. Ref. [23] maximized system performance by determining the optimal number of cores and the corresponding thread layout. Ref. [24] proposed an energy-aware virtual machine scheduling method based on thermal models. In terms of cooling system regulation, Ref. [25] constructed an integrated energy system for DCs to achieve efficient and low-carbon operation through the electro-thermal coupling among subsystems. Refs. [26,27] investigated the energy-saving benefits of air-side free-cooling technology under different meteorological conditions. Ref. [28] minimized the energy consumption of cooling systems through a novel economic model predictive control strategy while maintaining the stability of the thermal environment in DCs.

However, the above studies only optimize the computational domain (i.e., spatiotemporal scheduling of computing tasks) and physical domain (i.e., regulation of IT and cooling equipment) in isolation, ignoring their coupling relationship. Isolated optimization of either domain may degrade the performance of the other. Some studies have attempted to break down the barriers between task scheduling and physical control. Ref. [29] integrated three coupled regulation methods, namely geographic load balancing, batch task scheduling, and building thermal inertia-based cold storage, to achieve a reduction in energy consumption costs. Ref. [30] realized the goal of DC energy efficiency optimization through the regulation of internal components and load scheduling of DCs. Ref. [31] constructed a collaborative optimization model for spatiotemporal load balancing and energy management in DCs. However, these works still consider limited factors, often overlooking the intricate coupling between computing and physical domains under multidimensional constraints. In fact, the collaborative optimization is a complex process that requires balancing multiple objectives under the following key factors: (1) Network constraints: cross-DC task migration must ensure sufficient bandwidth to prevent transmission latency from violating service level agreements (SLAs). (2) Thermal constraints: task migration or IT equipment adjustment reshapes internal power distribution, requiring cooling regulation to avoid both local temperature violations and surges in cooling energy consumption [32,33]. (3) Computing constraints: adjustments to physical equipment parameters may lead to task execution timeout or failure, necessitating a balance between energy efficiency optimization and computational reliability. (4) Uncertainty: fluctuations in task arrivals, electricity prices, and other factors necessitate the design of algorithms to cope with decision-making risks in dynamic environments.

Thus, the regulation problem of DCs is essentially a high-dimensional, stochastic, and dynamic sequential decision-making problem. Traditional model-based or rule-based optimization methods struggle to achieve real-time and robust control in such environments. With the development of AI technology, reinforcement learning (RL) has been gradually introduced into the field of DC energy efficiency optimization [34,35,36,37,38,39]. However, most of them are not designed with a hierarchical decision-making structure, making it difficult to balance global coordination and local autonomy. Moreover, they cannot effectively handle heterogeneous hybrid actions coexisting with discrete task scheduling and continuous physical regulation, nor adapt to scenarios where the action dimension varies dynamically with tasks.

1.3. Contributions

As shown in

Table 1. Comparison between this paper and the existing literature.

Reference	[12]	[17]	[18]	[19]	[29]	[30]	[36]	[38]	[39]	This Paper
Task decomposition and subtask dependencies	×	×	×	×	×	×	√	√	×	√
Task temporal optimization	√	√	×	√	√	√	√	√	√	√
Cross-DC task migration	×	×	×	×	√	√	×	√	√	√
Intra-DC task migration	×	√	×	×	×	×	√	×	×	√
Proactive IT equipment state regulation	×	√	×	√	√	√	√	×	√	√
Cooling system operation optimization	√	√	√	√	√	√	×	×	×	√
Uncertainty handling	√	×	×	√	√	√	√	√	√	√

, which provides a comparison of this study with the existing literature, it can be concluded that this paper comprehensively considers multiple regulatory approaches to achieve the integrated energy management of DCs, with the main contributions as follows:

(1)

A computational–physical deeply coupled collaborative optimization model is established, which integrates spatiotemporal migration of tasks, dynamic adjustment of server core count and operating frequency, and adaptive switching of cooling modes.

(2)

The optimization problem is formulated as a two-layer POMG, which realizes the organic coordination between global cooperative decision-making and local autonomous control.

(3)

The H-MADDPG algorithm is proposed with a global coordinator agent (GCA) and dual heterogeneous actors in each local DC agent (LDCA), combined with task priority and a variable-dimension action mask to adapt to hybrid decision-making and dynamic task arrivals.

(4)

A geographically heterogeneous DC simulation is conducted, and the superior performance of the proposed method in cost, PUE, resource utilization, and load balancing is verified through benchmark, seasonal adaptability, and sensitivity analyses.

1.4. Structure of the Paper

The remainder of this paper is organized as follows. Section 2 establishes a collaborative optimization model. Section 3 transforms the model into a two-layer POMG. Section 4 proposes an H-MADDPG algorithm. Section 5 presents the experimental analysis, and Section 6 summarizes the work of this paper and prospects future research directions.

2. Computational–Physical Collaborative Optimization Model

This paper considers a system managed by a single operator, which comprises the following core components:

DC: Each DC $k\in \mathcal{K}=\{1,2,\dots ,K\}$ operates in a distinct external environment, including real-time electricity prices $C_k^e(t)$ and outdoor temperatures $T_k^{out}(t)$.

Zones: The interior of DC $k$ is divided into $M_k$ independent zones and the cooling parameter of each zone $m\in {\mathcal{M}}_k=\{1,2,\dots M_k\}$ can be adjusted independently.

Aggregated server (AS): We abstracted all servers within cooling zone $m$ as a single AS. This AS has the following attributes: (1) Total computing capacity $N_{k,m}$: the sum of the physical cores of all servers in the zone. (2) Peak single-core processing capacity ${\mu }_{k,m}^{peak}$: the processing speed of servers in the zone when running at the maximum operating frequency. (3) Active core ratio $a_{k,m}(t)\in [0,1]$: defined as the ratio of the number of online active cores to the total number of cores. (4) Frequency scaling factor $f_{k,m}(t)\in [f_{\min },1]$: the ratio of the actual operating frequency of cores to their peak frequency. (5) Core occupancy rate $U_{k,m}(t)\in [0,1]$: defined as the ratio of the number of cores executing tasks to the total number of active cores.

Computational tasks: Task $i\in \mathcal{I}=\{1,2,\dots ,I\}$ can be represented by a directed acyclic graph (DAG) $G_i=\{{\mathcal{V}}_i,{ℰ}_i\}$, where ${\mathcal{V}}_i$ denotes the set of subtasks inside task i and ${ℰ}_i$ defines the dependency relationships between subtasks. For task i, its arrival time is denoted as $A_i$, the initial arrival DC is $O_i$, the latest completion time is $D_i$, and the bandwidth requirement is $B_i$. Subtask $v\in {\mathcal{V}}_i=\{1,2,\dots V_i\}$ is the minimum unit for scheduling and execution, with an initial computing demand of $Co{m}_{i,v}^{ini}$, a remaining computing demand at time $t$ of $Co{m}_{i,v}(t)$, and a core demand of $Co{r}_{i,v}$.

The operator is required to dynamically make the following decisions to minimize the total cost on the premise of satisfying task service quality and ensuring the safe operation of equipment: (1) Spatial migration of tasks across DCs $z_{i,k}$. (2) Execution zone $y_{i,v,k,m}$ and execution time $s_{i,v}$ of subtasks within a DC. (3) IT equipment regulation: active core ratio $a_{k,m}(t)$ and frequency scaling factor $f_{k,m}(t)$. (4) Cooling system regulation: fan speed $s_{k,m}(t)$ or set temperature $T_{k,m}^{set}(t)$. Figure 1 depicts the architecture of the computational–physical collaborative optimization model. The computational domain handles task decomposition, spatiotemporal migration, and subtask scheduling. The physical domain manages IT equipment regulation (active core ratio and frequency) and cooling system control (fan speed or set temperature). The two domains are coupled through thermal and computational constraints: task allocation affects IT power consumption, which in turn impacts cooling load and indoor temperature, while equipment regulation influences task execution efficiency.

2.1. Modeling of Task Allocation and Execution

The number of subtasks within each task is determined by its internal DAG structure and is assumed to be known upon task arrival. Subtasks are not dynamically split by the system during runtime; instead, they are predefined at the time of task submission. This ensures that all dependency relationships and computing demands are fully observable to the scheduler.

$z_{i,k}\in \{0,1\}$ is a task-level allocation variable, where $z_{i,k}=1$ indicates that task $i$ is allocated to DC $k$, and $z_{i,k}=0$ otherwise. $y_{i,v,k,m}\in \{0,1\}$ is a subtask-level allocation variable, where $y_{i,v,k,m}=1$ indicates that subtask $v$ of task $i$ is allocated to zone $m$ of DC $k$ for execution, and $y_{i,v,k,m}=0$ otherwise.

Each task must be allocated to exactly one DC:

$${\displaystyle \sum _{k\in \mathcal{K}}{z}_{i,k}=1 }\forall i\in \mathcal{I}$$

(1)

Since the minimum granularity of spatial allocation is a complete task, and task decomposition is only performed after the task has been assigned to the target DC; all subtasks belonging to the same task must be allocated to the same DC as the task itself:

$${\displaystyle \sum _{m\in {\mathcal{M}}_k}{y}_{i,v,k,m}=z_{i,k}} \forall i\in I,v\in {\mathcal{V}}_i,k\in \mathcal{K}$$

(2)

This constraint ensures that if $z_{i,k}=1$, the sum of the allocations of each subtask $v$ of task i across all zones within DC $k$ equals 1, i.e., the subtask must be allocated within DC $k$; if $z_{i,k}=0$, the sum equals 0, i.e., the subtask cannot be allocated within DC $k$.

To characterize the arrival time of tasks at the original DC, a variable ${\delta }_{i,k}(t)$ is introduced here, where ${\delta }_{i,k}(t)=1$ denotes that task $i$ initially arrives at DC $k$ at time $t$.

$${\delta }_{i,k}(t)=\left\{\begin{array}{l}1 if O_i=k and A_i=t\\ 0 otherwise\end{array}\right.$$

(3)

At time $t$, if a task initially arrives at DC $k$ and is ultimately allocated to DC $l$, this indicates that task $i$ needs to be transmitted across DCs, which triggers bandwidth occupation. Transmission is assumed to be completed instantaneously, and the transmission is executed immediately upon the task’s arrival. The sum of the bandwidth requirements of all tasks to be transmitted from DC k to DC. $l$. must not exceed the link capacity $B_{k,l}$, so as to ensure that the bandwidth limit is not exceeded during the transmission instant:

$${\displaystyle \sum _{i\in {\mathcal{I}}_k^{arr}(t)}{B}_i{\delta }_{i,k}(t)z_{i,l}\le B_{k,l}} k\ne l$$

(4)

where ${\mathcal{I}}_k^{arr}(t)$ denotes the set of tasks arriving at DC $k$ at time $t$.

For task $i$ at time $t$, the set of its subtasks being executed, denoted as ${\mathcal{V}}_i^{exe}(t)$, is defined as follows:

$${\mathcal{V}}_i^{exe}(t)=\{v\in {\mathcal{V}}_i|s_{i,v}\le t\le f_{i,v}\}$$

(5)

where $s_{i,v}$ and $f_{i,v}$ denote the start execution time and finish execution time of subtask $v$ in task $i$, respectively.

Let $N_{k,m}^{active}(t)=a_{k,m}(t)N_{k,m}$ denote the number of active cores at time $t$. Among these active cores, some are occupied by executing tasks, while the remaining are idle. At any time $t$, the number of cores executing tasks in zone $m$ must not exceed its current number of available cores $N_{k,m}^a(t)$, so as to ensure that the activated computing cores are sufficient to meet the demand:

$$N_{k,m}^a(t)=N_{k,m}^{active}(t)(1-U_{k,m}(t))$$

(6)

$${\displaystyle \sum _{i\in \mathcal{I}}{\displaystyle \sum _{v\in {\mathcal{V}}_i^{exe}(t)}{y}_{i,v,k,m}Co{r}_{i,v}}\le N_{k,m}^a(t)}$$

(7)

The finish time of subtask $v$ is determined by its start time, computing demand, core demand, and the processing capacity of the AS in the corresponding zone. The remaining computing demand of subtask $v$ at time $t$ can be expressed as:

$$Co{m}_{i,v}(t)=Co{m}_{i,v}(t-1)-{\displaystyle \sum _{k\in \mathcal{K}}{\displaystyle \sum _{m\in {\mathcal{M}}_k}{y}_{i,v,k,m}Co{r}_{i,v}{\mu }_{k,m}(t-1)\Delta t}}$$

(8)

$${\mu }_{k,m}(t-1)=f_{k,m}(t-1){\mu }_{k,m}^{peak}$$

(9)

where ${\mu }_{k,m}(t)$ denotes the actual operating frequency after frequency scaling. Since $y_{i,v,k,m}$ is a binary variable and each subtask is allocated to exactly one server (as shown in(1) and (2)), only one term in the summation is actually valid.

A subtask is considered to be completed when its remaining computing demand is less than or equal to zero.

$$f_{i,v}=\min \left\{t\right|Co{m}_{i,v}(t)\le 0\}$$

(10)

For any dependency edge $(v,u)\in {ℰ}_i$, which indicates that subtask $v$ is the predecessor of subtask $u$, subtask $u$ can only start execution after subtask $v$ is completed, and the following constraint must be satisfied:

$$s_{i,u}\ge f_{i,v} \forall (v,u)\in {ℰ}_i , i\in \mathcal{I}$$

(11)

Herein, the subtask level $H_{i,u}$ is defined. A higher level means that the subtask is executed later in the dependency chain, and it can only be initiated after more predecessor subtasks are completed. The quantification rule for the level can be expressed as follows:

$$H_{i,u}=\left\{\begin{array}{l}1 if \{v\in {\mathcal{V}}_i|(v,u)\in {ℰ}_i\}=\varnothing \\ \max \{H_{i,v}+1|v\in {\mathcal{V}}_i|(v,u)\in {ℰ}_i\} otherwise\end{array}\right.$$

(12)

where $v$ is the predecessor subtask of $u$; if subtask $u$ has no predecessor subtasks, its level is 1; and if subtask $u$ has predecessor subtasks, its level is equal to the maximum level value of all its predecessor subtasks plus one.

For each task $i$, the start time of its first subtask shall not be earlier than its arrival time, and the completion time of its last subtask must be no later than its latest completion time [40]:

$$\underset{v\in {\mathcal{V}}_i}{\min }{s}_{i,v}\ge A_i \forall i\in \mathcal{I}$$

(13)

$$\underset{v\in {\mathcal{V}}_i}{\max }{f}_{i,v}\le D_i \forall i\in \mathcal{I}$$

(14)

2.2. Modeling of AS

The core occupancy rate $U_{k,m}$ of the AS in zone $m$ at time $t$ is defined as the ratio of the number of cores executing tasks to the total number of active cores:

$$U_{k,m}(t)={\displaystyle \frac{{\displaystyle \sum _{i\in \mathcal{I}}{\displaystyle \sum _{v\in {\mathcal{V}}_i^{exe}(t)}{y}_{i,v,k,m}Co{r}_{i,v}}+N_{k,m}^{pre}(t)}}{N_{k,m}^{active}(t)}}$$

(15)

where fixed tasks refer to critical workloads that cannot be delayed, migrated, or preempted due to operational continuity, security constraints, or user specifications. They occupy a fixed number of cores $N_{k,m}^{pre}(t)$ and are not subject to spatiotemporal migration optimization. However, they still participate in SLA constraints and must be completed before their deadlines.

The total IT power $P_{k,m}^{IT}(t)$ of the AS in zone $m$ at time $t$ consists of three components: static power $P_{k,m}^{st}(t)$, dynamic computing power $P_{k,m}^{dy}(t)$, and state transition power $P_{k,m}^{tr}(t)$ [41].

$$\left\{\begin{array}{l}{P}_{k,m}^{IT}(t)=P_{k,m}^{st}(t)+P_{k,m}^{dy}(t)+P_{k,m}^{tr}(t)\\ P_{k,m}^{st}(t)=N_{k,m}^{active}(t)P_{k,m}^{idle}\\ P_{k,m}^{dy}(t)=N_{k,m}^{active}(t)(P_m^{peak}-P_m^{idle}){(f_{k,m}(t))}^{\gamma }{U}_{k,m}(t)\\ P_{k,m}^{tr}(t)={\beta }_a|\Delta a_{k,m}|N_{k,m}+{\beta }_f|\Delta f_{k,m}|N_{k,m}\end{array}\right.$$

(16)

where the static power refers to the minimum power required to maintain the basic operation of the server. $P_{k,m}^{idle}$ denotes the idle power of a single core. The dynamic computing power is the additional power generated during the execution of tasks. $P_{k,m}^{peak}$ represents the peak power, and $\gamma$ is the dynamic power exponent. The state transition power refers to the instantaneous power overhead generated when the server state is altered. $\Delta a_{k,m}(t)$ and $\Delta f_{k,m}(t)$ denote the variation in the active core ratio and the frequency scaling factor, respectively; ${\beta }_a$ and ${\beta }_f$ are the energy consumption coefficients for core count transition and frequency transition, respectively.

The total power of DC $k$ at time $t$ is the sum of the IT power, cooling power, and auxiliary facility power. The power of auxiliary facilities is considered to be correlated with the power of IT equipment:

$$P_{k,m}^{aux}(t)=\beta \cdot P_{k,m}^{IT}(t)$$

(17)

$$P_k^{total}(t)={\displaystyle \sum _{m\in {\mathcal{M}}_k}(P_{k,m}^{IT}(t)}+P_{k,m}^{cool}(t)+P_{k,m}^{aux}(t))$$

(18)

where $\beta$ is a proportional coefficient covering the conversion power consumption of equipment, such as UPS, and the basic lighting energy consumption.

The PUE of DC $k$ is the ratio of the total input power of the DC to the actual power consumed by IT equipment, which can be expressed as:

$$PU{E}_k(t)={\displaystyle \frac{P_k^{total}(t)}{P_k^{IT}(t)}}$$

(19)

2.3. Modeling of Cooling System

The cooling system can switch between free cooling mode and mechanical cooling mode according to the outdoor temperature $T_k^{out}(t)$:

$$Mod{e}_k(t)=\left\{\begin{array}{l}0 T_k^{out}(t)\le T^{th}\\ 1 T_k^{out}(t)>T^{th}\end{array}\right.$$

(20)

where $Mod{e}_k(t)=0$ represents the free cooling mode and $Mod{e}_k(t)=1$ represents the mechanical cooling mode.

When the system is in the free cooling mode, the compressor is completely shut down, and heat is removed by outdoor low-temperature air only by adjusting the flow rate of the fluid. The power is regulated by controlling the rotation speed $s_{k,m}(t)$ of fans/water pumps. When the equipment operates at full speed, the maximum heat power that can be removed based on the temperature difference is given by:

$$Q_{k,m}^{\max }(t)={\kappa }_{k,m}(T_{k,m}^{in}(t)-T_{k,m}^{out}(t))$$

(21)

where ${\kappa }_{k,m}$ is the total heat transfer coefficient of the region, which depends on the heat exchanger area and air duct design.

The actual heat removal capacity is proportional to the rotation speed:

$$Q_{k,m}^{cool}(t)=s_{k,m}(t)\cdot Q_{k,m}^{\max }(t)$$

(22)

To ensure that the heat generated by IT equipment is completely removed, the following condition must be satisfied:

$$Q_{k,m}^{cool}(t)\ge P_{k,m}^{IT}(t)$$

(23)

The power consumption of fans/water pumps is proportional to the cube of the rotation speed:

$$P_{k,m}^{cool}(t)=P^{rated}{(s_{k,m}(t))}^3$$

(24)

where $P^{rated}(t)$ is the rated maximum power of the equipment.

When the system is in the mechanical cooling mode, the compressor is activated to actively transfer heat through the refrigeration cycle. The evaporation temperature is changed by adjusting the temperature set point $T_{k,m}^{set}(t)$.

The coefficient of performance (COP) is defined as the ratio of the heat removal capacity to the input electrical power, which can be modeled as:

$$CO{P}_{k,m}(t)=a+b{T}_{k,m}^{set}(t)$$

(25)

where $a$, $b$ are empirical fitting parameters calibrated to real-world DC cooling system characteristics.

The power consumption of the cooling system is jointly determined by the heat to be removed (approximately equal to the IT power consumption) and the cooling efficiency:

$$P_{k,m}^{cool}(t)={\displaystyle \frac{Q_{k,m}^{cool}(t)}{CO{P}_{k,m}(t)}}+P^{fix}={\displaystyle \frac{P_{k,m}^{IT}(t){\eta }_{k,m}(t)}{CO{P}_{k,m}(t)}}+P^{fix}$$

(26)

$${\eta }_{k,m}(t)=c-d\cdot CO{P}_{k,m}(t)$$

(27)

where ${\eta }_{k,m}(t)$ is the cooling efficiency coefficient, which is related to $CO{P}_{k,m}(t)$,$c$, $d$ are fitting coefficients; and $P^{fix}$ is the baseline power consumption of equipment during compressor operation.

To ensure equipment safety, the server air inlet temperature $T_{k,m}^{in}(t)$ must be maintained within the allowable range. Its dynamics are described by the continuous-time heat balance equation:

$$C_{k,m}{\displaystyle \frac{d{T}_{k,m}^{in}(t)}{dt}}=P_{k,m}^{IT}(t)-{\displaystyle \frac{T_{k,m}^{in}(t)-T_k^{out}(t)}{R_{k,m}}}-Q_{k,m}^{cool}(t)$$

(28)

where $C_{k,m}$ denotes the thermal capacity of the zone and $R_{k,m}$ is the thermal resistance of the zone. In practical control, the discrete form is usually adopted:

$$T_{k,m}^{in}(t+1)=T_{k,m}^{in}(t)+{\displaystyle \frac{\Delta t}{C_{k,m}}}[P_{k,m}^{IT}(t)-{\displaystyle \frac{T_{k,m}^{in}(t)-T_k^{out}(t)}{R_{k,m}}}-Q_{k,m}^{cool}(t)]$$

(29)

where $\Delta t$ denotes the length of the time interval.

The temperature safety constraint is expressed as follows:

$$T^{\min }\le T_{k,m}^{in}(t)\le T^{\max }$$

(30)

where $T^{\min }$ and $T^{\max }$ denote the minimum and maximum values of the safe temperature, respectively.

2.4. Modeling of Cost

The optimization objective of the model is to minimize the total operating cost, which includes the computing cost $C^{IT}$, cooling cost $C^{cool}$, transmission cost $C^{tr}$, and SLA violation cost $C^{SLA}$:

$$\left\{\begin{array}{l}\min J=C^{IT}+C^{cool}+C^{tr}+C^{SLA}\\ C^{IT}={\displaystyle \sum _{t\in \mathcal{T}}{\displaystyle \sum _{k\in \mathcal{K}}{C}_k^e{P}_k^{IT}\Delta t}}\\ C^{cool}={\displaystyle \sum _{t\in \mathcal{T}}{\displaystyle \sum _{k\in \mathcal{K}}{C}_k^e{P}_k^{cool}\Delta t}}\\ C^{tr}={\displaystyle \sum _{t\in \mathcal{T}}{\displaystyle \sum _{k\in \mathcal{K}}{\displaystyle \sum _{l\in \mathcal{K}}{\displaystyle \sum _{i\in I_k^{arr}(t)}{a}^{tr}{B}_i{\delta }_{i,k}(t)z_{i,l} l\ne k}}}}\\ C^{SLA}=a^{SLA}{r}^{SLA}\end{array}\right.$$

(31)

where $a^{tr}$ and $a^{SLA}$ denote the communication cost coefficient and SLA violation cost coefficient, respectively, and ρ represents the SLA violation rate.

3. Two-Layer POMG

Based on the computational–physical collaborative optimization model, this section further formulates the high-dimensional, dynamic, and strongly coupled optimization problem as a two-layer POMG. By defining hierarchical agents, state space, action space, and the reward mechanism, it provides a unified decision-making framework for the subsequent H-MADDPG algorithm.

3.1. Agent Architecture and Division of Responsibilities

This paper adopts a two-layer agent architecture to balance the efficiency of global coordination and local control.

GCA: As the central decision-making unit, the GCA is responsible for the top-level decision-making of global scheduling to ensure the optimal overall cost of the system, and its decisions correspond to the variables $z_{i,k}$ in the optimization model.

LDCA: One LDCA is deployed in each DC, and it is responsible for all micro-level decision-making within the DC: (1) subtask scheduling; (2) zone AS state control; and (3) cooling control decision-making.

3.2. Task Priority Ranking

At each time step, the GCA collects information about the tasks arriving at each DC. Let ${\mathcal{I}}^{wait}(t)$ denote the set of pending tasks of the GCA, and ${\mathcal{I}}^{cur}(t)$ denote the set of tasks processed by the GCA at the current time step. Given that the maximum number of tasks that the GCA can process per time step is $N^G$, it is necessary to perform priority ranking on ${\mathcal{I}}^{wait}(t)$. The priority score $P_i$ of each task $i$ depends on its total computing demand $Co{m}_{i,v}$ and deadline $D_i$. A higher priority score indicates a more urgent task, which should be processed first:

$$P_i={\alpha }_{1}Co{m}_i+{\alpha }_2{\displaystyle \frac{1}{D_i}}$$

(32)

where ${\alpha }_1$ and ${\alpha }_2$ are the weights of the task priority score.

The top $N^G$ tasks with the highest priority are included in ${\mathcal{I}}^{cur}(t)$, and their spatiotemporal migration is determined by the GCA.

Let ${\mathcal{V}}_k^{wait}(t)$ denote the set of pending subtasks in DC $k$, and ${\mathcal{V}}_k^{cur}(t)$ denote the set of subtasks processed in the current time step. The maximum number of subtasks that the LDCA can process per time step is $N^L$. Similarly, the subtasks are prioritized according to their computing demand and deadline, and the top $N^L$ subtasks with the highest priority scores are selected for decision-making.

3.3. State Space

The state observed by the GCA at time step $t$ is denoted as ${\mathcal{S}}^G=\{t,{\mathcal{J}}^{cur}(t),\mathcal{R}(t)\}$, where $t$ represents the current time step, and ${\mathcal{J}}^{cur}(t)=\{Co{m}_i,D_i,V_i|i\in {\mathcal{I}}^{cur}(t)\}$ is the set of task states in the task set ${\mathcal{I}}^{cur}(t)$ processed by the GCA at the current time step, including task computing demand $Co{m}_i$, deadline $D_i$, and the number of subtasks $V_i$. $\mathcal{R}(t)=\{C_k^e(t),T_k^{out}(t),{\widehat{\mathrm{C}}}_k^{e,f}(t),{\widehat{\mathrm{T}}}_k^{out,f}(t),{\overline{T}}_k^{in},{\overline{\mu }}_k(t),N_k^a(t)|k\in \mathcal{K}\}$ denotes the state set of all DCs, including the electricity price $C_k^e(t)$, outdoor temperature $T_k^{out}(t)$, predicted values of electricity price ${\widehat{\mathrm{C}}}_k^{e,f}(t)$ and outdoor temperature ${\widehat{\mathrm{T}}}_k^{out,f}(t)$ for the next $f$ time steps, zone indoor average temperature ${\overline{T}}_k^{in}(t)$, zone average frequency ${\overline{\mu }}_k(t)$, and the total number of available cores of the DC $N_k^a(t)$. Based on the above states, the GCA is trained to identify the urgency level of tasks and make optimal allocation decisions according to task requirements and the real-time states of each DC.

Two policy networks are configured for each LDCA: one is responsible for deciding the start execution time and execution zone of subtasks, which is referred to as the subtask scheduling actor; the other is responsible for regulating the states of each zone within the DC, which is referred to as the resource state regulation actor. Both policy networks take the same state as input but output different actions, which will be elaborated upon later. The state observed by LDCA-k at time step $t$ is denoted as ${\mathcal{S}}_k^L(t)=\{t,{\mathcal{J}}^{cur}(t),{\mathcal{Z}}_k(t),{\mathcal{R}}_k(t)\}$, where $t$ represents the current time step, ${\mathcal{J}}_k^{cur}(t)=\{H_{i,v},Co{m}_{i,v}|i\in {\mathcal{I}}_k^{tr}(t),v\in {\mathcal{V}}_k^{cur}(t)\}$ denotes the set of states of the subtask set ${\mathcal{V}}_k^{cur}(t)$ currently processed by LDCA-k, including the subtask level $H_{i,v}$ and subtask computing demand $Co{m}_{i,v}$, ${\mathcal{I}}_k^{tr}(t)$ denotes the set of tasks allocated to DC $k$ up to time step $t$, ${\mathcal{Z}}_k=\{T_{k,m}^{in}(t),{\mu }_{k,m}(t),N_{k,m}^a(t)|m\in {\mathcal{M}}_k\}$ denotes the states of each zone within the DC, including indoor temperature $T_{k,m}^{in}(t)$, frequency ${\mu }_{k,m}(t)$, and number of available cores $N_{k,m}(t)$, $R_k(t)=\{C_k^e(t),T_k^{out}(t),{\widehat{\mathrm{C}}}_k^{e,f}(t),{\widehat{\mathrm{T}}}_k^{out,f}(t)\}$ denotes the state of this DC, including the electricity price $C_k^e(t)$ and outdoor temperature $T_k^{out}(t)$ at the current moment, as well as their predicted values ${\widehat{\mathrm{C}}}_k^{e,f}(t)$ and ${\widehat{\mathrm{T}}}_k^{out,f}(t)$ for the next $f$ time steps.

3.4. Action Space

The GCA is responsible for the spatial migration of tasks processed at each time step, i.e., determining which DC the tasks are secondarily allocated to. Its action is defined as ${\mathcal{A}}^G(t)=\left\{a_i^{tr}\right|i\in {\mathcal{I}}^{cur}(t)\}$, where $a_i^{tr}\in \{1,2,\dots ,K\}$ indicates the DC to which the task $i$ in the task set ${\mathcal{I}}^{cur}(t)$ processed by the GCA at the current time step is allocated.

The subtask scheduling actor of LDCA-k is responsible for subtask decision-making, and its action can be expressed as ${\mathcal{A}}_k^{L,st}(t)=\{r_{k,i,v},t_{k,i,v}^{norm}|i\in {\mathcal{I}}_k^{tr}(t),v\in {\mathcal{V}}_k^{cur}(t)\}$. Herein, $r_{k,i,v}\in \{1,2,\dots ,M_k\}$ denotes the zone allocated to the subtask $v$ in the subtask set ${\mathcal{V}}_k^{cur}(t)$ processed by the LDCA-k at the current time step; $t_{k,i,v}^{norm}\in [0,1)$ denotes the planned start execution time of the subtask $v$ in the subtask set ${\mathcal{V}}_k^{cur}(t)$ processed by the LDCA-k at the current time step, which is converted to the actual time step via the mapping formula $t_{k,i,v}=t_{k,i,v}^{\min }+t_{k,i,v}^{norm}(t_{k,i,v}^{\max }-t_{k,i,v}^{\min })$. $t_{k,i,v}^{\min }$ is the earliest start time, namely the arrival time of the subtask; $t_{k,i,v}^{\max }$ is the latest start time, namely the deadline of the subtask. The resource state regulation actor of LDCA-k is responsible for decision-making regarding resource state regulation of each zone, and its action can be expressed as $A_k^{L,re}=\{\Delta a_{k,m}(t),\Delta f_{k,m}(t),\Delta s_{k,m}(t)/\Delta T_{k,m}^{set}(t)|m\in {\mathcal{M}}_k\}$. Herein, $\Delta a_{k,m}(t)$, $\Delta f_{k,m}(t)$, $\Delta s_{k,m}(t)$ and $\Delta T_{k,m}^{set}(t)$ denote the variation in the active core ratio, frequency scaling factor, fan speed, and set temperature, respectively. When in the free cooling mode, the action output by the policy network is $\Delta s_{k,m}(t)$; when in the mechanical cooling mode, the output action is $\Delta T_{k,m}^{set}(t)$.

3.5. Reward Function

The reward function $r^G(t)$ of the GCA consists of the global electricity cost reward $r^e(t)$, the global SLA compliance reward $r^{SLA}(t)$, the transmission bandwidth violation penalty $r^{band}(t)$, and the load balancing reward $r^{balance}(t)$:

$$r^G(t)={\omega }_1{r}^e(t)+{\omega }_2{r}^{SLA}(t)+{\omega }_3{r}^{band}(t)+{\omega }_4{r}^{balance}(t)$$

(33)

$$r^e(t)=1-{\displaystyle \frac{C^e(t)}{C^{\max }}}$$

(34)

$$r^{SLA}=\left\{\begin{array}{l}0 if s_{i,v}>D_i\\ 1 else\end{array}\right.$$

(35)

$$r^{band}=\left\{\begin{array}{l}0 if B(t)>B^{\max }\\ 1 else\end{array}\right.$$

(36)

$$r^{balence}=-{\displaystyle \sum _{k\in \mathcal{K}}(\frac{P_k^{total}}{P_k^{\max }}-}{\displaystyle \frac{{\displaystyle \sum _{k\in \mathcal{K}}(\frac{P_k^{total}}{P_k^{\max }})}}{K}}{)}^2$$

(37)

where $C^{total}$ is the total electricity cost at time and $C^{\max }$ is the maximum observed cost in the training buffer; $B(t)$ is the bandwidth requirement of tasks that violate the link capacity; and $B^{\max }$ is the total link capacity.

The weights ${\omega }_1~{\omega }_4$ are set to $[0.5,0.3,0.1,0.1]$ based on a preliminary grid search over the ranges ${\omega }_1\in [0.3,0.7]$, ${\omega }_2\in [0.2,0.5]$, ${\omega }_3\in [0.05,0.2]$.

The reward function $r_k^L(t)$ of the LDCA-k comprises the local electricity cost reward $r_k^e(t)$, the local SLA compliance reward $r_k^{SLA}(t)$, and the physical resource safety reward $r_k^{re}$, where the latter includes temperature safety and core occupancy rate non-violation constraints.

$$r_k^L(t)={\omega }_5{r}_k^e(t)+{\omega }_6{r}_k^{SLA}(t)+{\omega }_7{r}_k^{re}(t)$$

(38)

$$r_k^{re}(t)=-{\displaystyle \sum _{m\in M_k}(\frac{T_{k,m}^{in}(t)}{T^{\max }}}+U_{k,m}(t))$$

(39)

The definition of $r_k^e(t)$ and $r_k^{SLA}(t)$ is similar to the GCA. The weights ${\omega }_5~{\omega }_7$ are set to $[0.4,0.3,0.3]$.

4. Algorithm Design

4.1. Principle of the Classical MADDPG Algorithm

The MADDPG algorithm provides a fundamental framework for addressing the non-stationarity problem in multi-agent environments. In the training phase, a centralized critic network that can access the state and action information of all agents is introduced for each agent. In the execution phase, each agent makes distributed decisions solely based on its local observations [42].

In the multi-agent setting, the goal of agent $i$ is to maximize its expected discounted cumulative reward:

$$J({\pi }_i)=E[{\displaystyle \sum _{t\in \mathcal{T}}{\gamma }^t{r}_i(t)}]$$

(40)

where ${\pi }_i$ denotes the policy of agent $i$, $\gamma \in [0,1]$ is the discount factor that balances the importance of immediate and future rewards, and $r_i(t)$ represents the immediate reward obtained by the agent at time step $t$.

The MADDPG algorithm adopts a collaborative architecture of the centralized critic and distributed actor, and its parameter update logic is divided into two parts: the centralized critic network update and the distributed actor network update.

The core function of the centralized critic network is to evaluate the global value of the joint state–joint action. Its parameters $\varphi$ are updated by minimizing the temporal difference mean-squared error between the current Q-value and the target Q-value, so as to optimize the accuracy of value assessment.

The target Q-value is jointly calculated by the target actor network and the target critic network:

$$y_i=r_i+\gamma Q^{{\varphi }^{\prime }}(s^{\prime },a_1^{\prime },\dots a_N^{\prime }){|}_{a_j^{\prime }={\mu }^{{\theta }_j^{\prime }}(o_j)}$$

(41)

where $s^{\prime }$ denotes the next global state, $r_i$ is the current global reward, $\gamma$ is the discount factor, $a_j^{\prime }={\mu }^{{\theta }_j^{\prime }}(o_j)$ represents the action generated by the target actor network of agent $j$ (with parameters ${\theta }_j^{\prime }$) based on its local observation $o_j$, and $Q^{{\varphi }^{\prime }}$ is the function of the target critic network (with parameters ${\varphi }^{\prime }$) that evaluates the future value corresponding to the next state and target action.

The loss function can be expressed as:

$$L(\varphi )=E_{(s,a,r,s^{\prime })~\mathcal{D}}[{(y_i-Q^{\varphi }(s,a_1,\dots ,a_N))}^2]$$

(42)

where $D$ denotes the experience replay buffer, stored in the form of $(\mathrm{S}(t),\mathrm{A}(t),r^{total}(t),\mathrm{S}(t+1),done(t))$, where $s$ is the global state, $a=[a_1,\dots a_N]$ is the joint action of all agents, and $Q^{\varphi }(s,a_1,\dots ,a_N)$ is the value assessment of the online state and joint action performed by the current critic network (with parameters $\varphi$).

The parameter update of the online critic network is completed by minimizing the aforementioned loss via gradient descent: $\varphi \leftarrow \varphi -{\eta }_Q\cdot {\nabla }_{\varphi }L(\varphi )$, where ${\eta }_Q$ is the learning rate of the critic network.

The actor network of each agent is updated using the policy gradient theorem. Its core objective is to maximize the action-value Q-value evaluated by the critic network, thereby optimizing its own policy. The expression for agent $i$ is given by:

$${\nabla }_{{\theta }_i}J({\mu }^{{\theta }_i})\approx E_{s,a~\mathcal{D}}[{\nabla }_{{\theta }_i}{\mu }^{{\theta }_i}(o_i){\nabla }_{a_i}{Q}^{\varphi }(s,a_1,\dots ,a_N)]$$

(43)

where ${\mu }^{{\theta }_i}(o_i)$ denotes the action generated by the online actor network (with parameters ${\theta }_i$) of agent $i$ based on its local observation $o_i$.

To adapt to the gradient descent optimization logic, the objective of maximizing the Q-value is transformed into a loss minimization problem, whose loss function is defined as follows:

$$L({\theta }_i)=-E_{(s,a,r,s^{\prime })~\mathcal{D}}[Q^{\varphi }(s,a_1,\dots ,a_N)]$$

(44)

The parameter update of the online actor network is achieved by minimizing the aforementioned loss via gradient descent: ${\theta }_i\leftarrow {\theta }_i-{\eta }_{Actor}\cdot {\nabla }_{{\theta }_i}L({\theta }_i)$, where ${\eta }_{Actor}$ denotes the learning rate of the actor network.

To ensure training stability, target networks (with parameters denoted as ${\varphi }^{\prime }$ and ${\theta }_i^{\prime }$) are constructed for all critic/actor networks, and a soft update strategy is adopted to synchronize the parameter updates of the target networks:

$${\varphi }^{\prime }=\tau \varphi +(1-\tau ){\varphi }^{\prime }$$

(45)

$${\theta }_i^{\prime }=\tau {\theta }_i+(1-\tau ){\theta }_i^{\prime }$$

(46)

where τ denotes the soft update coefficient, which balances the stability and tracking flexibility of the target network.

4.2. H-MADDPG: Network Architecture Design for Hierarchical Decision-Making

Traditional MADDPG employs a single-layer architecture with homogeneous agents, where each agent independently makes decisions based on local observations while sharing a centralized critic. However, this architecture faces three key difficulties when applied to DC optimization: (1) the coupling between global task allocation and local equipment control; (2) the coexistence of discrete (task scheduling) and continuous (frequency, cooling) action spaces; and (3) the dynamically varying action dimensions caused by uncertain task arrivals.

To address these challenges, the proposed H-MADDPG introduces three core innovations in

Table 2. Three core innovations of the proposed H-MADDPG.

Aspects	Traditional MADDPG	H-MADDPG
Architecture	Single-layer, homogeneous agents	Two-layer heterogeneous agents
Action space	Continuous	Hybrid (discrete + continuous) with variable-dimension masking
Decision logic	Flat, peer-to-peer	Hierarchical: global allocation, local scheduling, and control

.

Specifically, the GCA handles sparse, high-level task allocation decisions, while each LDCA manages dense, low-level subtask scheduling and real-time equipment regulation. This hierarchical decomposition directly mirrors the natural structure of data center operations and reduces the joint action space complexity from exponential to polynomial.

The network architecture of H-MADDPG mainly consists of the upper-level GCA task allocation actor, the lower-level LDCA subtask scheduling actor, the resource state regulation actor, and a centralized critic, thus enabling end-to-end learning from global task allocation down to local resource regulation. The overall architecture is illustrated in Figure 2: (a) Centralized training phase: The global centralized critic network collects joint states $\mathrm{S}$ and joint actions $\mathrm{A}$ from all agents (GCA and LDCAs) to compute the global Q-value. This critic is only used during training to provide gradient signals to the actor networks. (b) Decentralized execution phase: Each agent (GCA or LDCA) makes decisions solely based on its local observations using its trained actor network. The GCA determines cross-DC task migration, while each LDCA handles subtask scheduling and equipment regulation within its DC. No communication among agents is required during execution. Algorithm 1 presents the algorithm flow of H-MADDPG.

Algorithm 1: H-MADDPG

Input: Initial parameters of each network, experience replay buffer parameters, hyperparameters Output: Task allocation policy, subtask scheduling policy, resource regulation policy 1: Initialize parameters 2: for episode = 1 to 1000 do Reset environment 3: $\mathrm{while} t<T$ do: 4: $\mathrm{Upper}-\mathrm{layer}: \mathrm{The} \mathrm{GCA} \mathrm{outputs} {\mathcal{A}}^G(t)$ via the online GCA actor network 5: $\mathrm{Lower}-\mathrm{layer}: \mathrm{Each} \mathrm{LDCA} \mathrm{outputs} {\mathcal{A}}_k^{L,st}(t) \mathrm{and} {\mathcal{A}}_k^{L,re}(t)$ via its online LDCA dual-actor network 6:                 Environment interaction 7: $\mathrm{Store} \mathrm{the} \mathrm{sample} (\mathrm{S}(t),\mathrm{A}(t),r^{total},\mathrm{S}(t+1),done(t))$ in $\mathcal{D}$ 8: $t\leftarrow t+1$ 9:       end while 10: $\mathrm{if} \left|D\right|\ge 64$ then 11:                 Sample a batch of samples $\mathcal{B}$ from $\mathcal{D}$ 12: $\mathrm{Calculate} \mathrm{the} \mathrm{target} \mathrm{Q}-\mathrm{value} y_i=r_i+\gamma Q^{{\varphi }^{\prime }}(s^{\prime },a_1^{\prime },\dots a_N^{\prime })$ 13: $\mathrm{Online} \mathrm{central} \mathrm{critic} \mathrm{network} \mathrm{update}: \varphi \leftarrow \varphi -{\eta }_Q\cdot {\nabla }_{\varphi }L(\varphi )$ 14: $\mathrm{Online} \mathrm{GCA} \mathrm{actor} \mathrm{network} \mathrm{update}: {\theta }^G\leftarrow {\theta }^G-{\eta }^G\cdot {\nabla }_{{\theta }^G}L({\theta }^G)$ 15: $\mathrm{Online} \mathrm{LDCA} \mathrm{dual}-\mathrm{actor} \mathrm{network} \mathrm{update}: {\theta }_k^{L,st}\leftarrow {\theta }_k^{L,st}-{\eta }_k^{L,st}\cdot {\nabla }_{{\theta }_k^{L,st}}L({\theta }_k^{L,st}) {\theta }_k^{L,re}\leftarrow {\theta }_k^{L,re}-{\eta }_k^{L,re}\cdot {\nabla }_{{\theta }_k^{L,re}}L({\theta }_k^{L,re})$ 16: $$\begin{gathered} \mathrm{Target} \mathrm{network} \mathrm{update}: {\varphi }^{\prime }\leftarrow \tau \varphi +(1-\tau ){\varphi }^{\prime }{\theta }^{G^{\prime }}\leftarrow \tau {\theta }^G+(1-\tau ){\theta }^{G^{\prime }} \\ {\theta }^{L,s{t}^{\prime }}\leftarrow \tau {\theta }^{L,st}+(1-\tau ){\theta }^{L,s{t}^{\prime }} {\theta }^{L,r{e}^{\prime }}\leftarrow \tau {\theta }^{L,re}+(1-\tau ){\theta }^{L,r{e}^{\prime }} \end{gathered}$$ 17:   end if 18: end for

4.2.1. Upper Layer: GCA Task Allocation Actor Network

The GCA output is a probability distribution over target DCs for each task. Since the original MADDPG is designed for continuous actions, we adopt the Gumbel–Softmax reparameterization trick to make discrete action sampling differentiable during training. Specifically, the GCA actor outputs logits $\log p_{i,k}$ for each task–DC pair, and the sampled action is:

$$a_{i,k}={\displaystyle \frac{\exp ((\log p_{i,k}+g_{i,k})/\tau )}{{\displaystyle {\sum }_{k^{\prime }}\exp ((\log p_{i,k^{\prime }}+g_{i,k^{\prime }})/\tau )}}}$$

(47)

where $g_{i,k}\sim \mathrm{Gumbel}\left(\mathrm{0,1}\right)$ is the independent Gumbel noise, and $\tau$ is the temperature parameter (annealed from 1.0 to 0.1 during training). The action mask sets $\mathrm{l}\mathrm{o}\mathrm{g}{p}_{i,k}=-\mathrm{\infty }$ for invalid DCs (e.g., those that cannot receive tasks due to bandwidth constraints). During inference (execution), we replace the Gumbel–Softmax with deterministic argmax: the task is allocated to the DC with the highest probability.

The GCA task allocation actor ${\mu }^{{\theta }^G}$ is responsible for receiving the task arrival information of the entire network and determining the cross-DC task allocation targets. Its key design lies in mapping a variable number of tasks to fixed-dimensional action outputs.

Input Layer: This dimension is consistent with that of the GCA state space, with the input being ${\mathcal{S}}^G(t)$.

Output Layer and Action Masking Mechanism: To address the uncertainty of the number of arriving tasks, an action masking mechanism is added to ensure that the output dimension remains fixed at $N^G\times K$ (where $N^G$ is the maximum number of parallel tasks of the GCA and $K$ is the number of DCs). The GCA actor outputs the probability distribution of task allocation to each DC. For the time step with the actual number of tasks $N_t<N^G$, an action mask is introduced:

$${[logits]}_{i,k}=\left\{\begin{array}{l}{[net\_output]}_{i,k} if i\le N_t\\ -\infty otherwise\end{array}\right.$$

(48)

$${\pi }^G(a_i=k|s)={\displaystyle \frac{\exp ({[logits]}_{i,k})}{{\displaystyle \sum _{k^{\prime }\in \mathcal{K}}\exp ({[\log it]}_{i,k^{\prime }})}}}$$

(49)

where $N_t$ denotes the actual number of tasks at time step $t$, ${[logit]}_{i,k}$ is the probability that task $i$ is allocated to DC $k$, and ${\pi }^G(a_i=k|s)$ represents the probability that task $i$ is allocated to DC $k$.

4.2.2. Lower Layer: LDCA Dual-Actor Network

Each DC deploys a subtask scheduling actor and a resource state regulation actor, which share the LDCA state ${\mathcal{S}}_k^L(t)$ and realize two-layer interaction by receiving allocation instructions from the GCA.

Subtask scheduling actor (${\mu }_k^{{\theta }^{L,st}}$): This actor receives the tasks allocated by the GCA and determines the subtask zone allocation and execution time scheduling.

Input Layer: This dimension is consistent with that of the LDCA state space, with the input being $S_k^L(t)$.

Output Layer:

Zone allocation branch: This dimension is $N^L\times M_k$ (where $N^L$ is the maximum number of parallel subtasks of the LDCA and $M_k$ is the number of zones), which is activated to output the probability of subtasks being allocated to each zone.

Start execution time branch: This dimension is $N^L\times 1$, with the output normalized to a continuous execution time within $[0,1)$.

Resource state regulation actor (${\mu }_k^{{\theta }^{L,re}}$): Based on the subtask scheduling results, this actor dynamically adjusts the core and frequency states of zone servers as well as the cooling set temperature.

Input Layer: This layer shares the LDCA state vector with the subtask scheduling actor, with the input being ${\mathcal{S}}_k^L(t)$.

Output Layer: This dimension is $M_k\times 3$, which outputs the action increment.

4.2.3. Global Centralized Critic Network

The global centralized critic network $Q^{\varphi }$ is used to integrate the joint state and action information of the two-layer agents, evaluate the global decision value, and provide gradient signals for the actor networks.

Input Layer: This dimension is the sum of the dimensions of the joint state and joint action, with the input being $\mathrm{S}(t)=[{\mathcal{S}}^G(t),{\mathcal{S}}_1^L(t),\dots ,{\mathcal{S}}_K^L(t)]$ and $\mathrm{A}(t)=[{\mathcal{A}}^G(t),{\mathcal{A}}_1^{L,st}(t),{\mathcal{A}}_1^{L,re}(t),\dots ,{\mathcal{A}}_K^{L,st}(t),{\mathcal{A}}_K^{L,re}(t)]$.

Output Layer: This dimension is 1, activated to output the global Q-value $Q(t,\mathrm{S}(t),\mathrm{A}(t))$, which represents the expected cumulative reward under the joint state–action pair.

5. Case Study

5.1. Parameter Settings

The stochastic elements include workload arrivals, electricity prices, and ambient temperature, which are modeled based on real-world data traces and standard stochastic processes [43,44,45]. This paper sets up three heterogeneous DCs, and the configuration parameters of each DC are provided in

Table 3. Zone parameters of each DC.

	Number of Zones	Zone Heat Capacity (kWh/°C)	Zone Thermal Resistance (°C/kW)	Zone Peak Frequency (GHz)	Number of Zone Cores
DC1	4	[8000, 7200, 6800, 7200]	[0.02, 0.02, 0.02, 0.02]	[3.8, 3.6, 3.5, 3.7]	[3200, 2800, 2500, 3000]
DC2	3	[3500, 3200, 2400]	[0.04, 0.04, 0.04]	[3.0, 2.9, 3.1]	[1800, 1500, 1600]
DC3	2	[800, 700]	[0.08, 0.08]	[2.6, 2.5]	[800, 600]

. The electricity price and ambient temperature used during the testing phase are shown in Figure 3. In the assumptions of this study, long-term power purchase agreements for DCs are not considered. The algorithm parameters are presented in

Table 4. Parameters of the algorithm.

Parameter Name	Value
Learning rate	1 × 10−5
Discount factor	0.99
Batch size	128
Experience replay buffer capacity	106
Simulation steps per episode	96
Number of training episodes	1000

. All experiments are carried out under the two typical operation scenarios mentioned in the introduction, where DCs adjust their power consumption according to real-time electricity price signals and take part in demand response as valuable flexible resources.

To ensure the reproducibility of this case study, all key parameters are specified as follows. Electricity price traces are sourced from the U.S. Energy Information Administration (EIA) for three ISO regions (PJM, CAISO, and ERCOT) covering January to December 2024, while ambient temperature data are obtained from the NOAA Global Historical Climatology Network (GHCN-Daily) for three corresponding cities (Chicago, Los Angeles, and Houston) with hourly averaging. Workload arrivals follow a Poisson process with a mean rate of 120 tasks per hour, split among the three DCs proportionally to their core counts. Task computing demand follows a lognormal distribution with a mean of 500 core·GHz·ms and a standard deviation of 200. The number of subtasks per task is uniformly distributed between one and eight, and the DAG structure of each task is generated using the Gaussian elimination method with a dependency density of 0.3, based on the Alibaba Cluster Trace V2018. The bandwidth capacity between any two DCs is set to 1 Gbps (symmetric), with a communication cost coefficient of 0.005 $/MB and an SLA violation penalty coefficient of 50 $/violation. For the cooling system, the COP fitting coefficients are calibrated from a real CRAC unit as $a=6.0$, $b=0.02$, $c=0.1$, and $d=5.0$; the rated fan power per zone is 15 kW, and the safe indoor temperature range is set to 18 °C, 27 °C. All stochastic processes are initialized with a random seed of 42, and each experimental configuration is run over 10 independent trials with results reported as mean values.

5.2. Decision Result Analysis

To verify the superiority of the method proposed in this paper, comparative experiments were designed for result analysis:

Case 1: The proposed method.

Case 2: Double-layer scheduling without physical regulation. The hierarchical multi-agent architecture is retained, focusing on the global spatiotemporal optimization of computing tasks while physical parameters are fixed.

Case 3: Single-layer local scheduling. The GCA and the inter-DC migration mechanism are removed, and the LDCA only optimizes the locally arrived tasks, without any physical parameter regulation.

Case 4: Rule-based static scheduling. After the arrival of tasks, they are evenly allocated to each region inside the DC and executed immediately.

Table 5. Comparison of multidimensional indices under four cases.

	SLA Violation Rate/%	Average Overtime Duration/min	Total Cost/$	PUE
Case 1	0	0	15,741.23	1.60/1.50/1.47
Case 2	0	0	23,403.37	1.80/1.51/1.60
Case 3	0	0	24,229.59	1.73/1.64/1.47
Case 4	0	0	24,667.14	1.73/1.64/1.47

presents the SLA violation rate, average overtime duration, total cost, and PUE under the four cases. First, all four cases completed the computing tasks without violating the SLA, which indicates that the designed scheduling methods can meet the basic service requirements. Second, in terms of total cost, compared with Case 4, Case 3, Case 2, and Case 1 achieved cost reductions of 1.77%, 5.12%, and 36.19%, respectively. This shows that reducing costs can be achieved through time delay of computing tasks, DC spatial migration, and adjustment of physical equipment parameters. However, the effect of different methods varies significantly in terms of reduction magnitude. Compared with Case 4, Case 3 only delays computing tasks in time and migrates them between regions within the DC, enabling task execution to avoid peak electricity price periods and optimize the utilization of regional resources. On the basis of Case 3, Case 2 enables cross-DC task migration. Due to the significant differences in electricity prices and temperatures between different DCs, migrating computing tasks to DCs with low electricity prices and low temperatures can prevent a sharp increase in cooling costs caused by excessive IT equipment power, thereby reducing the total cost. Case 1 further adjusts the cores and frequencies of AS on the basis of Case 2. By combining the spatiotemporal adjustment of computing tasks and the adjustment of physical equipment operating parameters, Case 1 achieves the lowest cost, fully reflecting the advantages of its collaborative hierarchical architecture. Finally, in terms of PUE values, Cases 2–4 do not adjust the operating parameters of physical equipment and only involve spatiotemporal migration of computing tasks, resulting in higher PUE values. In contrast, Case 1 can adjust AS and cooling equipment, which not only reduces the total power but also effectively restricts cooling energy consumption, thereby lowering the PUE.

Figure 4 shows the task execution status under the four cases. Each color in the figure represents one DC, and a darker color indicates a larger number of tasks being executed. In Case 4, after receiving computing tasks, they are evenly allocated to each region for execution with zero delay. Tasks are concentrated in the early stage of the cycle, and the number of tasks is dense during the peak electricity price period. However, the number of executing tasks is small during the low electricity price period after 21:00, which leads to high electricity costs. In Case 3, tasks are executed with delay and migrated between regions within the DC, and most computing tasks are executed during the low electricity price period (00:00–08:00). However, due to the large number of arriving tasks, the small capacity of DC3, and its high electricity price and temperature, there is almost no possibility of delay, and a large number of computing tasks are executed throughout the scheduling cycle, resulting in limited adjustment capability of this strategy. In Case 2, a large number of tasks are migrated to DC2, which has low electricity prices and low temperatures, and are concentrated in the low electricity price period (00:00–08:00) for execution. This reduces the electricity cost to a certain extent but leads to unbalanced load distribution. DC3 only executes fixed loads, resulting in serious core idleness. In contrast, Case 1 avoids load imbalance and achieves cost reduction through load balancing and physical equipment adjustment. The cores and frequency of AS match the tasks being executed, and the cooling power consumption is coordinated with the IT power consumption, ensuring that the temperature does not exceed the limit.

Figure 5 shows the total power of each DC under the four cases. It can be observed that, regardless of the DC, the total power in Case 1 is the lowest. Case 1 reduces costs by combining the spatiotemporal adjustment of computing tasks and the adjustment of physical equipment parameters. In Case 2, due to the excessive migration of computing tasks to DC2, the power of DC2 is extremely high, while DC3 always operates with a fixed load, resulting in extremely low power. This leads to a high degree of load imbalance. The power of Case 3 is similar to that of Case 4, but Case 3 avoids excessive load during peak periods by delaying tasks, such as DC3, during 08:00–10:00 and 18:00–21:00.

Figure 6 presents the core occupancy rate of servers within each DC under the four cases. It is evident that the core occupancy rate in Case 1 is concentrated between 60% and 90%. This range not only reduces idle cores to lower IT power consumption but also ensures that there are standby cores to cope with the sudden arrival of new tasks, achieving a balance between resource utilization and service stability. In Case 2, DC1 and DC3 are allocated a small number of tasks, but all cores remain active, resulting in low core occupancy rates and significant resource waste. In Case 3 and Case 4, the core occupancy rate is related to the number of arriving tasks. In this experiment, the number of computing tasks arriving at the three DCs is set to be nearly the same. However, due to differences in DC capacity, the core occupancy rate of DC1 is the lowest, followed by DC2, and DC3 has the highest core occupancy rate.

Figure 7 shows the box plots of indoor temperature in each DC under the four cases. The effect of different scheduling strategies on indoor temperature can be clearly observed. The temperature distribution of the three DCs under Case 1 is the most concentrated and stable, with the core range focusing on 23–25 °C and the smallest average interquartile range. By dynamically adapting the operating parameters of cooling equipment, it can respond to real-time changes in IT power consumption, effectively suppress temperature fluctuations, and achieve precise temperature control. In contrast, Cases 2–4 exhibit obvious shortcomings in temperature regulation performance. While their temperatures in DC1 and DC3 fall within a reasonable range, DC2 suffers from excessively low temperatures, leading to unnecessary waste of cooling power consumption. This is because such strategies do not dynamically adjust the parameters of cooling equipment; the fan speed is fixed at a high level during natural cooling periods. When the heat generation of IT equipment is mismatched with cooling capacity, over-cooling becomes prominent. Notably, the temperature of DC2 in Case 2 is slightly higher than that in Cases 3 and 4. The core reason is that Case 2 allocates more computing tasks to DC2, and the increase in IT power consumption partially offsets the impact of over-cooling, resulting in a relative rise in temperature.

5.3. Sensitivity Analysis

5.3.1. Differences in Regulation Results Under Different Seasons

Different seasons require distinct regulation strategies for DCs. This paper analyzes the regulatory differences across three seasonal categories: spring/autumn, summer, and winter. Figure 8 depicts the total cost and PUE of the four scenarios under three seasonal conditions, which intuitively reflects the adaptability of each scheduling strategy to seasonal changes. The horizontal axis of the figure represents different seasons, while the vertical axis is divided into two parts to respectively display the total cost and PUE, realizing the synchronous comparison of the two key indicators. In terms of seasonal variation characteristics, summer has the highest total cost and PUE in all four cases, followed by spring/autumn, and winter has the lowest. This is mainly because the high ambient temperature in summer increases the load of the cooling system, leading to a significant rise in cooling power consumption and further increasing the total cost and PUE. In winter, the low ambient temperature allows most DCs to adopt natural cooling, which greatly reduces cooling energy consumption, thus reducing the total cost and PUE.

Figure 9 illustrates the power consumption of each DC under different seasons. From the perspective of cooling power, the proportion of cooling power in the total power is the highest in summer, followed by spring/autumn, and the lowest in winter. This difference stems from the dominant role of seasonal ambient temperature in cooling modes: the outdoor temperature in summer is significantly high, so each DC needs to operate mechanical cooling throughout the day, and the continuous operation of compressors leads to a surge in cooling power consumption. In contrast, the outdoor temperature in winter is low, and natural cooling can be achieved only through fan-driven heat exchange between indoor and outdoor environments without starting compressors, resulting in a substantial reduction in cooling power consumption. In terms of IT power allocation, the IT power in spring/autumn and summer is evenly distributed among the three DCs. Given that the ambient temperature is moderate in spring/autumn and the cooling load is already high in the hot summer environment, excessive concentration of IT power in a single DC would lead to a sudden increase in local IT power consumption, which in turn triggers a synchronous rise in cooling energy consumption and ultimately drives up the total cost. However, the IT power allocation in winter adopts a differentiated strategy: since winter can fully rely on natural cooling, there is no need to worry about the surge in cooling power consumption caused by excessively high IT power. Therefore, most tasks are prioritized to DC1 and DC2 with lower electricity prices, while DC3, with the highest electricity price, only undertakes a small number of tasks. Through the differentiated configuration of electricity prices, the total cost is significantly reduced.

Figure 10 presents the box plots of server state regulation of each DC under different seasons, which focuses on reflecting the regulation effect on server state and adaptability to seasonal changes. Each subgraph corresponds to one DC, and the box plots of active core ratio and frequency scaling factor are distinguished by different colors. In summer, each DC tends to reduce the active core ratio and frequency scaling factor to minimize additional power consumption. This is reflected in the fact that the median frequency scaling factor of the three DCs is the lowest in summer, and the median active core ratio of DC1 and DC2 is also the lowest in summer. Notably, the active core ratio and frequency scaling factor of DC2 are consistently higher than those of DC1 and DC3. The core reason is that DC2 has a lower electricity price, allowing it to appropriately activate more cores and increase the operating frequency to ensure fast task execution and avoid task accumulation or timeout.

Figure 11 shows the variation curve of the indoor temperature of each DC under different seasons. In spring/autumn, the indoor temperature of each DC is stably maintained at 23–26 °C; in summer, the indoor temperature ranges from 24 to 27 °C, showing a slight upward trend overall; and in winter, the indoor temperature is the lowest, ranging from 18 to 25 °C, and exhibits a slow downward trend throughout the whole period. The indoor temperature in winter is the lowest and shows an overall downward trend. This is because the outdoor ambient temperature in winter is significantly low, and the proportion of natural cooling duration in each DC is greatly increased. It is only necessary to drive heat exchange between indoor and outdoor environments through fans to maintain the temperature above the safe lower limit without additional mechanical heating assistance. The temperature in spring/autumn is the most stable. The core reason is that the outdoor ambient temperature in this season is moderate, which is highly consistent with the optimal operating temperature range of DCs. The cooling system does not need to frequently switch operating modes, and natural cooling combined with short-term mechanical cooling can balance the heat generation of IT equipment. The temperature in summer shows an upward trend. This is because the outdoor high temperature lasts for a long time in summer, and each DC needs to operate mechanical cooling for an extended period. The outdoor temperature reaches its peak from noon to evening, resulting in a decrease in the heat dissipation efficiency of the cooling system. The imbalance between cooling load and heat generation load causes the indoor temperature to gradually rise, showing an obvious upward trend.

5.3.2. Impact of Learning Rate on Training Results

The learning rate is a step-size coefficient for network updates, which affects the training convergence efficiency and stability of the model. By changing its value, the impacts on the convergence, stability of the training process, and the final regulation effect are studied. Figure 12 shows the impact of the learning rate on the training process. It can be found that convergence can be achieved at around 700 iterations under different learning rates, but the reward after convergence is the maximum when the learning rate is 1 × 10⁻⁵.

5.4. Computational Complexity and Deployment Feasibility

This section discusses the online inference time, scalability, and real-time deployment feasibility of the proposed H-MADDPG algorithm.

Online decision-making time: The inference time of the trained H-MADDPG policy is dominated by forward passes through the actor networks. On a standard server (Intel Xeon Gold 6248, 2.5 GHz, 32 GB RAM), the average inference time per time step is approximately 23 ms for the GCA and 15 ms for each LDCA, totaling less than 80 ms for three DCs. Since each time step in the simulation corresponds to 15 min in real operation, this inference overhead is negligible and well within real-time control constraints.

Scalability: The hierarchical decomposition reduces the joint action space from exponential to polynomial complexity, making the framework inherently easy to scale. Specifically, the GCA handles only high-level task allocation while each LDCA independently manages its local subtask scheduling and equipment regulation. This divide-and-conquer design avoids the exponential growth of the joint action space and allows the system to accommodate additional DCs or zones without fundamentally altering the decision structure. In practice, the framework exhibits strong operational feasibility for geo-distributed data center deployments of realistic scale.

Real-time deployment feasibility: During execution (decentralized phase), each agent makes decisions based solely on its local observations, requiring no online communication with the centralized critic or other agents. The GCA can run on a central controller, while each LDCA runs on an edge server located with its corresponding DC. The lightweight inference makes real-time control feasible. However, practical deployment on physical hardware would require additional validation against hardware delays, measurement noise, and occasional communication faults, as acknowledged in the limitations section.

6. Conclusions

This paper proposes a computational–physical collaborative optimization model for geographically distributed DCs based on hierarchical reinforcement learning. By integrating spatiotemporal task migration, adaptive adjustment of server core count and frequency, and mode-switchable cooling control, the proposed H-MADDPG algorithm effectively bridges the gap between global coordination and local autonomy. The experimental results demonstrate that the proposed strategy reduces the total cost by 36.19%, lowers PUE to 1.47–1.60, maintains a 0% SLA violation rate, and achieves balanced resource utilization and stable temperature control compared to benchmark schemes.

Despite the promising results, several limitations must be acknowledged to frame the contribution realistically. First, this study is entirely simulation-based and has not been validated on a real DC control platform, where hardware delays, measurement noise, and communication faults may affect performance. Second, the cooling system employs a lumped thermal parameter model, and servers are abstracted as AS models, which simplify spatial temperature gradients and transient thermal dynamics. Third, while this paper notes that long-term power purchase agreements (PPAs) are not considered, this is explicitly acknowledged here as a limitation, as PPAs could significantly affect optimal migration decisions under stable electricity prices. Finally, the current framework assumes a single operator and does not address multi-operator coordination or data privacy concerns.

Based on these limitations, we propose the following specific research directions:

(1)

Real-platform validation: Deploy the H-MADDPG algorithm on a small-scale experimental DC testbed (e.g., with 3–5 servers and controllable cooling units) to evaluate its real-time feasibility, robustness to communication delays, and generalization to non-stationary workloads.

(2)

Refined cooling and server modeling: Replace the aggregated thermal model with zonal computational fluid dynamics (CFD) surrogate models or physics-informed neural networks (PINNs) to capture spatial temperature distributions and transient responses.

(3)

Long-term PPA integration: Formulate a two-timescale optimization framework where PPAs are signed on a monthly/yearly basis (upper level) and real-time task migration is optimized on an hourly basis (lower level), with contract compliance constraints.

(4)

Privacy-preserving multi-operator coordination: Develop a federated multi-agent reinforcement learning (Fed-MADDPG) framework that enables DCs of different operators to coordinate without sharing local workload or cooling state data.

(5)

Stress condition evaluation: Conduct a systematic evaluation under stressed operating conditions, including tighter deadlines (e.g., 30% reduction), heavier workloads (e.g., doubled arrival rate), and limited inter-DC bandwidth, to further validate the robustness of the hierarchical framework.

(6)

Carbon-aware co-optimization: Incorporate real-time carbon intensity signals into the reward function to achieve a joint reduction in electricity cost and carbon footprint, supporting green DC operations.

In summary, this work establishes a systematic and extensible hierarchical RL framework for DC energy optimization. The explicit discussion of limitations and corresponding concrete future directions provides a realistic foundation for transitioning from simulation-based research to practical deployment.