A Carbon-Efficient Framework for Deep Learning Workloads on GPU Clusters

Abstract

The explosive growth of artificial intelligence (AI) services has led to massive scaling of GPU computing clusters, causing sharp rises in power consumption and carbon emissions. Although hardware-level accelerator enhancements and deep neural network (DNN) model compression techniques can improve power efficiency, they often encounter deployment barriers and risks of accuracy loss in practice. To address these issues without altering hardware or model architectures, we propose a novel Carbon-Aware Resource Management (CA-RM) framework for GPU clusters. In order to minimize the carbon emission, the CA-RM framework dynamically adjusts energy usage by combining real-time GPU core frequency scaling with intelligent workload placement, aligning computation with the temporal availability of renewable generation. We introduce a new metric, performance-per-carbon (PPC), and develop three optimization formulations: carbon-constrained, performance-constrained, and PPC-driven objectives that simultaneously respect DNN model training deadlines, inference latency requirements, and carbon emission budgets. Through extensive simulations using real-world renewable energy traces and profiling data collected from NVIDIA RTX4090 GPU running representative DNN workloads, we show that the CA-RM framework substantially reduces carbon emission while satisfying service-level agreement (SLA) targets across a wide range of workload characteristics. Through experimental evaluation, we verify that the proposed CA-RM framework achieves approximately 35% carbon reduction on average, compared to competing approaches, while still ensuring acceptable processing performance across diverse workload behaviors.

1. Introduction

Recent explosive growth in artificial intelligence (AI)-driven applications and services has led to a rapid expansion of graphics processing unit (GPU) clusters, the essential infrastructure for processing large-scale AI workloads [1]. Consequently, the energy consumption of GPU clusters has also increased at an exponential rate. Although GPU device architectures continue to improve in terms of performance-per-watt (PPW), the continuous growth in computational demand has far outpaced these efficiency gains, resulting in a continuous rise in the total energy consumed by modern GPU clusters. For example, NVIDIA’s Blackwell-based GPU architecture such as RTX5090, released in late 2024, features a total graphics power (TGP) of approximately 575W [2]. This level of consumption is roughly 1.5 times higher than that of recent high-performance CPUs such as the Intel Xeon Platinum 8480+ (350W TDP) [3] and AMD EPYC 9654 (360W TDP) [4]. According to [5], global data center electricity consumption is expected to exceed 945TWh by 2030, and this level corresponds to roughly 3% of global electricity demand. In particular, the energy demand of AI-related processing will be approaching 35% of overall data center energy consumption in 2030 [6].

Rising energy consumption leads to an increase in carbon emissions. According to a recent global assessment report, the average carbon intensity of electricity generation in 2023 was approximately 480 $\mathrm{g}{\mathrm{CO}}_2\mathrm{e}/\mathrm{Wh}$, indicating that each additional unit of electricity consumption proportionally contributes to carbon emissions [7]. This continued growth is critical because ${\mathrm{CO}}_2$ emissions directly accelerate global warming. Empirical climate–carbon cycle analyses suggest that every additional 1000 GtCO₂ emitted is associated with roughly 0.45 °C of long-term global temperature increase [8]. To address these problems, the Paris Agreement (2015) established a global climate framework aiming to keep the rise in global mean temperature well below 2 °C above pre-industrial levels while pursuing efforts to limit warming to 1.5 °C [9]. Under this framework, each Party must submit a Nationally Determined Contribution (NDC), a national emissions-reduction pathway and carbon budget, and is subject to increasing economic, political, and regulatory pressure if it fails to meet these commitments.

Given the constrained global carbon budgets and the rapid growth of AI workloads, reducing the carbon emissions associated with GPU-based AI processing has become essential. Conventional approaches for lowering carbon emissions in GPU clusters generally fall into three categories: (1) reducing the energy consumption of accelerator hardware through improved architecture-level design, (2) modifying the structure of deep neural network (DNN) models to reduce computational and energy demands, and (3) conducting scheduling-based energy- and carbon-aware GPU cluster management. From the perspective of architecture-level design, Google reports that its Cloud TPU v4 delivers approximately 2.7× higher performance-per-watt (PPW) compared to TPU v3, with typical chip power in the range of 200–300 W [10]. Similarly, FuriosaAI’s NPUs (Warboy and Renegade/RNGD) demonstrate over 2× higher inference PPW and more than 60% greater energy efficiency than comparable NVIDIA GPU-based systems for LLM inference—corresponding to roughly 40–50% energy savings for equal throughput [11]. From the perspective of NN model optimization, techniques such as model pruning, quantization, and knowledge distillation can significantly reduce energy consumption while maintaining acceptable accuracy. For example, training the LLaMA-7B model requires approximately 36 MWh of electricity, whereas LLaMA-65B requires around 449 MWh, meaning the 7B model consumes only about 8% of the training energy while still achieving sufficient performance for many downstream tasks [12]. From the perspective of energy- and carbon-aware scheduling, ref. [13] proposes an energy-optimized cluster management scheme for clusters containing heterogeneous GPU devices. They design the GPU utilization prediction method and GPU heterogeneity-aware energy consumption model. Based on the Alibaba trace, they validated that their work improves energy saving by about 35%. Ref. [14] proposes a carbon-efficient machine learning job scheduler that enables the job moving to greener hours so as to decrease the carbon emission. Their proposed approach improves the carbon emission reduction by about 40% with negligible performance degradation.

However, the strategies described above remain challenging in practice. Hardware design and fabrication for new AI accelerators typically require long development cycles, substantial capital investment, and specialized manufacturing processes. As a result, newly developed devices cannot be immediately or universally deployed across existing GPU-based infrastructures, which often operate under strict compatibility and reliability constraints [15]. Similarly, DNN model compression techniques reduce parameter counts and representational capacity, and therefore may introduce non-trivial accuracy degradation, particularly when applied with aggressive compression ratios or on complex, fine-grained downstream tasks [16,17]. Thus, compression must be carefully balanced against application-specific accuracy, robustness, and latency requirements rather than treated as a universally applicable solution for carbon or energy reduction. Prior studies on energy- and carbon-aware scheduling for GPU clusters do not explicitly consider both training and inference workloads under time-varying renewable energy generation. Moreover, their proposed formulations typically lack sophisticated constraints for enforcing predefined service-level agreements (SLAs) across heterogeneous user requests. As a result, these approaches are not readily applicable to general-purpose GPU-based AI data centers.

To address these challenges, in this paper, we propose a novel Carbon-Aware Resource Management (CA-RM) framework for AI workload processing on GPU clusters, focusing on the carbon emission factor and maximizing the utilization of low-carbon renewable electricity sources, without hardware re-design and DNN model compression. The carbon emission factor represents the amount of carbon dioxide emitted per unit of electricity generated. It is worth noting that even the same GPU energy usage can lead to very different carbon footprints depending on whether the marginal electricity comes from renewable (low-carbon emission factor) sources like wind farm and solar photovoltaics or from fossil (high-carbon emission factor) sources such as coal, oil, or natural gas [18]. Table 1 shows the carbon emission factors for each energy source.

Table 1. Carbon emission factors ($\mathrm{g}{\mathrm{CO}}_2\mathrm{e}/\mathrm{Wh}$) [18].

Coal	Wind	Solar Photovoltaics	Geothermal	Diesel	Natural Gas (LNG)
0.96	0.034	0.05	0.038	0.778	0.611

In our CA-RM framework, we utilize two methodologies for the reduction of carbon emission: the GPU core frequency scaling and computing worker allocation. The GPU core frequency scaling dynamically tunes the GPU core operating frequency for inference service processing in real time, which directly adjusts GPU power draw, processing performance, and resulting energy consumption [19]. Note that GPU core frequency scaling does not require any non-negligible control overheads and most modern GPU devices deploy the scaling instruction set within themselves. Based on these findings, our CA-RM framework uses frequency scaling as an energy-shifting lever; when renewable generation is scarce, we lower the GPU core clock to reduce instantaneous consumption and thus minimize brown-energy usage; when renewable generation is abundant, we raise the clock to increase consumption and maximize renewable utilization. This allows the cluster’s energy demand to better track renewable supply over time. In the computing worker allocation process, we decide how many computing workers are assigned to DNN model training jobs and when they begin to process the allocated jobs. When renewable generation is low, we reduce the number of parallel workers or postpone the training jobs to later time slots and when renewable generation is high, we increase the number of workers and immediately process the assigned training jobs. Note that the training jobs are batch workload, not like inference tasks as transactional workload, and they are deferrable within the predefined deadline. Combined to GPU core frequency scaling for inference service processing, the computing worker allocation enables the adaptive energy-shifting corresponding to renewable generation behaviors.

This paper makes the following contributions.

• We propose a Carbon-Aware Resource Management (CA-RM) framework for GPU clusters that process AI workloads. The framework explicitly targets minimizing operational carbon emissions by maximizing the utilization of low-carbon renewable electricity, while still meeting performance objectives such as training deadline and inference latency bound.
• To quantify the carbon usage efficiency, we introduce the novel metric called performance-per-carbon (PPC) for both DNN model training jobs and inference tasks. By using the PPC metric, we can compare our proposed CA-RM framework to other methods quantitatively and verify the practicalness of our work. Furthermore, we deploy the carbon budget constraints to our formulation so as to achieve the acceptable carbon emission under various workload patterns.
• We utilize both real-time GPU core frequency scaling and computing worker allocation approaches that adapt GPU workers’ power draw and energy use to temporal changes in renewable generation. In order to minimize the carbon emission while guaranteeing the performance requirement of AI workloads, we present the sophisticated mathematical formulation that explicitly controls the carbon emission, DNN model training time, and inference service latency. In particular, we design carbon-constrained, performance-constrained, and PPC-driven optimization approaches in our CA-RM framework so as to achieve various targets under the various problems described.
• To demonstrate the performance of our proposed CA-RM framework, we utilize the real renewable generation traces of wind farms and solar photovoltaics, and retrieve the profiling data from an NVIDIA RTX4090 device with well-known DNN models such as vgg16, inceptionv4, resnet152, and stochasticdepth101. Based on the raw data, we conduct an extensive simulation experiment and validate that our CA-RM framework is functional given various workload patterns.

Section 2 depicts the structure of our proposed CA-RM framework. In Section 3, we present the system model including performance, energy consumption, and carbon emission models for DNN model training and inference. Section 4 formulates the optimization problems involving carbon emission constraints, performance constraints, PPC-aware objectives, and the resource management algorithm. Section 5 shows the raw profiling data of the RTX4090 device with DNN models and various evaluation results of our CA-RM framework over its competitors. We summarize and conclude our work in Section 6.

2. Proposed Framework Structure

Our proposed Carbon-Aware Resource Management (CA-RM) framework is illustrated in Figure 1.

Figure 1. The structure of the proposed Carbon-Aware Resource Management (CA-RM) framework.

(1)

Electricity Market: The electricity market provides the fossil-based grid energy, denoted as $r^c\left(t\right)$. This component enables the CA-RM framework to purchase fossil energy when the renewable supply is insufficient. The renewable generation signal is periodically updated and incorporated into the CA-RM framework decision-making process to achieve carbon-efficient operation.

(2)

Renewable Energy Generators: This module produces green energy from renewable generation sources such as wind farms denoted as $r^w\left(t\right)$ and solar photovoltaics $r^{pv}\left(t\right)$. This energy is first directed to satisfy the operational demand of the power delivery system (PDS) and is then used to supply the training cluster and the inference sub clusters. Corresponding to dynamic $r^w\left(t\right)$ and $r^{pv}\left(t\right)$, the CA-RM framework decides the amount of fossil energy to be purchased, $r^c\left(t\right)$, and supplies the required power to each cluster.

(3)

Training Cluster: The training cluster executes multiple DNN model training jobs that are typically energy-intensive and long-duration. Its operation is coordinated by the parallel worker allocation plan $n\left(t\right)$ and time slot assignment plan $x\left(t\right)$. The CA-RM framework may defer or slow down training jobs during high-carbon periods to align energy consumption with low-carbon availability.

(4)

Inference Cluster: The inference cluster consists of multiple sub-clusters $(1,2,\dots ,H)$ designed to handle real-time inference requests. Because inference workloads are latency-sensitive, this module prioritizes responsiveness while still allowing for a limited frequency scaling plan $f\left(t\right)$. The CA-RM framework dynamically assigns inference tasks among sub-clusters by involving DNN models. Through sub-cluster partitioning, the inference infrastructure achieves both scalability and fine-grained energy control.

(5)

Carbon-Aware Resource Management Framework: The Carbon-Aware Resource Management (CA-RM) framework integrates DNN workload shifting and carbon emission management. It consists of two functional components: Tuning Manager and Optimization. The Tuning Manager is responsible for GPU core frequency scaling, parallel worker allocation, and time slot assignment. It aligns cluster operation with renewable availability and carbon intensity signals. For optimization, we deploy the carbon-constrained, performance-constrained, and PPC-driven formulations in the framework. According to each objective, our framework adaptively tunes the carbon emission by adjusting $f\left(t\right)$, $n\left(t\right)$, and $x\left(t\right)$ given workload characteristics.

3. System Model

In this section, we present the system model that captures job processing performance, cluster-level energy consumption, and carbon emissions associated with both renewable and non-renewable energy sources. We denote by $f^{Tr}\left(t\right)$ and $f_i^{\mathit{Inf}}\left(t\right)$ the GPU core frequency of the GPU training cluster, and the i-th sub-inference cluster, which corresponds to service level i at time slot t, respectively. In addition, we denote $\delta$ as the duration of a time slot.

For simplicity, our framework focuses on tuning only the GPU core frequency while excluding the GPU memory frequency from the decision variables. This design is motivated by the observation that the dynamic power consumption of GPU core processing units dominates total GPU power, while the memory subsystem contributes a relatively smaller fraction, depending on workload characteristics [20]. In addition, the power–performance behavior induced by GPU memory frequency scaling is qualitatively similar to that of GPU core frequency scaling: increasing the memory frequency leads to higher power consumption and improved performance, analogous to core frequency scaling. Therefore, explicitly including GPU memory frequency as a decision variable provides limited additional optimization benefits while significantly increasing the complexity of the overall optimization problem. To reduce this complexity, we fix the GPU memory frequency at its maximum value and optimize only the GPU core frequency throughout this study.

3.1. Performance Model for Inference Tasks

The hardware specifications and the GPU core frequency values directly influence the service rate for continuously arriving inference requests. Thus, we are able to dynamically tune the service latency of the inference tasks by using the GPU core frequency scaling. Following the models in [21,22], the service latency of the arbitrary inference task in the i-th sub-inference cluster at time slot t, $L_i\left(t\right)$ is defined as follows:

$$l_i\left(t\right)={\displaystyle \frac{P_{Q=1}}{{\rho }_i·m_i·f_i^{\mathrm{Inf}}\left(t\right)-{\lambda }_i\left(t\right)}}+{\displaystyle \frac{1}{{\rho }_i·f_i^{\mathrm{Inf}}\left(t\right)}}+e_i\left(t\right),\forall i,t,$$

(1)

Here, ${\lambda }_i\left(t\right)$ and $e_i\left(t\right)$ represent the arrival rate of the inference requests to the i-th sub-inference cluster and the corresponding network transmission delay at time slot t, respectively. We assume that all workers remain busy, and thus the queue-related probability term $P_Q$ is always 1. $m_i$ denotes the number of available workers in the i-th sub-inference cluster. In this paper, we do not dynamically adjust $m_i$; instead, these values are predetermined by the cluster operator. Note that in order to reduce the energy consumption of idle workers, we are able to implement the power-proportional data center through changing $m_i$. Interested readers may refer to the concept of the dynamic right sizing (DRS) method in [23]. The parameter ${\rho }_i$ denotes a performance coefficient corresponding to the hardware specifications and the model characteristics of inference requests to the i-th sub-inference cluster. The value of ${\rho }_i$ differs across inference service categories, which are involved with the DNN model, and clusters due to heterogeneity in both software characteristics and GPU hardware configurations.

In order to penalize the degradation of the inference service latency, we present the penalty function $F^{\mathrm{Inf}}$. The functions $F^{\mathrm{Inf}}$ and $f_i^{\mathrm{Inf}}\left(t\right)$ are defined as follows:

$$\begin{array}{c}{F}^{\mathrm{Inf}}=\sum _{\forall t}\sum _{\forall i}{F}_i^{\mathrm{Inf}}\left(t\right),\hfill \end{array}$$

(2)

$$\begin{array}{c}{F}_i^{\mathrm{Inf}}\left(t\right):=f_i^{\mathrm{Inf}}(l_i\left(t\right)-\widehat{l_i}):={\omega }_i^{\mathrm{Inf}}·\max \{l_i\left(t\right)-\widehat{l_i},0\},\forall i,t.\hfill \end{array}$$

(3)

where $\widehat{l_i}$ and ${\omega }_i^{\mathrm{Inf}}$ denote the bound of the inference service latency and the weighted value to draw the priority of the inference tasks in the i-th sub-inference cluster, respectively. Note that $f_i^{\mathrm{Inf}}$ is the monotonic non-increasing function that linearly penalizes the violation of the service latency bound. The function value $f_i^{\mathrm{Inf}}$ is zero when $l_i\left(t\right)\ge \widehat{l_i}$, as shown in the right term of (3).

3.2. Performance Model for Training Jobs

Training jobs are considered batch workloads, where the overall completion time is more critical than response latency. This characteristic contrasts with inference tasks, which operate as transactional workloads [21]. Therefore, the delayed or deferrable scheduling is acceptable for training jobs, allowing the system to select appropriate time slots for executing each job as long as the predefined training deadline is met. In particular, we aim to focus on determining the number of allocated workers and the time slots to which training jobs are assigned, in order to adaptively adjust the energy consumption according to the amount of renewable energy generated.

Let ${\tau }_j$ denote the number of completed epochs of the j-th training job within a time slot of fixed duration $\delta$. Following [24,25], we adopt a statistical approach to model the relationship between GPU frequency and per-epoch processing performance as

$${\tau }_j\left(t\right)={\displaystyle \frac{n_j\left(t\right)\times \delta }{\frac{{\alpha }_{j,0}^{\mathrm{Tr}}}{f^{\mathrm{Tr}}}+{\alpha }_{j,1}^{\mathrm{Tr}}}},\hspace{1em}\hspace{1em}\forall j.$$

(4)

where $n_j\left(t\right)$ represents the number of parallel GPU workers assigned to the j-th training job at time slot t, while ${\alpha }_{j,0}^{\mathrm{Tr}}$ and ${\alpha }_{j,1}^{\mathrm{Tr}}$ are scaling factors specific to the j-th training job’s characteristics. These scaling factors can be obtained through offline profiling. Note that $f^{\mathrm{Tr}}$ is always fixed as the maximum value, and we do not conduct frequency scaling for the training cluster in contrast to the case of the inference clusters.

Each training job typically requires hundreds of epochs to be trained to achieve a reasonable level of inference accuracy. Let $M_j$ and N denote the number of epochs to be necessarily trained for the j-th training job and the number of available workers in the GPU training cluster, respectively. We set $x_j\left(t\right)\in \{0,1\}$ as the decision variable, which represents whether the time slot t is assigned to j-th training job or not. Then the job completion time of the j-th training job, $h_j$, is defined as follows:

$$h_j={argmax}_t\left\{t\right|x_j\left(t\right)=1,\forall t\},\forall j.$$

(5)

This value must satisfy

$$\begin{array}{c}\hfill h_j\ge a_j,\forall j,\end{array}$$

(6)

$$\begin{array}{c}\hfill \sum _{\forall t}{\tau }_j\left(t\right)·x_j\left(t\right)\ge M_j,\forall j,\end{array}$$

(7)

$$\begin{array}{c}\hfill \sum _{\forall j}{n}_j\left(t\right)·x_j\left(t\right)\le N,\forall t.\end{array}$$

(8)

where $a_j$ denotes the arrival time slot of the j-th training job. Constraint (6) presents that the job completion time should be later than the job arrival time. Constraint (7) presents that the time slot assignment plan should cover the number of required epochs to be trained. Constraint (8) presents that the number of the parallelized workers for entire training jobs cannot exceed the total amount of available GPU resources.

Similar to (2), we define the performance penalty function for training jobs as follows:

$$\begin{array}{c}{F}^{\mathrm{Tr}}=\sum _{\forall j}{F}_j^{\mathrm{Tr}},\hfill \end{array}$$

(9)

$$\begin{array}{c}{F}_j^{\mathrm{Tr}}=f_j^{\mathrm{Tr}}(h_j-{\hat{h}}_j):={\omega }_j^{\mathrm{Tr}}·max\{h_j-{\hat{h}}_j,0\},\forall j.\hfill \end{array}$$

(10)

where ${\hat{h}}_j$ and ${\omega }_i^{\mathrm{Tr}}$ denote the deadline and the priority for the j-th training job, respectively. $f_j^{\mathrm{Tr}}$ is the penalty function for the j-th training job, similar to (2).

3.3. Energy Consumption Model for Inference Cluster

We provide the power consumption and the associated energy consumption for inference clusters based on the GPU utilization. Following the models presented in [26], the power consumption of the i-th sub-inference cluster, $p_i\left(t\right)$, is defined as follows:

$$\begin{array}{cc}{p}_i\left(t\right)\hfill & ={\alpha }_{i,0}^{\mathrm{Inf}}·f_i^{\mathrm{Inf}}\left(t\right)·u_i^{\mathrm{Inf}}\left(t\right)+{\alpha }_{i,1}^{\mathrm{Inf}}·f_i^{\mathrm{Inf}}\left(t\right)+{\alpha }_{i,2}^{\mathrm{Inf}}·u_i^{\mathrm{Inf}}\left(t\right)+{\alpha }_{i,3}^{\mathrm{Inf}},\forall i,t\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill & ={\alpha }_{i,0}^{\mathrm{Inf}}·{\lambda }_i^{\mathrm{Inf}}\left(t\right)+{\alpha }_{i,1}^{\mathrm{Inf}}·f_i^{\mathrm{Inf}}\left(t\right)+{\alpha }_{i,2}^{\mathrm{Inf}}{\displaystyle \frac{{\lambda }_i^{\mathrm{Inf}}\left(t\right)}{f_i^{\mathrm{Inf}}\left(t\right)}}+{\alpha }_{i,3}^{\mathrm{Inf}},\forall i,t.\hfill \end{array}$$

(12)

where ${\alpha }_{i,0}^{\mathrm{Inf}},{\alpha }_{i,1}^{\mathrm{Inf}},{\alpha }_{i,2}^{\mathrm{Inf}},{\alpha }_{i,3}^{\mathrm{Inf}}$ are model coefficients and can be derived by the offline profiling approach similar to (4). Here, $u_i^{\mathrm{Inf}}\left(t\right)={\displaystyle \frac{{\lambda }_i^{\mathrm{Inf}}\left(t\right)}{f_i^{\mathrm{Inf}}\left(t\right)}}$. In this model, we assume that all the workers in the sub-inference cluster are available and staying in the power-on state.

Based on (12), we present the energy consumption model of the i-th sub-inference cluster as follows:

$$\begin{array}{c}{E}^{\mathrm{Inf}}=\sum _{\forall i}{E}_i^{\mathrm{Inf}},\hfill \end{array}$$

(13)

$$\begin{array}{c}{E}_i^{\mathrm{Inf}}=\sum _{\forall t}{E}_i^{\mathrm{Inf}}\left(t\right),\forall i,\hfill \end{array}$$

(14)

$$\begin{array}{c}{E}_i^{\mathrm{Inf}}\left(t\right)=\delta \times p_i\left(t\right),\forall i,\hfill \end{array}$$

(15)

where $E_i^{\mathrm{Inf}}\left(t\right)$, $E_i^{\mathrm{Inf}}$ and $E^{\mathrm{Inf}}$ denote the energy consumption of the i-th sub-inference cluster at time slot t, the sum of the energy consumption of the i-th sub-inference cluster for all time slots, and the whole energy consumption of the entire inference clusters, respectively.

3.4. Energy Consumption Model for Training Cluster

To model the energy consumption of the GPU training cluster, we adopt the statistical modeling approach. Based on [25], we present the energy consumption for training jobs as follows:

$$\begin{array}{c}{E}^{\mathrm{Tr}}=\sum _{\forall j}{E}_j^{\mathrm{Tr}},\hfill \end{array}$$

(16)

$$\begin{array}{c}{E}_j^{\mathrm{Tr}}=\sum _{\forall t}{E}_j^{\mathrm{Tr}}\left(t\right),\forall j,\hfill \end{array}$$

(17)

$$\begin{array}{c}{E}_j^{\mathrm{Tr}}\left(t\right)=n_j\left(t\right)\times x_j\left(t\right)\times \delta \times ({\beta }_{j,0}^{\mathrm{Tr}}·f^{\mathrm{Tr}}+{\beta }_{j,1}^{\mathrm{Tr}}),\forall j,t.\hfill \end{array}$$

(18)

where ${\beta }_{j,0}^{\mathrm{Tr}}$ and ${\beta }_{j,1}^{\mathrm{Tr}}$ are power model coefficients corresponding to the j-th training job. $E_j^{\mathrm{Tr}}\left(t\right)$, $E_j^{\mathrm{Tr}}$ and $E^{\mathrm{Tr}}$ denote the energy consumption of the j-th training job at time slot t, the sum of the energy consumption of the GPU training cluster for all time slots, and the whole energy consumption of all training jobs, respectively. Note that if we do not assign a certain time slot to the training job (i.e., $x_j\left(t\right)=0$), then the involved energy consumption might be zero, as shown in (18). Obviously, we can shift the energy curves of the GPU training cluster by tuning the decision variables $n_j\left(t\right)$ and $x_j\left(t\right)$.

3.5. Carbon Emission Model

The amount of carbon emission depends on both the amount of energy consumption and what source the energy is generated from. The carbon emission factor is the metric that represents the carbon efficiency of the energy source. Fossil energy such as coal and natural gas has a higher carbon emission factor than renewable energy such as wind and solar, as shown in Table 1. Even within the same renewable energy category, this value might differ according to the detailed sources. Let ${\theta }^{\mathrm{c}}$, ${\theta }^{\mathrm{pv}}$ and ${\theta }^{\mathrm{w}}$ denote the carbon emission factors for coal-based, solar photovoltaics-based, and wind farm-based energy generation, respectively. Then the total carbon emission $c\left(t\right)$ at time slot t can be calculated as follows:

$$\begin{array}{c}\hfill c\left(t\right)=\sum _{\forall i}{c}_i^{\mathrm{Inf}}\left(t\right)+c^{\mathrm{Tr}}\left(t\right),\forall t,\end{array}$$

(19)

$$\begin{array}{c}\hfill c_i^{\mathrm{Inf}}\left(t\right)={\theta }^{\mathrm{pv}}·i{r}_i^{\mathrm{pv}}\left(t\right)+{\theta }^{\mathrm{w}}·i{r}_i^{\mathrm{w}}\left(t\right)+{\theta }^{\mathrm{c}}·i{r}_i^c\left(t\right),\forall i,t,\end{array}$$

(20)

$$\begin{array}{c}\hfill c^{\mathrm{Tr}}\left(t\right)={\theta }^{\mathrm{pv}}·t{r}^{\mathrm{pv}}\left(t\right)+{\theta }^{\mathrm{w}}·t{r}^{\mathrm{w}}\left(t\right)+{\theta }^{\mathrm{c}}·t{r}^c\left(t\right),\forall t,\end{array}$$

(21)

$$\begin{array}{c}\sum _{\forall i}i{r}_i^{\mathrm{pv}}\left(t\right)+t{r}^{\mathrm{pv}}\left(t\right)\le r^{\mathrm{pv}}\left(t\right),\forall t\hfill \end{array}$$

(22)

$$\begin{array}{c}\sum _{\forall i}i{r}_i^{\mathrm{w}}\left(t\right)+t{r}^{\mathrm{w}}\left(t\right)\le r^{\mathrm{w}}\left(t\right),\forall t\hfill \end{array}$$

(23)

$$\begin{array}{c}\sum _{\forall i}i{r}_i^{\mathrm{c}}\left(t\right)+t{r}^{\mathrm{c}}\left(t\right)\le r^{\mathrm{c}}\left(t\right),\forall t\hfill \end{array}$$

(24)

$$\begin{array}{c}\hfill \sum _{\forall i}{E}_i^{\mathrm{Inf}}\left(t\right)+\sum _{\forall j}{E}_j^{\mathrm{Tr}}\left(t\right)\le r^{\mathrm{pv}}\left(t\right)+r^{\mathrm{w}}\left(t\right)+r^c\left(t\right),\forall t.\end{array}$$

(25)

where $r^{\mathrm{pv}}\left(t\right)$, $r^{\mathrm{w}}\left(t\right)$ and $r^c\left(t\right)$ represent the amount of generated energy at solar photovoltaics, wind farm, and coal-based generators at time slots t. $i{r}_i^{\mathrm{pv}}\left(t\right)$, $i{r}_i^{\mathrm{w}}\left(t\right)$, and $i{r}_i^{\mathrm{c}}\left(t\right)$ which are solar photovoltaics energy, wind energy, and coal energy delivered to the i-th inference cluster. Similarly, $t{r}^{\mathrm{pv}}\left(t\right)$, $t{r}^{\mathrm{w}}\left(t\right)$, and $t{r}^{\mathrm{c}}\left(t\right)$ are delivered to the training cluster. Note that the sum of each delivered energy cannot exceed the amount of generated energy, as presented in (22)–(24). Constraint (25) presents that the whole energy consumption by both the GPU training cluster and inference clusters should be covered by the total energy generation at every time slot. If the energy amount by the renewable generators is not enough to cover the energy usage, then we should increase $r^c\left(t\right)$ accordingly.

In this paper, corresponding to the carbon emission, we propose a new metric called performance-per-carbon (PPC) that represents the efficiency of the processing capacity for DNN workloads over the generated carbon. We define the PPC for inference tasks and training jobs as follows, respectively:

$$\begin{array}{c}\hfill \overline{{\mathrm{PPC}}^{\mathrm{Inf}}}=\mathbb{E}\{\sum _{\forall i,t}{\mathrm{PPC}}_i^{\mathrm{Inf}}\left(t\right)\},\end{array}$$

(26)

$$\begin{array}{c}\hfill {\mathrm{PPC}}_i^{\mathrm{Inf}}\left(t\right)={\displaystyle \frac{l_i{\left(t\right)}^{-1}}{c_i^{\mathrm{Inf}}\left(t\right)}},\forall i,t,\end{array}$$

(27)

$$\begin{array}{c}\hfill \overline{{\mathrm{PPC}}^{\mathrm{Tr}}}=\mathbb{E}\{\sum _{\forall t}{\mathrm{PPC}}^{\mathrm{Tr}}\left(t\right)\},\end{array}$$

(28)

$$\begin{array}{c}\hfill {\mathrm{PPC}}^{\mathrm{Tr}}\left(t\right)={\displaystyle \frac{{\sum }_{\forall j}{x}_j\left(t\right)\times {\tau }_j\left(t\right)}{c^{\mathrm{Tr}}\left(t\right)}},\forall t.\end{array}$$

(29)

Here, $l_i{\left(t\right)}^{-1}$ denotes the inverse of the service latency for tasks executed in the i-th sub-inference cluster at time slot t, and thus corresponds to the throughput of inference processing. Consequently, ${\mathrm{PPC}}_i^{\mathrm{Inf}}\left(t\right)$ and ${\mathrm{PPC}}^{\mathrm{Tr}}\left(t\right)$ represent the number of inference tasks processed on the i-th sub-inference cluster and the number of training epochs completed on the training cluster per carbon emission at time slot t, respectively. $\overline{{\mathrm{PPC}}^{\mathrm{Inf}}}$ and $\overline{{\mathrm{PPC}}^{\mathrm{Tr}}}$ denote the average PPC values over whole sub-inference clusters and the training cluster, respectively. Note that the whole energy consumption by both the inference clusters and training cluster cannot exceed the limited carbon budget. Let B denote the carbon budget assigned to the clusters. Then, we define the carbon emission constraints as follows:

$$\begin{array}{c}\sum _{\forall t}c\left(t\right)\le B.\hfill \end{array}$$

(30)

Since achieving higher performance typically requires greater energy usage, the carbon budget can be interpreted as the available capacity for assigning computational performance. Note that even for the same amount of energy consumption, the amount of actual carbon emission can be different depending on the energy usage ratio of renewable energy generation. Therefore, under the fixed carbon budget, it is also essential to account for the carbon emission factor in addition to energy consumption. A lower carbon emission factor is more favorable, so it is desirable to schedule and execute as many tasks as possible during periods when renewable energy generation is high and the carbon emission factor is low.

4. Problem Formulation

The goal of our formulation is to minimize carbon emission by jointly optimizing (i) the number of parallel workers and time slot allocation for the training cluster, (ii) the GPU core frequency settings for the inference clusters, and (iii) the amount of energy procured from fossil-based generators, while ensuring acceptable performance for both training jobs and inference tasks. To this end, we present three classes of optimization formulations, the carbon-constrained problem, the performance-constrained problem, and the PPC-driven problem, as detailed in the following subsections.

4.1. Carbon-Constrained Optimization

In this formulation, we aim to simultaneously maximize the performance of both inference tasks and training jobs while ensuring that the total carbon budget is not violated. The problem is formulated as follows:

Problem 1.

Carbon-constrained optimization problem

$$\begin{array}{c}\hfill \underset{n,x,f,ir,tr,r^c}{min}{F}^{\mathit{Inf}}+F^{\mathit{Tr}},\end{array}$$

(31)

$$\begin{array}{cc}\hfill s.t.& h_j\ge a_j,\forall k,\hfill \end{array}$$

(32)

$$\begin{array}{c}\sum _{\forall t}{\tau }_j\left(t\right)·x_j\left(t\right)\ge M_j,\forall j,\hfill \end{array}$$

(33)

$$\begin{array}{c}\sum _{\forall j}{n}_j\left(t\right)·x_j\left(t\right)\le N,\forall t,\hfill \end{array}$$

(34)

$$\begin{array}{c}{\underline{f}}_i^{\mathit{Inf}}\le f_i^{\mathit{Inf}}\le {\overline{f}}_i^{\mathit{Inf}},\forall i,\hfill \end{array}$$

(35)

$$\begin{array}{c}\sum _{\forall i}i{r}_i^{\mathit{pv}}\left(t\right)+t{r}^{\mathit{pv}}\left(t\right)\le r^{\mathit{pv}}\left(t\right),\forall t,\hfill \end{array}$$

(36)

$$\begin{array}{c}\sum _{\forall i}i{r}_i^w\left(t\right)+t{r}^w\left(t\right)\le r^w\left(t\right),\forall t,\hfill \end{array}$$

(37)

$$\begin{array}{c}\sum _{\forall i}i{r}_i^c\left(t\right)+t{r}^c\left(t\right)\le r^c\left(t\right),\forall t,\hfill \end{array}$$

(38)

$$\begin{array}{c}\sum _{\forall i}{E}_i^{\mathit{Inf}}\left(t\right)+\sum _{\forall j}{E}_j^{\mathit{Tr}}\left(t\right)\le r^{\mathit{pv}}\left(t\right)+r^w\left(t\right)+r^c\left(t\right),\forall t,\hfill \end{array}$$

(39)

$$\begin{array}{c}\sum _{\forall t}c\left(t\right)\le B,\hfill \end{array}$$

(40)

$$\begin{array}{c}i{r}_i\left(t\right),tr\left(t\right),r^c\left(t\right)\ge 0,\forall i,t,\hfill \end{array}$$

(41)

$$\begin{array}{c}{n}_j\left(t\right)\in \mathcal{N},\forall j,t,\hfill \end{array}$$

(42)

$$\begin{array}{c}{x}_j\left(t\right)\in \{0,1\},\forall j,t.\hfill \end{array}$$

(43)

Here, ${\underline{f}}_i^{\mathrm{Inf}}$ and ${\overline{f}}_i^{\mathrm{Inf}}$ denote the lower bound and upper bound for frequency scaling of the inference clusters, respectively. The decision variables $r^c$ are automatically minimized depending on $r^{\mathrm{pv}}$, $r^{\mathrm{w}}$, and constraint (40). In Problem 1, we are able to maximize the processing performance by minimizing the objective terms, $F^{\mathrm{Inf}}$ and $F^{\mathrm{Tr}}$, if the carbon budget B is large enough.

4.2. Performance-Constrained Optimization

In this formulation, given predefined performance bounds in terms of service latency and training completion time, our objective is to minimize the carbon emission produced by the clusters. This formulation is applicable when it is not necessary to enforce a strict carbon budget. Even under identical performance levels, the resulting carbon emissions may vary depending on the performance-tuning strategy and job-assignment policy. The problem is formulated as follows:

Problem 2.

Performance-constrained optimization problem

$$\begin{array}{c}\underset{n,x,f,ir,tr,r^c}{min}\sum _{\forall t}c\left(t\right),\end{array}$$

(44)

$$\begin{array}{cc}\hfill s.t& h_j\ge a_j,\forall k,\hfill \end{array}$$

(45)

$$\begin{array}{c}\sum _{\forall t}{\tau }_j\left(t\right)·x_j\left(t\right)\ge M_j,\forall j,\hfill \end{array}$$

(46)

$$\begin{array}{c}\sum _{\forall j}{n}_j\left(t\right)·x_j\left(t\right)\le N,\forall t,\hfill \end{array}$$

(47)

$$\begin{array}{c}{\underline{f}}_i^{\mathit{Inf}}\le f_i^{\mathit{Inf}}\le {\overline{f}}_j^{\mathit{Inf}},\forall i,\hfill \end{array}$$

(48)

$$\begin{array}{c}\sum _{\forall i}i{r}_i^{\mathit{pv}}\left(t\right)+t{r}^{\mathit{pv}}\left(t\right)\le r^{\mathit{pv}}\left(t\right),\forall t,\hfill \end{array}$$

(49)

$$\begin{array}{c}\sum _{\forall i}i{r}_i^w\left(t\right)+t{r}^w\left(t\right)\le r^w\left(t\right),\forall t,\hfill \end{array}$$

(50)

$$\begin{array}{c}\sum _{\forall i}i{r}_i^c\left(t\right)+t{r}^c\left(t\right)\le r^c\left(t\right),\forall t,\hfill \end{array}$$

(51)

$$\begin{array}{c}\sum _{\forall i}{E}_i^{\mathit{Inf}}\left(t\right)+\sum _{\forall j}{E}_j^{\mathit{Tr}}\left(t\right)\le r^{\mathit{pv}}\left(t\right)+r^w\left(t\right)+r^c\left(t\right),\forall t,\hfill \end{array}$$

(52)

$$\begin{array}{c}{l}_i\left(t\right)\le {\hat{l}}_i,\forall i,t,\hfill \end{array}$$

(53)

$$\begin{array}{c}{h}_j\le {\hat{h}}_j,\forall j,\hfill \end{array}$$

(54)

$$\begin{array}{c}i{r}_i\left(t\right),tr\left(t\right),r^c\left(t\right)\ge 0,\forall i,t,\hfill \end{array}$$

(55)

$$\begin{array}{c}{n}_j\left(t\right)\in \mathcal{N},\forall j,t,\hfill \end{array}$$

(56)

$$\begin{array}{c}{x}_j\left(t\right)\in \{0,1\},\forall j,t.\hfill \end{array}$$

(57)

In contrast to Problem 1, in the formulation above, the inference latency and training completion time are considered tight constraints, as presented in (53) and (54). In this formulation, we assume that we are always able to find the feasible solution that enables avoiding violation of the performance constraints.

4.3. PPC-Driven Optimization

In this formulation, our goal is to maximize our novel PPC metric presented above, without consideration for performance bound and carbon budget limitations; thus, we are able to focus only on the carbon efficiency. For the training cluster, we can apply the delayed scheduling of training jobs toward the carbon-efficient time slots (i.e., when the amount of renewable energy generation is high). For the inference clusters, we can apply adaptive frequency scaling of computing workers in real time. To do this, the problem is formulated as follows:

Problem 3.

PPC-driven optimization problem

$$\begin{array}{cc}\hfill \underset{n,x,f,ir,tr,r^c}{min}& {\omega }^{\mathit{Inf}}·\overline{{\mathit{PPC}}^{\mathit{Inf}}}+{\omega }^{\mathrm{Tr}}·\overline{{\mathit{PPC}}^{\mathit{Tr}}},\hfill \end{array}$$

(58)

$$\begin{array}{cc}\hfill s.t.& h_j\ge a_j,\forall k,\hfill \end{array}$$

(59)

$$\begin{array}{c}\sum _{\forall t}{\tau }_j\left(t\right)·x_j\left(t\right)\ge M_j,\forall j,\hfill \end{array}$$

(60)

$$\begin{array}{c}\sum _{\forall j}{n}_j\left(t\right)·x_j\left(t\right)\le N,\forall t,\hfill \end{array}$$

(61)

$$\begin{array}{c}{\underline{f}}_i^{\mathit{Inf}}\le f_i^{\mathit{Inf}}\le {\overline{f}}_j^{\mathit{Inf}},\forall i,\hfill \end{array}$$

(62)

$$\begin{array}{c}\sum _{\forall i}i{r}_i^{\mathit{pv}}\left(t\right)+t{r}^{\mathit{pv}}\left(t\right)\le r^{\mathit{pv}}\left(t\right),\forall t,\hfill \end{array}$$

(63)

$$\begin{array}{c}\sum _{\forall i}i{r}_i^w\left(t\right)+t{r}^w\left(t\right)\le r^w\left(t\right),\forall t,\hfill \end{array}$$

(64)

$$\begin{array}{c}\sum _{\forall i}i{r}_i^c\left(t\right)+t{r}^c\left(t\right)\le r^c\left(t\right),\forall t,\hfill \end{array}$$

(65)

$$\begin{array}{c}\sum _{\forall i}{E}_i^{\mathit{Inf}}\left(t\right)+\sum _{\forall j}{E}_j^{\mathit{Tr}}\left(t\right)\le r^{\mathit{pv}}\left(t\right)+r^w\left(t\right)+r^c\left(t\right),\forall t,\hfill \end{array}$$

(66)

$$\begin{array}{c}i{r}_i\left(t\right),tr\left(t\right),r^c\left(t\right)\ge 0,\forall i,t,\hfill \end{array}$$

(67)

$$\begin{array}{c}{n}_j\left(t\right)\in \mathcal{N},\forall j,t,\hfill \end{array}$$

(68)

$$\begin{array}{c}{x}_j\left(t\right)\in \{0,1\},\forall j,t.\hfill \end{array}$$

(69)

Here, ${\omega }^{\mathrm{Inf}}$ and ${\omega }^{\mathrm{Tr}}:=1-{\omega }^{\mathrm{Inf}}$ denote the processing priority for inference and training, respectively. Note that in contrast to Problems 1 and 2, this problem does not include the terms for bounds like (40), (53), and (54). If the carbon budget is high enough and the performance bound is loose, then this formulation can optimize both the carbon efficiency and the processing performance for DNN workloads, simultaneously.

4.4. DNN-Based Surrogate Optimizer

Note that Problems 1–3 are all nonlinear and non-convex due to terms such as the multiplication of decision variables in (7) and (8), as well as the reciprocal of decision variables in (27). To address this challenge, we adopt a meta-heuristic approach, which enables approximate optimization for non-convex formulations. In particular, we employ a variant of the DNN-based surrogate optimizer presented in our previous work [27]. The DNN-based surrogate optimizer enables deriving the sufficiently approximated optimal solution even for large-scale non-convex problems by simply transforming the original constrained problems to unconstrained ones. For example, we define the unsupervised DNN loss function based on Problem 1 as follows:

$$\begin{array}{c}{\mathrm{\Theta }}_{\mathrm{carbon}}\left(\mathbf{v}\right)=F^{\mathrm{Inf}}(\mathbf{s}=DNN(\mathbf{v};\mathbf{w}))+F^{\mathrm{Tr}}(\mathbf{s}=DNN(\mathbf{v};\mathbf{w}))+{\Phi }_{\mathrm{carbon}}\left(\mathbf{s}\right).\hfill \end{array}$$

(70)

where

$$\begin{array}{cc}\hfill {\mathrm{\Phi }}_{\mathrm{carbon}}(\mathbf{s})& ={\varphi }_d^{\mathrm{carbon}}\sum _{\forall j}{(a_j-h_j)}^{+}+{\varphi }_M^{\mathrm{carbon}}\sum _{\forall j}{(\sum _{\forall t}{M}_j-{\tau }_j(t)·x_j(t))}^{+}\hfill \end{array}$$

(71)

$$\begin{array}{c}+{\varphi }_N^{\mathrm{carbon}}\sum _{\forall t}{(\sum _{\forall j}{n}_j(t)·x_j(t)-N)}^{+}\hfill \end{array}$$

(72)

$$\begin{array}{c}+{\varphi }_{\underline{f}}^{\mathrm{carbon}}\sum _{\forall i}{({\underline{f}}_i^{\mathrm{Inf}}-f_i^{\mathrm{Inf}})}^{+}+{\varphi }_{\overline{f}}^{\mathrm{carbon}}\sum _{\forall i}{(f_i^{\mathrm{Inf}}-{\overline{f}}_i^{\mathrm{Inf}})}^{+}\hfill \end{array}$$

(73)

$$\begin{array}{c}+{\varphi }_{r^{\mathrm{pv}}}^{\mathrm{carbon}}\sum _{\forall i}{(i{r}_i^{\mathrm{pv}}\left(t\right)+t{r}^{\mathrm{pv}}\left(t\right)-r^{\mathrm{pv}}\left(t\right))}^{+}\hfill \end{array}$$

(74)

$$\begin{array}{c}+{\varphi }_{r^{\mathrm{w}}}^{\mathrm{carbon}}\sum _{\forall i}{(i{r}_i^{\mathrm{w}}\left(t\right)+t{r}^{\mathrm{w}}\left(t\right)-r^{\mathrm{w}}\left(t\right))}^{+}\hfill \end{array}$$

(75)

$$\begin{array}{c}+{\varphi }_{r_1^{\mathrm{c}}}^{\mathrm{carbon}}\sum _{\forall i}{(i{r}_i^{\mathrm{c}}\left(t\right)+t{r}^{\mathrm{c}}\left(t\right)-r^{\mathrm{c}}\left(t\right))}^{+}\hfill \end{array}$$

(76)

$$\begin{array}{c}+{\varphi }_e^{\mathrm{carbon}}\sum _{\forall t}{(\sum _{\forall i}{E}_i^{\mathrm{Inf}}(t)+\sum _{\forall j}{E}_j^{\mathrm{Tr}}(t)-(r^{\mathrm{pv}}(t)+r^{\mathrm{w}}(t)+r^c(t)))}^{+}\hfill \end{array}$$

(77)

$$\begin{array}{c}+{\varphi }_b^{\mathrm{carbon}}{(\sum _{\forall t}c(t)-B)}^{+}\hfill \end{array}$$

(78)

$$\begin{array}{c}+{\varphi }_{ir}^{\mathrm{carbon}}\sum _{\forall i,t}{(-i{r}_i\left(t\right))}^{+}\hfill \end{array}$$

(79)

$$\begin{array}{c}+{\varphi }_{tr}^{\mathrm{carbon}}\sum _{\forall t}{(-tr\left(t\right))}^{+}\hfill \end{array}$$

(80)

$$\begin{array}{c}+{\varphi }_{r_2^{\mathrm{c}}}^{\mathrm{carbon}}\sum _{\forall t}{(-r^c\left(t\right))}^{+},\hfill \end{array}$$

(81)

and ${\varphi }_d^{\mathrm{carbon}}-{\varphi }_{r_2^{\mathrm{c}}}^{\mathrm{carbon}}$ are penalty weight values for constraint violation. $\mathrm{v},\mathrm{w}$ and $\mathrm{s}$ denote the input data (i.e., user requests and renewable generation) to the DNN model weight parameters and output data (i.e., decision variables $n,x,f,ir,tr,r^c$), respectively. Here, ${\left(x\right)}^{+}=\max (x,0)$. The DNN loss functions based on Problems 2 and 3 can be derived in the same way presented above. Interested readers may refer to [27] for further details.

4.5. Proposed Algorithm

Algorithm 1 presents the overall procedures for deriving the optimal decision variables n, x, f, $ir$, $tr$, and $r^c$ for each target problem. Given model coefficients and constants, our framework efficiently finds the approximate decision variables corresponding to target formulations.

Algorithm 1: Optimization of the CA-RM Framework

INPUT: Model coefficients and constants $\rho$, m, e, a, $\omega$, $\delta$, ${\alpha }^{\mathrm{Tr}}$, ${\alpha }^{\mathrm{Inf}}$, ${\beta }^{\mathrm{Tr}}$, M, N, B, $\underline{f}$, $\overline{f}$, $\varphi$; Performance bound for inference, $\hat{l}$ and for training, $\hat{h}$; Carbon emission factors $\theta$; OUTPUT: Optimal decision variables, n, x, f, $ir$, $tr$, $r^c$. STEPS: 1 Predict the renewable generation $r^{\mathrm{pv}}$ and $r^{\mathrm{w}}$. 2 Select the target formulation among Problems 1–3. 3 Find the approximate optimal solution for the target formulation by using the DNN-based surrogate optimizer [27]. 4 Return the optimal solution $n^{*}$, $x^{*}$, $f^{*}$, $i{r}^{*}$, $t{r}^{*}$, $r^{c*}$.

5. Experiments

5.1. Cluster and Workload Setup

To conduct extensive simulation-based experiments, we empirically collected real processing data for DNN model training and inference from the computing node equipped with the NVIDIA RTX4090 GPU. The detail of the node is shown in Table 2. We deployed four well-known DNN models, vgg16, inceptionv4, resnet152, and stochasticdepth101, and executed both training jobs and inference tasks for each model, with the dataset CIFAR-100 [28]. The details of each DNN model are shown in Table 3. To execute DNN model training jobs and inference tasks, we set the computing node with CUDA 13.0, cuDNN 9.15, and PyTorch 2.9.0+cu130. We measured the training and inference time, as well as the associated energy consumption, under various GPU core clock configurations. To systematically adjust the GPU core clock values, we utilized the NVIDIA Management Library (NVML) [29] and implemented the Python (3.14)-based scripts that automate the frequency scaling process, thereby overriding the default GPU frequency behavior. We conducted GPU frequency scaling over the 5five available values for the RTX4090 device (210 MHz, 810 MHz, 1410 MHz, 2010 MHz, 2760 MHz). After configuring each device, our customized logging modules continuously recorded instantaneous power draw, training time, and inference latency throughout the experiments.

Table 2. Specification of the RTX4090-based computing node.

CPU	Mainboard	RAM	DISK	OS	GPU
i9-13900KF (32 cores)	Z790-P WIFI	DDR5 64 GB	SHPP41-1000 GM 1 TB	Windows10	RTX4090 (GDDR6X 24 GB, 16,384 CUDA cores)

Table 3. DNN model specification.

Info	VGG16 [30]	Inceptionv4 [31]	Resnet152 [32]	Stochasticdepth101 [33]
Architecture	Plain CNN	Inception	Residual CNN	Residual CNN
Depth (Layers)	16	75	152	101
Parameters	∼138 M	∼43 M	∼60 M	∼44 M
Use Cases	Feature Extraction	Image Classification	Image Recognition	Robust CNN Training

Using the empirical testbed results obtained from our RTX4090-based node, we conduct an extensive simulation to compare the proposed CA-RM framework with previous studies. The simulated GPU cluster consists of 100 RTX4090-based nodes (50 nodes in the training cluster and remain 50 nodes in the inference clusters). The time slot duration is set to 15 min, and the total simulation period is 24 h. The generated DNN workload behaviors are grouped into three categories: light workloads (inceptionv4 0.1, resnet152 0.1, stochasticdepth101 0.2, vgg16 0.6), medium workloads (inceptionv4 0.2, resnet152 0.2, stochasticdepth101 0.2, vgg16 0.4), and heavy workloads (inceptionv4 0.4, resnet152 0.4, stochasticdepth101 0.1, vgg16 0.1). The deadlines for training and latency bounds for inference are configured according to the characteristics of each DNN model. For renewable energy modeling, we use the practical data obtained from [34]. In particular, we incorporate solar irradiance, ambient temperature, and wind speed data collected during 24 h on 1 October 2018, from the NREL-M2 site in Boulder, CO, USA.

5.2. Evaluation Metric

To quantitatively evaluate the performance of the proposed CA-RM framework, we define the following metrics:

(1)

Training Completion Time and Inference Throughput: The training completion time represents the total time required to complete the assigned number of training epochs. For each deployed DNN model, we obtain the training time using our Python-based parsing scripts. The inference throughput measures the number of inference requests processed per unit time. We use one minute as the evaluation interval. The inverse of inference throughput corresponds to the service latency experienced by inference requests.

(2)

Service Level Agreement (SLA) Violation: This metric evaluates the reliability of our CA-RM framework with respect to both training and inference processing performance. Note that when a predefined SLA (e.g., service latency bound and training deadline) must be satisfied, maximizing performance is not always necessary. Instead, it is desirable to maintain performance at the level required to meet the SLA while reducing operational costs such as energy consumption and carbon emission.

(3)

Energy Consumption: This metric quantifies the energy consumed during DNN model training and inference processing. Although power consumption is generally inversely related to processing performance, energy usage does not always draw a simple behavior. The configuration that minimizes energy consumption varies depending on node specifications, DNN model architecture, and the characteristics of the input dataset.

(4)

Carbon Emission and Efficiency: The metric of carbon emission measures the amount of carbon emitted during DNN workload processing on GPU clusters. The carbon emission is closely tied to energy usage, but even with identical energy consumption, the resulting emissions differ depending on the energy source. Maximizing the utilization of renewable energy is therefore essential to reducing the carbon emission. The carbon efficiency is based on the proposed PPC metrics defined in (27) and (29), and is used to evaluate how effectively the framework minimizes the carbon emission while achieving acceptable processing performance. Higher PPC values indicate greater carbon efficiency. However, note that maximizing PPC does not inherently guarantee adherence to a given carbon budget. To ensure that the carbon budget is not violated, Problem 1 must be applied, potentially at the cost of reduced carbon efficiency.

5.3. Competitors

In order to verify the performance of our CA-RM framework, we compare our proposed approach to two previous studies and two baselines. The first previous method, EnergyCost [21], presents the deep learning job (containing training and inference) allocation considering both the electricity price and the heterogeneity of jobs. This method considers the dynamic electricity price from the grid market and conducts adaptive available time slot allocation accordingly. In addition, it adopts the dynamic right sizing (DRS) approach that turns the idle servers off, in order to avoid the waste of the unnecessary power usage of clusters. This method exploits the mixed-integer nonlinear problem (MINLP) formulation so as to delicately control the trade-off between the processing performance and energy usage cost. Although this approach shows noticeable energy saving while guaranteeing acceptable DNN model processing performance, it cannot explicitly consider the carbon emission associated with the energy usage. It does not conduct budget-based performance throttling like ours, so it cannot avoid unacceptable carbon emission from the cluster. The second previous method, PowerCap [25], tries to optimize the DNN model training performance without the violation of the predefined power budget, through real-time-based GPU core and memory frequency scaling based on economic model predictive control (EMPC). It utilizes recursive least square-based model estimation to capture the proper model coefficients for frequency scaling without time consuming offline profiling processes. In particular, this method adopts a distributed optimization structure so as to enable timely control even for hundreds of GPU servers. However, this method only considers the model training and not the inference tasks that are not deferrable compared to model training jobs that are allowed to be delayed within the deadline. Additionally, this method only focuses on power consumption, and not energy consumption and involved carbon emission that are formulated in nonlinear forms cannot be easily solved.

As baseline approaches, we conduct the analysis under the baseline-MIN and baseline-MAX. In the baseline-MIN, we fix the GPU core clocks to the minimum values for all the DNN workloads. Regardless of the deadline and latency bound, GPU core clocks are fixed during the time slots. This method enables the minimization of power consumption derived by GPU devices, but cannot ensure optimized energy consumption. When the DNN model training time significantly increases, the energy consumption may be undesirably increased. In the baseline-MAX, we fix the GPU core clocks to the maximum values for all the DNN workloads. In contrast to the baseline-MIN, this method maximizes the power consumption of GPU devices and the processing capacity as well. It enables maximizing the processing performance, but does not ensure optimized energy consumption and involved carbon emission due to the unacceptably high power consumption. Note that the time slot allocation decision-making process presented in Algorithm 1 is not applied to the baseline-MIN and baseline-MAX approaches. Instead, in these baseline approaches, user requests are allocated sequentially upon arrival to available workers and the nearest available empty time slots.

5.4. Evaluation Results

Figure 2a,b depict the DNN model training completion time and the inference throughput, corresponding to the change of GPU core frequency values on an RTX4090-based node. Obviously, the processing performance improves as the GPU core clock increases since the GPU processing capacity also increases. It is worth noting that the performance variation due to GPU frequency scaling differs significantly depending on the computational characteristics of DNN models. For instance, as shown in Figure 2a, the inceptionv4 model takes over 220 s to process a single epoch at the lowest GPU core clock (210 MHz), whereas the time drops to under 100 s at the highest clock (2760 MHz), resulting in a performance gain of more than 50%. In contrast, the vgg16 model shows a smaller improvement—from 20 s to 13 s—across the same clock range, reflecting only about a 35% increase in performance. This discrepancy across models is primarily determined by the proportion of GPU core operations in the overall training computation, that is, the compute intensity of the model. Compared to inceptionv4 (about 41 M parameters), vgg16 (about 34 M parameters) is a smaller model, and a greater portion of its execution time is attributed to CPU operations and data transfer between the host and GPU memory, rather than massively parallel computations on the GPU. As a result, increasing the GPU core clock has a more limited impact on the overall training performance. The inference throughput results in Figure 2b also reflect this pattern.

Figure 2. Performance and energy consumption of RTX4090-based node to process DNN models: vgg16, inceptionv4, resnet152, and stochasticdepth101.

We can see the energy consumption for both training and inference processing over GPU core clock configurations in Figure 2c,d. In contrast to performance results, the energy usage curves are shown in somewhat differently complex manners. The energy consumption curve for training shown in Figure 2c does not exhibit a monotonic trend with respect to the GPU core clock. Notably, the minimum energy usage occurs at a core clock of 2010 MHz, not at the minimum one (210 MHz). Although both power consumption and processing performance decrease monotonically as the core clock is reduced, the resulting energy consumption does not follow the same monotonic pattern. This discrepancy arises because energy consumption is defined as the product of power and processing time, and the rate at which power decreases and the rate at which processing time increases under lower core clock settings may differ. When the degradation in performance (i.e., the increase in processing time) outweighs the power reduction, the total energy consumption can in fact increase. Consequently, merely lowering the core clock does not guarantee improved energy efficiency; delicate profiling is required to find the accurate optimal frequency values. The energy consumption curve for inference in Figure 2d exhibits a trend similar to that observed in Figure 2c. Across all evaluated DNN models, the optimal GPU core clock that yields the lowest energy consumption is 2010 MHz. In contrast, both the lowest clock setting (210 MHz) and the highest setting (2760 MHz) result in noticeably inferior energy efficiency compared with other intermediate clock configurations.

Figure 3a–d present the average carbon emissions generated during the training of each DNN model under various energy source configurations. We vary the proportion of coal, wind, and solar photovoltaics energy contributing to the total energy usage. For instance, the configuration (c(0.8), w(0.1), pv(0.1)) indicates that 80% of the required energy is supplied by coal, 10% by wind, and 10% by solar photovoltaics. A higher proportion of wind and solar photovoltaics corresponds to a greater share of renewable energy generation. Across all models, we observe a consistent trend: carbon emissions decrease as the renewable energy contribution increases. For vgg16, carbon emissions are reduced from 0.08 gCO₂e/Wh under the configuration (c(0.8), w(0.1), pv(0.1)) to 0.015 gCO₂e/Wh under the configuration (c(0.1), w(0.3), pv(0.6)), an approximately 80% reduction. Similar levels of reduction are observed for the other models, including inceptionv4, resnet152, and stochasticdepth-101. These findings suggest that substantial improvements in carbon efficiency can be achieved solely by shifting the timing of training jobs or inference tasks to periods with higher renewable energy availability, without requiring any modifications to GPU hardware efficiency or DNN model architectures. Figure 4a–d depict the average carbon emissions associated with inference tasks under the same energy source configurations, showing similar trends to those in Figure 3.

Figure 3. Carbon emission of RTX4090-based node for DNN model Training.

Figure 4. Carbon emission of RTX4090-based node for DNN model inference.

We compare our proposed CA-RM framework to previous methods presented earlier in view of carbon emission and efficiency (PPC), as shown in Figure 5. Figure 5a shows the carbon emission of two baseline approaches (MAX and MIN), two existing methods, EnergyCost [21] and PowerCap [25], and our proposed approaches including carbon-constrained, performance-constrained, and PPC-driven optimization. The unit of relative carbon emissions in Figure 5a is normalized with respect to the carbon emissions produced by the baseline-MAX approach given heavy workloads. Specifically, a unit value of 1 corresponds to 154,688 gCO₂e/Wh. For light workloads, our proposed carbon-constrained approach achieves the best performance among all methods, as it focuses exclusively on adhering to the carbon budget. The performance-constrained and PPC-driven approaches exhibit similar outcomes because the workloads are not computationally intensive, and the resulting carbon emissions remain relatively low. Under such conditions, these methods can maximize the processing performance without incurring significant carbon emission. The EnergyCost method also performs favorably compared with our CA-RM framework, as it reduces energy cost by prioritizing the use of available renewable energy. This strategy indirectly leads to lower carbon emissions. In contrast, the PowerCap method yields carbon inefficiency because it focuses on limiting instantaneous power consumption rather than total energy consumption, which makes it ineffective at reducing cumulative carbon emissions. The two baseline methods, MAX and MIN, perform unfavorably from a carbon efficiency perspective. The baseline-MIN approach, which applies the minimum GPU core clock, significantly increases training time and inference latency, thereby raising total energy consumption. The baseline-MAX approach uses the maximum GPU core clock setting, which leads to substantially higher power consumption while offering only marginal improvements in DNN processing performance.

Figure 5. Comparison of the proposed CA-RM framework to EnergyCost [21], PowerCap [25], and baselines in views of relative carbon emission and carbon efficiency (PPC).

For medium workloads, the experimental results exhibit carbon emission patterns that are consistent with those observed under light workloads. However, compared with the light workload case, both the PPC-constrained and performance-constrained approaches generate substantially higher carbon emissions. This increase arises because the larger volume of tasks leads to a higher absolute amount of energy consumption. The results produced by the EnergyCost method under medium workloads remain similar to those for light workloads. Since the EnergyCost method consistently aims to minimize energy cost by maximizing the utilization of renewable energy, its carbon emissions also tend to decrease correspondingly. Under medium workloads, the PowerCap method applies stricter power capping than in the light workload case due to the heavier processing demand. While this stricter capping reduces instantaneous power consumption, it also increases inference latency and training completion time, making the associated carbon emission non-negligible.

For heavy workloads, the carbon-constrained approach continues to achieve the lowest carbon emissions because it strictly adheres to the predefined carbon budget, even at the expense of DNN processing performance. Although the performance-constrained approach also attempts to reduce carbon emissions using (44), it results in higher emission than the carbon-constrained method since maintaining the performance bound is treated as the highest priority during resource allocation. The PPC-driven approach also yields the high emission because it focuses solely on optimizing the PPC metric defined in (58) and does not explicitly account for the absolute carbon emission level or the available carbon budget. Note that under heavy workload conditions, a large number of user requests must be processed within a short time window, which inevitably requires increasing the processing capacity of computing nodes. Higher energy consumption and increased carbon emissions are unavoidable in such cases. Unlike the carbon-constrained approach whose primary objective is to strictly limit absolute carbon emissions, the PPC-driven approach is designed to maximize performance per carbon emission rather than minimizing total carbon output. From this perspective, an increase in carbon emissions under heavy workloads when using the PPC-driven approach is an expected outcome. Both the EnergyCost and PowerCap methods lead to relatively high carbon emissions under heavy workloads, as they operate on large volumes of DNN jobs and inference tasks that inherently require substantial energy consumption. Among all approaches, the baseline-MAX method performs the worst in terms of carbon emission because it consistently triggers maximum power consumption for all training jobs and inference tasks. The baseline-MIN method performs better than baseline-MAX, as operating at the minimum GPU core clock reduces power consumption and thus lowers total energy usage. Note that, across all workloads, the proposed approaches achieve an average carbon emission of approximately 60,000 gCO₂e/Wh. This corresponds to an improvement of 35% compared to all other competing methods, and an improvement of 28% relative to the PowerCap and EnergyCost approaches. Figure 5b compares the carbon efficiency quantified using the PPC values in (27) and (29), across our proposed approaches and the other competitors. Obviously, the PPC-driven approach consistently achieves the highest carbon efficiency for all workload cases because it focuses only on optimizing the PPC value without explicitly considering performance constraints or carbon budget limits. From the perspective of carbon efficiency, the carbon-constrained approach outperforms the performance-constrained approach. This is because the carbon-constrained approach directly limits the absolute carbon emissions through a predefined carbon budget, whereas the performance-constrained approach prioritizes meeting the required DNN processing performance bounds, even at the cost of incurring higher carbon emissions. Both the EnergyCost and PowerCap methods exhibit relatively high carbon emissions, as neither explicitly accounts for carbon efficiency in their optimization objectives. The baseline-MAX and baseline-MIN methods show the lowest carbon efficiency regardless of the workload characteristics, since they rely on fixed-frequency settings and therefore lack any adaptive carbon-aware tuning capability.

Figure 6a–c present the average SLA violation of each approach for both training and inference processing across the three workload characteristics. As shown in Figure 6a, all approaches except baseline-MIN satisfy the performance bounds under the light workload. The performance-constrained approach results in near-zero SLA violations because it explicitly enforces both the training deadline and the inference latency bound. The carbon-constrained approach also effectively prevents SLA violations, as the assigned carbon budget is sufficiently large to accommodate the DNN workloads. The PPC-driven approach exhibits a small amount of SLA violation due to the absence of an explicit performance constraint, focusing solely on maximizing the PPC metric. The baseline-MAX method, as expected, incurs no SLA violations because it consistently maintains the maximum GPU core clock configuration across all combinations of renewable energy availability and model types. In contrast, baseline-MIN produces the worst SLA performance, as fixing the GPU to the minimum core clock leads to substantial performance degradation across all cases. Figure 6b,c reveal results consistent with those in Figure 6a. Under medium and heavy workloads, our proposed approaches, except for the performance-constrained method, show SLA violation levels comparable to those of the EnergyCost and PowerCap methods (approximately 15% and 30%, respectively). This stems from the fact that the carbon-constrained and PPC-driven approaches prioritize reducing carbon emissions or improving carbon efficiency, even at the cost of certain performance degradation. As shown in Equations (31)–(43), the carbon-constrained formulation does not explicitly impose constraints on DL job performance requirements (i.e., the training deadline $\hat{h}$ and inference latency bound $\hat{l}$). Instead, this formulation explicitly enforces compliance with a predefined carbon budget, as expressed in Equation (40), and subsequently seeks to maximize processing performance (i.e., $F^{\mathrm{Inf}}$ and $F^{\mathrm{Tr}}$) within the feasible carbon budget range. In contrast, as shown in Equations (44)–(57), the performance-constrained formulation explicitly incorporates the DL job performance requirements as hard constraints, and then minimizes carbon emissions $c\left(t\right)$ within the feasible performance range. Under light workload conditions, the performance requirements are relatively modest, allowing the carbon-constrained approach to achieve low SLA violation even under a limited carbon budget. However, under medium and heavy workloads, satisfying stricter performance requirements inevitably leads to increased energy consumption and carbon emissions. As a result, the approaches that do not explicitly enforce SLA constraints, except for the performance-constrained formulation, can incur substantial SLA violations. This issue can be mitigated by increasing the carbon budget B in the carbon-constrained formulation. In the PPC-driven formulation, we can redesign the PPC metric in Equations (27) and (29) by adding the exponent coefficient to the $c\left(t\right)$ of the denominator (e.g., ${\left\{c\left(t\right)\right\}}^2$) so as to differentiate each weight value to performance and carbon emission. We will address this topic in future work. Nevertheless, it is important to note that our proposed approaches do not incur greater performance degradation than the existing methods. Overall, the results in Figure 6 demonstrate that our approaches within the CA-RM framework are available for GPU cluster management, effectively balancing carbon emission reduction with acceptable DNN processing performance.

Figure 6. Service-level agreement (SLA) violation given (light, medium, and heavy) workload characteristics.

6. Conclusions

In this paper, we present a novel Carbon-Aware Resource Management (CA-RM) framework maximizing the utilization of renewable energy, in order to achieve carbon-efficient GPU cluster operation with DNN model processing. We introduce a new metric called performance-per-carbon (PPC) so as to quantitatively evaluate the carbon efficiency. We utilize the GPU core frequency scaling and allocation planning of computing nodes as energy-shifting methodologies adaptive to dynamically renewable generation behaviors. To achieve the delicate tuning of both carbon emission and DNN workload processing performance simultaneously, we design the carbon-constrained, performance-constrained, and PPC-driven optimization formulations based on sophisticated mathematical modeling. To validate the availability of the CA-RM framework, we retrieved the profiling data from the NVIDIA RTX4090-based node with well-known DNN models, vgg16, inceptionv4, resnet152, and stochasticdepth101, and conducted extensive simulation experiment. Through the experimental results, we demonstrate that our proposed CA-RM framework outperforms recent existing methods for GPU cluster management. The CA-RM framework achieves carbon reduction of more than 35% on average, compared to its competitors while guaranteeing acceptable DNN workload processing performance. We verify that our work is proper for recent modern sustainable AI data centers.

We note that, despite the experimental results demonstrating that the CA-RM framework achieves carbon-efficient deep learning job allocation, the scope of this work is limited in terms of model generality. Specifically, the proposed CA-RM framework is designed for conventional deep learning models, such as DNNs and CNNs, and is not directly applicable to LLMs, including Transformer-based neural networks. Transformer-based models employ deeply stacked and recursive layer structures that differ fundamentally from those of traditional DNNs and CNNs. As a result, accurately capturing their performance and carbon emission characteristics would require redesigning the underlying system models used in our framework. Extending the proposed CA-RM framework to support state-of-the-art LLM workloads is an important direction for future work, and we plan to investigate such extensions in subsequent studies. In addition, we focus exclusively on validating the carbon efficiency of the CA-RM framework implemented in-house, and we do not consider its integration with general-purpose cluster management frameworks. As for future research, we plan to extend the CA-RM framework to be deployed as a plug-in for widely adopted cluster management frameworks, such as Kubernetes and Slurm, and to experimentally evaluate its performance and practicality in real-world cluster environments.