Joint Optimization of Container Resource Defragmentation and Task Scheduling in Queueing Cloud Computing: A DRL-Based Approach

Abstract

Container-based virtualization has become pivotal in cloud computing, and resource fragmentation is inevitable due to the frequency of container deployment/termination and the heterogeneous nature of IoT tasks. In queuing cloud systems, resource defragmentation and task scheduling are interdependent yet rarely co-optimized in existing research. This paper addresses this gap by investigating the joint optimization of resource defragmentation and task scheduling in a queuing cloud computing system. We first formulate the problem to minimize task completion time and maximize resource utilization, then transform it into an online decision problem. We propose a Deep Reinforcement Learning (DRL)-based two-layer iterative approach called DRL-RDG, which uses a Resource Defragmentation approach based on a Greedy strategy (RDG) to find the optimal container migration solution and a DRL algorithm to learn the optimal task-scheduling solution. Simulation results show that DRL-RDG achieves a low average task completion time and high resource utilization, demonstrating its effectiveness in queuing cloud environments.

1. Introduction

Cloud computing provides on-demand access to a shared pool of configurable computing resources, such as servers, storage, networks, applications, and so on, over the Internet. Cloud computing systems provide services by allocating underlying physical resource flexibly to users via virtualization technologies [1,2]. In recent years, container-based virtualization has emerged in cloud computing, driving widespread attention and adoption across both industry and academia due to the lightweight and flexibility of features [3]. For example, major cloud providers (e.g., AWS, Google Cloud, and Azure) offer container orchestration services (e.g., Kubernetes and ECS) to support large-scale application deployment and management [4,5].

The proliferation of Internet of Things (IoT) applications further amplifies the importance of efficient resource management in cloud environments [4]. Some IoT systems generate massive volumes of time-sensitive data that require rapid processing and low-latency responses, placing stringent demands on underlying infrastructure. Containerized virtualization has emerged as a key enabler for IoT cloud deployments.

When receiving tasks from users, the container scheduler in a cloud computing system determines the priority of tasks (i.e., which tasks should be processed first) and maps each task’s container to a server (i.e., which server should host the container to fulfill resource requirements). The container scheduler is a key enabling technology, since it affects the resource utilization efficiency and users’ service-level agreements [6]. However, one common challenge associated with the containerized service model is the issue of resource fragmentation. As containers are rapidly deployed and terminated, available CPU, memory, and storage resources on servers are split into small blocks. The considerable heterogeneity of IoT tasks and cumulative resource allocation/deallocation cycles naturally lead to resource fragmentation, making it difficult to achieve high resource utilization and fast job completion [7,8].

Against this backdrop, jointly optimizing resource defragmentation and task scheduling becomes an even more formidable challenge in queuing cloud systems. Due to the high cost and substantial energy consumption of cloud infrastructure, it is costly to over-procure such infrastructure to process a large number of user requests in real time. In this scenario with limited resources, task queuing becomes inevitable, especially in private cloud environments. In such a queuing cloud system, resource defragmentation and task scheduling are inherently interdependent processes. Effective resource defragmentation consolidates scattered idle resources, which enables the scheduler to allocate resources to more tasks and reduces the average task completion time. Conversely, task scheduling can proactively prevent excessive fragmentation. This bidirectional interdependence highlights the necessity of an integrated optimization framework that unifies defragmentation and scheduling to simultaneously enhance resource utilization efficiency while reducing task completion time.

A number of resource defragmentation policies have been explored by leveraging container/Virtual Machine (VM) live migration. Live migration allows running containers/VMs to be transferred between physical servers without service interruption, making it possible to consolidate the resource fragmentation of physical servers and optimize resource allocations. The existing resource defragmentation schemes mainly focus on metrics such as the number of active hosts [9,10], power consumption [11,12], migration cost [10,13,14], resource fragmentation [9,15], etc. While these efforts have made meaningful progress in reducing resource fragmentation and improving infrastructure efficiency, these schemes mainly focus on the scenario where the sequence of incoming user tasks is unknown and fail to account for the joint optimization of resource defragmentation and task scheduling.

To address the above problem, we investigate the joint optimization of resource defragmentation and task scheduling in a queuing cloud computing system. As shown in Figure 1, users continuously submit tasks, which enter a task queue to wait for processing. A container scheduler migrates running containers as needed and dispatches these queued tasks to the three servers. Figure 2 illustrates a comparative analysis of different resource defragmentation and task-scheduling strategies. Figure 2a depicts a Best fit-based task scheduling strategy without resource defragmentation, where no container is migrated; $k_{12}$ is scheduled to $s_2$ [16]. Figure 2b depicts a separate optimization of resource defragmentation and task scheduling, where $c_7,c_9,c_{10}$ are migrated to $s_3,s_2,s_1$, respectively, for maximum available resources [15]; then, $k_{11},k_{12}$ are scheduled to $s_3$. In the resource defragmentation stage, containers are migrated to minimize resource fragmentation, without regard for resource requirements of queued tasks. Figure 2c depicts a joint optimization of resource defragmentation and task scheduling, where only $c_{10}$ is migrated to $s_1$, under the consideration of resource requirements of queued tasks; then, $k_{11},k_{12}$ are scheduled to $s_3,s_2$, respectively. This example indicates that, compared with task scheduling alone, resource defragmentation improves the resource utilization by scheduling more tasks at the cost of container migration. Through the joint optimization of resource defragmentation and task scheduling, identical tasks are scheduled while involving fewer container migration operations.

There are two primary challenges in the joint optimization of resource defragmentation and task scheduling: the heterogeneous and dynamic nature of task resource requirements, as well as variations in task duration. These factors collectively contribute to persistent resource fragmentation, which, in turn, increases task queuing delays and degrades overall system efficiency. Effectively addressing these issues requires not only intelligent task sequencing to minimize makespan but also proactive container migration strategies that consolidate fragmented resources with minimal overhead. Furthermore, the interdependence between scheduling and defragmentation necessitates a coordinated approach that dynamically balances immediate task execution needs with long-term resource availability.

In this paper, we formulate the joint resource defragmentation and task-scheduling problem, with the aim of minimizing the task completion time and maximizing resource utilization. The problem is then transformed to an online decision problem that aligns with the dynamic, real-time nature of cloud environments. We further propose a Deep Reinforcement Learning (DRL)-based resource defragmentation and task-scheduling approach called DRL-RDG, which is a two-layer iterative approach to solve the queued task-scheduling subproblem through a DRL algorithm and the resource defragmentation subproblem through a Resource Defragmentation approach based on a Greedy strategy (RDG). The proposed DRL-RDG can balance short-term migration costs against long-term scheduling gains in task completion time and utilization.

The remainder of this paper is organized as follows. Section 2 introduces the related work. Section 3 formulates the joint resource defragmentation and task-scheduling problem. Section 4 transforms the problem to an online decision problem, then introduces the DRL-RDG approach. Simulation studies are conducted to demonstrate the efficiency of the proposal in Section 5. Finally, Section 6 concludes this paper.

2. Related Work

A number of VM/container resource defragmentation approaches and task scheduling approaches in cloud computing have been proposed in recent years.

To mitigate the resource fragmentation issue, VM/container consolidation has been extensively researched, primarily focusing on a better assignment of VMs/containers to hosts. The exiting VM/container consolidation approaches mainly focus on power consumption [11,12], resource utilization [17], the number of active hosts [9,10,18], migration cost [10,13,14], resource fragmentation [9,15], etc. To solve the consolidation problem, many metaheuristics and heuristics have been proposed. Metaheuristics provide approximate solutions based on genetic algorithms [19], ant colony optimization [20], particle swarm optimization [21], etc. Heuristics are proposed for better performance in terms of the solvable instance size and time complexity [9,10,15,22]. For example, Gudkov et al. [10] proposed a heuristic for solving the VM consolidation problem with a controlled trade-off between the number of released hosts and the amount of migration memory. The key idea is to place a VM in a host with a lack of free space using induced migrations [10]. Zhu et al. proposed a nimble, fine-grained consolidation algorithm, which focuses on utilizing resource fragmentation to increase the number of additional VM allocations [15]. Kiaee et al. considered joint VM and container consolidation in an edge-cloud environment and proposed an autoencoder-based solution comprising two stages of consolidation subproblems—namely, a joint VM and container multi-criteria migration decision and an edge-cloud power service-level agreement for VM placement [7]. However, these consolidation approaches mainly focus on scenarios where the sequence of incoming user tasks is unknown and fail to account for the joint optimization of resource defragmentation and task scheduling.

Many works on task scheduling in cloud computing have been proposed to improve load balancing [23,24], minimize task completion time [25], maximize throughput [26], etc. Early research often focused on heuristic methods to efficiently allocate tasks to available virtual machines, such as first in–first out (FIFO), shortest job first (SJF), genetic algorithms, and so on [25,27,28]. More recently, machine learning-based techniques, including reinforcement learning, have been applied to adaptively learn optimal scheduling policies in dynamic and heterogeneous cloud environments [29,30,31]. For example, Guo et al. studied delay-optimal scheduling in a queuing cloud computing system and proposed a heuristic algorithm in which a min–min best fit policy is used to solve inter-queue scheduling and an SJF policy is used to solve intra-queue buffering [25]. Kang et al. proposed an automatic generation network-based DRL approach to learn the optimal policy of dispatching arriving user requests, with the reward aiming to minimize the task response time and maximize resource utilization [31].

However, these studies on task scheduling overlook the issue of resource fragmentation. This limitation becomes particularly critical in high-load scenarios, where fragmented resources prevent efficient task placement, leading to increased delays and resource under-utilization. Consequently, there is a growing recognition of the need to integrate resource defragmentation awareness into task-scheduling strategies, especially in queuing cloud systems where resource continuity and task scheduling are deeply interdependent.

3. System Model and Problem Formulation

3.1. System Model

This paper considers a queuing cloud computing system with a resource pool consisting of a number of computing servers. S denotes the set of computing servers, and m is the number of computing servers. Each server ($s_i\in S(1\le i\le m)$) has limited resources (e.g., CPU, memory, etc.). $R_c$ and $R_m$ denote the capacity of the CPU and memory of servers ($s_i$). These resources are provided to users in the form of containers. Specifically, users submit task requirements to the cloud computing system, which allocates container instances based on these requirements. The capacity of a server is larger than that of a container instance. Therefore, multiple instances can be run on a server in parallel, and the instances are dispersed across servers according to the task allocation policy. To ensure flexible resource allocation and efficient resource utilization, we assume that the resource requirements of container instances are exactly equal to the task requirements of users.

3.2. Task Model

Heterogeneous and dynamic tasks are considered. The CPU and memory requirements of tasks are heterogeneous, and each task arrives independently. We consider a non-preemptive time-slotted system. At the beginning of each time slot ($t\in [1,\tau ]$), users submit tasks to the system, where $\tau$ is a sufficiently large integer. $k_j(1\le j\le n)$ denotes the tasks, where n is the total number of tasks. We assume that when a task arrives, its resource requirement and its duration are specified. The resource requirement of each task ($k_j(1\le j\le n)$) contains the CPU requirement ($r_c\left(j\right)$) and memory requirement ($r_m\left(j\right)$). The duration ($T_d\left(j\right)$) of task $k_j$ represents its required processing time. The submitted time of task $k_j$ is the time at which task $k_j$ is submitted to the scheduling queue, denoted by $T_s\left(j\right)$. The custom container instance ($c_j(1\le j\le n)$) allocated to the task ($k_j$) is also assigned resources $r_c\left(j\right)$ and $r_m\left(j\right)$.

As shown in Figure 1, users continuously submit tasks ($k_1,k_2,\dots$), which enter a task queue to wait for processing. The height and length of the task blocks and container blocks represent the CPU requirement ($r_c\left(j\right)$) and memory requirement ($r_m\left(j\right)$), respectively. Tasks $k_1,k_2,\dots$ are scheduled to specific containers $c_1,c_2,\dots$ on servers $s_1,s_2,s_3$ according to the scheduling strategy. Each task ($k_i$) is executed in its designated container ($c_i$). The gray areas represent remaining available resources. A container scheduler migrates running containers as needed and dispatches these queued tasks to the three servers. The details of some notations are listed in

Table 1. The details of some notations.

Symbols	Definition
$s_i(1\le i\le m)$	The servers in the queuing cloud computing system
$k_j(1\le j\le n)$	The tasks submitted to the queuing cloud computing system
$c_j(1\le j\le n)$	The container instance in which $k_j$ is performed
$R_c,R_m$	The CPU and memory capacity of each server
$r_c\left(j\right),r_m\left(j\right)$	The CPU and memory requirement of task $k_j$
$T_d\left(j\right)$	The duration of task $k_j$
$T_s\left(j\right)$	The submitted time of task $k_j$
$T_m\left(j\right)$	The downtime of task $k_j$ caused by container migration
$T_r\left(j\right)$	The overall running time of task $k_j$
$x_{i,j,t}$	The task-scheduling decision variable
$y_{i,i^{\prime },j,t}$	The resource defragmentation decision variable
$T_c\left(j\right)$	The overall completion time of task $k_j$
$R_u$	The resource utilization rate
${\mathcal{K}}^f$, ${\mathcal{K}}^q$, and ${\mathcal{K}}^r$	The sets of completed tasks, queued tasks, and running tasks
$H=\left\{h_{i,j}\right\}$	The task-scheduling decision matrix during time slot $t-1$
${\tilde{x}}_{i,j}$	The task-scheduling decision variables in the current time slot (t)
${\tilde{y}}_{i,i^{\prime },j}$	The resource defragmentation decision variables in the current time slot (t)

.

3.3. Task-Scheduling Decision

Let $x_{i,j,t}$ denote the task-scheduling decision, where $x_{i,j,t}\in \{0,1\}$, $1\le i\le m$, $1\le j\le n$, and $t\in [1,\tau )$. $x_{i,j,t}=1$ denotes that container instance $c_j$ is hosted by server $s_i$ to execute task $k_j$ in time slot t and $x_{i,j,t}=0$ otherwise. Each container ($c_j$) must run continuously once started, and it can be migrated but not interrupted until its task finishes.

$$\sum _i{x}_{i,j,t}+\sum _i\sum _{l=t+2}^{\tau }{x}_{i,j,l}-\sum _i{x}_{i,j,t+1}\le 1,\forall j\in [1,n],t\in [1,\tau )$$

(1)

Formula (1) enforces the non-interruption of task execution. Specifically, it ensures that if a task is active in time slot t and becomes inactive in $t+1$, it cannot be reactivated in $t+2$ or any subsequent time slot. This prevents fragmented execution and guarantees continuous resource allocation once a task begins.

Additionally, containers cannot be partitioned and placed on multiple servers, which leads to another constraint—namely, that a container should be hosted on, at most, one server at any time.

$$\sum _{i=1}^m{x}_{i,j,t}\le 1,\forall j\in [1,n],t\in [1,\tau )$$

(2)

For each server, the allocated resource for containers should be below the capacity ($R_c$ and $R_m$).

$$\sum _{j=1}^n{x}_{i,j,t}{r}_c\left(j\right)\le R_c,\sum _{j=1}^n{x}_{i,j,t}{r}_m\left(j\right)\le R_m,\forall i\in [1,m],t\in [1,\tau )$$

(3)

3.4. Resource Defragmentation Decision

For running containers, container migration between servers can be implemented to resolve resource fragmentation. Let $y_{i,i^{\prime },j,t}$ denote the resource defragmentation decision variable, indicating whether $c_j$ is migrated from server $s_i$ to $s_{i^{\prime }}$ in t, where $y_{i,i^{\prime },j,t}\in \{0,1\}$, $1\le i,i^{\prime }\le m$, $i\ne i^{\prime }$, $1\le j\le n$, and $t\in [1,\tau )$. $y_{i,i^{\prime },j,t}=1$ denotes that container instance $c_j$ is migrated from server $s_i$ to $s_{i^{\prime }}$ in t and $y_{i,i^{\prime },j,t}=0$ otherwise. Note that ∀$i=i^{\prime }$, $y_{i,i^{\prime },j,t}=0$. The relationship between the resource defragmentation decision variable and the task-scheduling decision variable is

$$y_{i,i^{\prime },j,t}=\left\{\begin{array}{cc}1& x_{i,j,t}=1\mathrm{and}{x}_{i^{\prime },j,t+1}=1\mathrm{and}i\ne i^{\prime }\hfill \\ 0& \mathrm{otherwise}\hfill \end{array}\right.$$

(4)

Container migration can bring about a service disruption, as denoted by service downtime ($T_m\left(j\right)$). For each container ($c_j$), the overall running time ($T_r\left(j\right)$) is the summation of its duration and its service downtime. It is defined as

$$T_r\left(j\right)=\sum _{t=1}^{\tau }\sum _{i^{\prime }=1}^m\sum _{i=1}^m{y}_{i,i^{\prime },j,t}{T}_m\left(j\right)+T_d\left(j\right).$$

(5)

This definition leads to the task-scheduling constraint:

$$\sum _{t=1}^{\tau }\sum _{i=1}^m{x}_{i,j,t}=T_r\left(j\right),\forall j\in [1,n].$$

(6)

3.5. Objective Function

In this paper, we focus on the task completion time and the resource fragmentation rate in a cloud system. The task completion time of a task ($k_j$) is the summation of its queuing time and the running time of $c_j$, and it relates directly to the quality of experience of users. The resource fragmentation rate refers to a metric that quantifies the degree to which available resources (e.g., memory and CPU) are divided into small, non-contiguous blocks, reducing their usability for efficient allocation. A task-scheduling policy achieving a lower resource fragmentation rate is more efficient in terms of resource utilization.

Let $T_c\left(j\right)$ be the task completion time of $k_j$, which is defined as

$$T_c\left(j\right)=min\left\{t\right|\sum _{i=1}^m{x}_{i,j,t}=1\}-T_s\left(j\right)+T_r\left(j\right),$$

(7)

where $min\left\{t\right|{\sum }_{i=1}^m{x}_{i,j,t}=1\}$ is the start execution time of $k_j$. Let $R_u$ be the resource utilization rate, which is defined as

$$R_u={\mu }_1{\displaystyle \frac{{\sum }_{t=1}^{\tau }{\sum }_{i=1}^m{\sum }_{j=1}^n{x}_{i,j,t}{r}_c\left(j\right)}{m{R}_c\tau }}+{\mu }_2{\displaystyle \frac{{\sum }_{t=1}^{\tau }{\sum }_{i=1}^m{\sum }_{j=1}^n{x}_{i,j,t}{r}_m\left(j\right)}{m{R}_m\tau }},$$

(8)

where ${\mu }_1$ and ${\mu }_2$ are weight coefficients for CPU and memory utilization metrics, respectively. ${\mu }_1$ and ${\mu }_2$ are constrained to the interval of [0, 1], with ${\mu }_1+{\mu }_2=1$. Note that ${\mu }_1$ and ${\mu }_2$ are hyperparameters that balance the relative importance of CPU utilization versus memory utilization in the reward function. Their values are assigned based on the focus of the application scenario. For instance, ${\mu }_1$ is assigned a higher weight in CPU-intensive scenarios, whereas ${\mu }_2$ is assigned a higher weight in memory-intensive scenarios.

The goal of this paper is to search for a task-scheduling and resource defragmentation policy that solves the following problem:

$$\begin{array}{cc}\hfill \mathrm{P}1:& min{\eta }_1{\displaystyle \frac{{\sum }_{j=1}^n{T}_c\left(j\right)}{n}}-{\eta }_2{R}_u\hfill \\ \hfill & s.t.\left(1\right)–\left(4\right),\left(6\right),\hfill \end{array}$$

where ${\eta }_1$ and ${\eta }_2$ are weight coefficients for the average task completion time and the resource utilization rate, respectively. ${\eta }_1$ and ${\eta }_2$ are constrained to the interval of [0, 1], with ${\eta }_1+{\eta }_2=1$. Note that ${\displaystyle \frac{{\sum }_{j=1}^n{T}_c\left(j\right)}{n}}$ carries the dimension of time, while $R_u$ is a dimensionless ratio. The ${\eta }_1$ and ${\eta }_2$ coefficients serve not only as weights to balance the relative importance of minimizing the average task completion time and maximizing the resource utilization rate but also effectively normalize the different units of the two terms.

According to the objective function and Equations (5), (7) and (8), the decision variables are $x_{i,j,t}$ and $y_{i,i^{\prime },j,t}$. According to Equation (4), the value of $y_{i,i^{\prime },j,t}$ is inherently determined once $x_{i,j,t}$ is determined. Therefore, the above problem is reduced to finding the optimal task-scheduling sequence ($x_{i,j,t}^{⁎}$).

4. Joint Resource Defragmentation and Task-Scheduling Approach

In this paper, we consider an online dynamic scheduling system where container migration and task-scheduling decisions must be made at a specific moment to respond to real-time task arrivals. In this section, we first transform the problem to an online decision problem, then introduce the DRL-RDG approach.

4.1. Problem Transformation

In the online scheduling system, we consider the following process: at the beginning of a specific time slot (t), the state of tasks and server resource information required for scheduling decision-making are captured, including the queued tasks, the running containers, and the resource availability of servers. Concomitantly, the scheduler determines which tasks in the queue are to be assigned to which servers, as well as whether any running containers need to be migrated and, if so, to which servers.

Let ${\mathcal{K}}^f$, ${\mathcal{K}}^q$, and ${\mathcal{K}}^r$ denote the sets of completed tasks, queued tasks, and running tasks, respectively. The three sets are dynamically updated at the beginning of each time slot (t) to reflect the latest system state. For tasks in ${\mathcal{K}}^r$, let $H=\left\{h_{i,j}\right\}$ denote the task-scheduling decision matrix in time slot $t-1$. Let ${\tilde{x}}_{i,j}$ be the task-scheduling decision variables in the current time slot (t). For tasks in ${\mathcal{K}}^q$, ${\tilde{x}}_{i,j}=1$ denotes that task $k_j$ is assigned to server $s_i$. For tasks in ${\mathcal{K}}^r$, ${\tilde{x}}_{i,j}=1$ and $h_{i,j}=0$ denotes that container $c_j$ of $k_j$ is migrated to server $s_i$.

Let $\tilde{T}$ be the current cumulative task completion time, which is defined as

$$\tilde{T}=\sum _{j\in {\mathcal{K}}^q}(1-\sum _{i=1}^m{\tilde{x}}_{i,j})+\left|{\mathcal{K}}^r\right|+\sum _{j\in {\mathcal{K}}^r}\sum _{i=1}^m({\tilde{x}}_{i,j}-h_{i,j})T_m\left(j\right).$$

(9)

Let ${\tilde{R}}_u$ be the resource utilization rate during time slot t, which is defined as   

$${\tilde{R}}_u={\mu }_1{\displaystyle \frac{{\sum }_{i=1}^m{\sum }_{j=1}^n{\tilde{x}}_{i,j}{r}_c\left(j\right)}{m{R}_c}}+{\mu }_2{\displaystyle \frac{{\sum }_{i=1}^m{\sum }_{j=1}^n{\tilde{x}}_{i,j}{r}_m\left(j\right)}{m{R}_m}}.$$

(10)

The goal of the online task-scheduling and resource defragmentation problem is to minimize ${\eta }_1{\displaystyle \frac{\tilde{T}}{n}}-{\eta }_2{\tilde{R}}_u$.

4.2. DRL-Based Resource Defragmentation and Task-Scheduling Approach

The proposed DRL-RDG is a two-layer iterative approach to address the coupled challenges of the online resource fragmentation and task-scheduling problem in dynamic computing environments. In the first layer, given a resource defragmentation decision, DRL-RDG leverages a task-scheduling approach based on DRL to solve the queued task-scheduling subproblem. In the second layer, conversely, given the current task-scheduling decisions, DRL-RDG employs an RDG approach to resolve the resource fragmentation subproblem. The iterative of two layers ensures that task scheduling and resource defragmentation mutually reinforce each other: effective scheduling reduces the need for frequent defragmentation, while proactive defragmentation creates more efficient resource configurations for subsequent scheduling decisions.

In the following, we first introduce the proposed RDG. Then, we introduce an RL-based resource defragmentation and task-scheduling approach called RL-RDG to highlight and substantiate the advantages of the DRL algorithm. Finally, we present the proposed DRL-RDG.

4.2.1. Resource Defragmentation Approach Based on Greedy Strategy (RDG)

In RDG, we define a resource imbalance index and a free-space size index. The resource imbalance index is a measure of the disparity between CPU and memory utilization on a server or for a container. This index captures the degree to which one resource (either CPU or memory) is over-utilized relative to the other. The resource imbalance index of container $c_j$ is

$$RI{I}_j={\displaystyle \frac{r_c\left(j\right)}{R_c}}-{\displaystyle \frac{r_m\left(j\right)}{R_m}},$$

(11)

and the resource imbalance index of a server ($s_i$) is

$$RI{I}_i={\displaystyle \frac{{\sum }_{j=1}^n{\tilde{x}}_{i,j}{r}_c\left(j\right)}{R_c}}-{\displaystyle \frac{{\sum }_{j=1}^n{\tilde{x}}_{i,j}{r}_m\left(j\right)}{R_m}}.$$

(12)

The domain of $RII$ is (−1, 1), where the larger absolute value of $RI{I}_i$ indicates a more severe imbalance, highlighting servers that require adjustment to align CPU and memory utilization.

The free-space size index is used to quantify the underutilized resource capacity of a server. As the overall free-space size increases, the resource fragments are further reduced. The free-space size of a server ($s_i$) is

$$F_i=(R_c-\sum _{j=1}^n{\tilde{x}}_{i,j}{r}_c\left(j\right))(R_m-\sum _{j=1}^n{\tilde{x}}_{i,j}{r}_m\left(j\right)),$$

(13)

and the overall free-space size of servers is $F_O={\sum }_{i=1}^m{F}_i$.

The detailed procedure of RDG is outlined in Algorithm 1. The core objective of RDG is to systematically select containers for migration from imbalanced servers to target servers, aiming to improve overall resource balance and reduce fragmentation. In row 2, RDG computes the priority of each server based on

$$p{r}_i=w_p\ast \left|RI{I}_i\right|+(1-w_p)(1/(F_i+1e-10)),$$

(14)

where $w_p$ is the weight of the resource imbalance index. This formula prioritizes servers that are both highly imbalanced (high $|RI{I}_i|$) and have small free space (high $1/F_i$), as these are the most critical candidates for defragmentation. Then, RDG sorts servers by priority. In rows 4–19, RDG attempts to migrate containers in the server with the highest priority. In rows 8–9, RDG evaluates the migration priority of containers through a correlation metric between the container and the server:

$$cor{r}_{i,j}=RI{I}_j·RI{I}_i·max\left({\displaystyle \frac{r_c\left(j\right)}{R_c}}.{\displaystyle \frac{r_m\left(j\right)}{R_m}}\right).$$

(15)

A higher positive $cor{r}_{i,j}$ value indicates that the container’s resource profile (its own imbalance ($RI{I}_j$)) strongly aligns with and potentially exacerbates the server’s overall imbalance ($RI{I}_i$). The $max(\dots )$ term ensures containers with significant resource demands are considered. Thus, containers with the highest positive scores are prioritized for migration, as they are likely the main contributors to the server’s imbalance.

Algorithm 1 Resource Defragmentation approach based on Greedy strategy (RDG)

Input: the initial matrix $H=\left\{h_{i,j}\right\}$, the maximum allowed migrations $M_{max}$, $R_c,R_m,{\mathcal{K}}^r$ Output: the resource defragmentation decision $\{{\tilde{y}}_{i,i^{\prime },j}\}$   1: Compute $RI{I}_j$, $RI{I}_i,F_i,F_O$   2: Compute the priority $p{r}_i$, and sort servers in descending order to form a priority list L   3: Initialize migration counter: $M_{total}$ = 0   4: for $s_i\in L$ do   5:     if $M_{total}\ge M_{max}$ then   6:        Break   7:     end if   8:     Identify containers on $s_i$: ${\mathcal{C}}_j=\left\{c_j\right|h_{i,j}=1\}$, and compute correlation $cor{r}_{i,j}$ for $c_j\in {\mathcal{C}}_j$   9:     Sort ${\mathcal{C}}_j$ in descending order of $cor{r}_{i,j}$ to get migration candidates 10:     for $c_j\in {\mathcal{C}}_j$ do 11:        Find optimal target server $s_{i^{*}}$ 12:        if optimal $s_{i^{*}}$ exists and $\Delta F_O>0$ then 13:             ${\tilde{y}}_{i,i^{*},j}=1$ 14:             Update H, $RI{I}_i,RI{I}_{i^{*}},F_i,F_{i^{*}},F_O,L$ 15:             $M_{total}=M_{total}+1$ 16:             Break 17:        end if 18:     end for 19: end for 20: return$\{{\tilde{y}}_{i,i^{\prime },j}\}$

Containers are then ranked by the correlation value, and the container with the highest positive scores is marked as the top migration candidate. The target server in row 11 should have sufficient resources and satisfy $s_i^{*}=arg{min}_{i^{*}}\{\Delta F_O=F_O(x_{i^{*},j}=1)-F_O(x_{i,j}=1)\}$. This condition seeks the target server that minimizes the increase in overall system fragmentation ($\Delta F_O$) after the hypothetical migration. If such a target server exists, the container is migrated from $s_i$ to $s_i^{*}$. The system state metrics are updated. This process repeats for the highest-priority server until the migration counter reaches the maximum times ($M_{max}$). The output of RDG is a resource defragmentation decision matrix ($\{{\tilde{y}}_{i,i^{\prime },j}\}$). Based on the greedy strategy, RDG ensures efficient resource defragmentation, thereby improving overall resource utilization efficiency.

4.2.2. RL-RDG

In RL-RDG, the RDG algorithm is used to solve the resource defragmentation subproblem, and the Q-learning algorithm is used to solve the task-scheduling subproblem.

Q-learning is a common reinforcement learning algorithm that learns the rewards of specific actions in given states by constructing and updating a Q-table. The Q-table consists of Q-values ($Q(s_Q,a)$) representing the expected rewards for taking an action in a specific state. In RL-RDG, the state contains the resource allocation status of each server, denoted as $s_Q=\langle H,{\mathcal{K}}^f,{\mathcal{K}}^q,{\mathcal{K}}^r\rangle$. An action in state $s_Q$ is $a\left(s_Q\right)=\{(k_{j1},s_{i1}),(k_{j2},s_{i2}),\dots \}$, which means assigning a set of one or more queued tasks ($k_{j1},k_{j2},\dots \in {\mathcal{K}}^q$) to the according servers. This action is valid only if $s_i$ has enough free CPU and memory to accommodate $k_j$. The action causes a system state transition: $s_Q^{\prime }=a\left(s_Q\right)$. The reward of $a\left(s_Q\right)$ is $R={\eta }_2{\tilde{R}}_u-{\eta }_1\tilde{T}$. The Q-value is updated using the formula expressed as $Q(s_Q,a)=R+\gamma ·{max}_{a^{\prime }}Q(s_Q^{\prime },a^{\prime })$, where $\gamma$ is the discount factor for future rewards.

The detailed procedure of RL-RDG is outlined in Algorithm 2. The algorithm operates through nested loops. The outer loop (Lines 2–16) continues until no valid actions exist for $s_{Q0}$ or all queued tasks are allocated ${\mathcal{K}}^q=\varnothing$. The inner loop involves learning Q-values by updating the Q-table. In each iteration,

• Line 2: The RDG algorithm first performs container migration based on the current system state to optimize resource fragmentation, updating the server state (H);
• Lines 3–6: The Q-table is initiated, and valid actions in the initial state ($s_{Q0}$) are checked for;
• Lines 8–14: The inner Q-learning loop selects actions using the $ϵ$-greedy policy, observes rewards and next states, and updates Q-values until convergence;
• Line 15: The optimal action sequence is selected based on the learned Q-values to schedule tasks.

Algorithm 2 RL-RDG

Input: $\gamma ,ϵ$, $H,M_{max},R_c,R_m$,${\mathcal{K}}^r$, ${\mathcal{K}}^q$, ${\mathcal{K}}^f$ Output: The resource defragmentation and task scheduling decisions $\{{\tilde{x}}_{i,j}\}$, $\{{\tilde{y}}_{i,i^{\prime },j}\}$   1: repeat   2:    Migrate containers according to RDG algorithm, and update H   3:    $Q(s_Q,a)=0$, $s_{Q0}=\langle H,{\mathcal{K}}^f,{\mathcal{K}}^q,{\mathcal{K}}^r\rangle$   4:    if $a\left(s_{Q0}\right)=\varnothing$ then   5:       Break   6:    end if   7:    $s_Q=s_{Q0}$   8:    repeat   9:       if ${\mathcal{K}}^q\ne \varnothing$ then 10:          Select $a\left(s_Q\right)$ according to $ϵ$-greedy Policy 11:          Observe $s_Q^{\prime }$, R and update $Q(s_Q,a)$, $H,{\mathcal{K}}^r,{\mathcal{K}}^q,{\mathcal{K}}^f$ 12:          $s_Q=s_Q^{\prime }$ 13:       end if 14:    until $Q(s_Q,a)$ converge 15:    Select the optimal action to schedule tasks 16: until ${\mathcal{K}}^q=\varnothing$ 17: return Task allocation records and migration decisions

The action selection strategy in row 10 is the $ϵ$-greedy policy, which is defined as follows:

$$\pi \left(a\right|s)=\left\{\begin{array}{cc}1-ϵ+{\displaystyle \frac{ϵ}{\left|A\right(s\left)\right|}}& a=arg\underset{a}{min}Q(s,a)\hfill \\ {\displaystyle \frac{ϵ}{\left|A\right(s\left)\right|}}& \mathrm{otherwise},\hfill \end{array}\right.$$

(16)

where $\left|A\right(s\left)\right|$ denotes the number of optional actions. The above policy classifies optional actions into two categories: one category consists of the action(s) that maximize(s) the Q-value, and the other includes all remaining actions. The iterative process continues until $Q(s_Q,a)$ is converged. The convergence criterion is defined as either reaching the maximum iteration limit of 1000 or the maximum change in Q-values between consecutive iterations falling below a threshold of 0.001. It is worth noting that the episode limit and the threshold value can be adjusted according to specific application scenarios and problem scales. After task scheduling, RDG is triggered to perform container migration based on the updated server state. The outer iteration continues until no valid actions exist for $s_{Q0}$ or all queued tasks are allocated ${\mathcal{K}}^q=\varnothing$.

4.2.3. DRL-RDG

In our proposed DRL-RDG, the deep Q-network algorithm replaces the traditional Q-learning algorithm to solve the task-scheduling subproblem. The deep Q-network is a common deep reinforcement learning algorithm that learns the expected long-term rewards of specific actions in given states using a neural network. It can overcome the limitation of Q-learning’s discrete Q-table, which fails to handle high-dimensional state spaces efficiently.

In DRL-RDG, there are two neural networks (Q and $\widehat{Q}$) and a replay memory (D). The two network (Q and $\widehat{Q}$) have the same neural network structure, while they have different network parameter vectors ($\theta$ and ${\theta }^{-}$). The neural networks contain four layers: an input layer, two fully connected layers, and an output layer. The Q-values are represented by $Q(s_Q,a;\theta )$ and $\widehat{Q}(s_Q,a;{\theta }^{-})$ under state $s_Q$ and action a, respectively. The replay memory (D) stores the observed experience $(s_Q,a,R,s_Q^{\prime })$, which is used to train the neural network.

The DRL-RDG algorithm is outlined in Algorithm 3. The algorithm operates in two nested loops:

• The outer loop (Lines 4–26) iterates until all queued tasks are scheduled. In each iteration, RDG first performs container migration to optimize resource fragmentation (Line 3). The initial state ($s_{Q0}$) is then constructed from the updated system state (Line 4). If no valid scheduling actions exist from this state, the loop terminates (Lines 5–7).
• The inner loop (Lines 8–23) represents the DQN training and decision-making phase over multiple episodes and steps. Within each step, if queued tasks exist, an action (a) is selected using an $ϵ$-greedy policy (Line 12). This policy balances exploration (random action with probability) and exploitation (selecting the action with the highest Q-value from Q). After executing the action, the system observes the new state and reward, stores the experience in replay memory (Line 13), and updates the current state (Line 14).

Algorithm 3 DRL-RDG

Input: $(\gamma ,ϵ)$, $H,M_{max},R_c,R_m$, ${\mathcal{K}}^r,{\mathcal{K}}^q,{\mathcal{K}}^f$ Output: The resource defragmentation and task scheduling decisions $\{{\tilde{x}}_{i,j}\}$, $\{{\tilde{y}}_{i,i^{\prime },j}\}$   1: Initialize replay memory D, network Q with random parameter $\theta$, target network $\widehat{Q}$ with parameter ${\theta }^{-}=\theta$   2: repeat   3:    Migrate containers according to RDG algorithm, and update H   4:    Obtain $s_{Q0}=\langle H,{\mathcal{K}}^f,{\mathcal{K}}^q,{\mathcal{K}}^r\rangle$   5:    if $a\left(s_{Q0}\right)=\varnothing$ then   6:       Break   7:    end if   8:    for episode = 1, …, E do   9:       $s_Q=s_{Q0}$ 10:       for step = 1, …, C do 11:           if ${\mathcal{K}}^q\ne \varnothing$ then 12:              Select $a\left(s_Q\right)$ according to $ϵ$-greedy Policy 13:              Observe $s_Q^{\prime }$, R, update $H,{\mathcal{K}}^r,{\mathcal{K}}^q,{\mathcal{K}}^f$, and store $(s_Q,a,R,s_Q^{\prime })$ in D 14:              $s_Q=s_Q^{\prime }$ 15:           end if 16:           Sample a set of experience from D 17:           Compute $\widehat{Q}$, and update $\theta$ to minimize $L(\theta ,\widehat{Q})$ 18:           step = step + 1 19:           if mod (step, updatestep) = 0 then 20:              ${\theta }^{-}=\theta$ 21:           end if 22:       end for 23:    end for 24:    Select the optimal action to schedule tasks 25: until ${\mathcal{K}}^q=\varnothing$ 26: return Task allocation records and migration decisions

The $ϵ$-greedy policy in row 12 is based on Equation (16), where the Q-value is $Q(s_Q,a;\theta )$. The neural network training phase (Lines 16–23) involves sampling a batch of experiences ($\mathcal{I}$) from replay memory (D). Based on these experience samples, the key idea of the update of Q is to minimize the difference between the Q-values predicted by the online network ($Q(s_Q,a;\theta )$) and the target values generated by the target network ($\widehat{Q}(s_Q,a;{\theta }^{-})$). The loss function is expressed as follows:

$$L(\theta ,\widehat{Q})={\displaystyle \frac{1}{\left|\mathcal{I}\right|}}\sum _{i\in \mathcal{I}}{(Q(s_Q\left(i\right),a\left(i\right);\theta )-\widehat{Q}(s_Q\left(i\right),a\left(i\right);{\theta }^{-}))}^2$$

(17)

where $\left|\mathcal{I}\right|$ is the cardinality of set $\mathcal{I}$. This mean squared error loss ensures that the online network’s predictions gradually align with the more stable target values.

For every updatestep, $\widehat{Q}$ is updated by copying Q, i.e., the target network parameters (${\theta }^{-}$) are synchronized with the online network parameters ($\theta$; Line 21) to stabilize training. The iteration between RDG and the deep Q-network is the same as the iteration between RDG and Q-learning in the RL-RDG algorithm. The iterationalso continues until no valid actions exist for $s_{Q0}$ or all queued tasks are allocated (${\mathcal{K}}^q=\varnothing$).

5. Performance Evaluation

In this section, we use simulations to evaluate the performance of RL-RDG and DRL-RDG.

The system investigated in this paper is a queuing cloud computing system. The queuing environment is simulated by configuring servers with 96 CPUs and 256G memory. The results of Google trace show that the number of tasks that arrive every 5 min follows a Poisson distribution [32]. The task duration follows a heavy-tailed distribution, shaped with 80% of jobs in the trace being shorter than the average job duration [32,33]. Therefore, in the simulation, a Poisson distribution is used to generate a per time-slot number of arriving tasks. The task duration time and the resource requirement follow heavy-tailed distributions [25]. Therefore, the number of tasks arriving in each time slot follows P$\left(\lambda \right)$. The CPU requirement of tasks follows the log-normal distribution (LN(3, 0.75)), and the maximum CPU requirement is set as 64. The memory requirement of tasks follows the log-normal distribution (LN(4, 0.75)), and the maximum memory requirement is set as 128. The task duration time of tasks follows the Pareto distribution (Pa(1.5, 2)). The weight coefficients of CPU and memory utilization (${\mu }_1$ and ${\mu }_2$) are set as 0.5. The weight coefficients of the average task completion time and the resource utilization rate (${\eta }_1$ and ${\eta }_2$) are set as 0.1 and 0.9.

To evaluate the efficiency of DRL-RDG, we investigate the performance in terms of the task completion time and the resource utilization rate under various server cluster scales and task intensities, which are controlled through different parameter settings of the number of servers (m) and task arrival rates ($\lambda$). The benchmark methods include FIFO-RDG, SJF-RDG, and DRL-o. FIFO-RDG and SJF-RDG are approaches under the two-layer iterative framework. DRL-o only considers task scheduling without resource defragmentation. Their core mechanisms are introduced as follows:

• SJF-RDG: In the first layer, the Shortest Job First (SJF) algorithm which prioritizes tasks with shorter execution times, is employed to solve the task-scheduling subproblem. In the second layer, the RDG algorithm is employed to solve the resource defragmentation subproblem.
• FIFO-RDG: In the first layer, the First In–First Out (FIFO) algorithm is employed to solve the task-scheduling subproblem. FIFO-scheduled tasks follow the chronological order of task arrival. In the second layer, the RDG algorithm is employed to solve the resource defragmentation subproblem.
• DRL-o: In DRL-o, only the deep Q-network algorithm is employed to solve the task-scheduling problem without resource defragmentation.

Note that all experimental results are the average values obtained from 100 independent simulation runs conducted under identical environmental conditions. The source code has been released (https://jihulab.com/guoyan/resource-defragmentation-and-task-scheduling, accessed on 16 October 2025).

5.1. Server Cluster Scale

In this scenario, the number of servers varies from 30 to 70, and the average number of tasks arriving per time slot is equal to the number of servers. This parameter setting is designed to simulate a high-intensity and dynamically scalable cluster environment, where the system faces consistent task intensity in terms of the task-to-server ratio as the number of servers increases. Figure 3 shows the performance comparison in terms of the task completion time and the resource utilization rate under various server cluster scales.

Table 2. Quantitative comparison of performance metrics between DRL-RDG and RL-RDG.

Number of Servers		30	40	50	60	70
Average Task Completion Time	RL-RDG	9.53	9.50	8.45	8.66	8.35
	DRL-RDG	9.25	8.98	7.86	7.52	8.10
Average CPU Utilization	RL-RDG	82.5	83.93	84.2	85.1	86.3
	DRL-RDG	84.1	86.5	86.8	87.5	89.7
Average Memory Utilization	RL-RDG	83.2	79.5	83.1	85.3	84.7
	DRL-RDG	85.6	81.9	85.7	87.8	86.9

shows the quantitative comparison of performance metrics between DRL-RDG and RL-RDG.

Figure 3a shows that DRL-RDG and RL-RDG outperform SJF-RDG and FIFO-RDG in terms of average task completion time. This is because both algorithms can learn and dynamically adjust scheduling strategies to adapt to load changes, unlike the relatively fixed strategies of SJF-RDG and FIFO-RDG. DRL-RDG and RL-RDG also outperform DRL-o due to resource defragmentation. Specifically, for DRL-RDG, as the number of servers increases, its advantage over RL-RDG in average latency becomes more significant. In high-dimensional scenarios (with more servers), DRL-RDG can better handle complex state spaces, while RL-RDG is limited by insufficient Q-value learning, resulting in slightly higher latency compared to DRL-RDG. Compared with FIFO-RDG, the SJF-RDG scheduling strategy prioritizes short tasks, avoiding short tasks being blocked by long tasks, thereby reducing the overall average latency. DRL-o’s average task completion time lies between those of RL-RDG/SJF-RDG and FIFO-RDG. While DRL-o benefits from DRL’s ability, it suffers from persistent resource fragmentation, leading to a longer queuing time.

Figure 3b,c show that DRL-RDG and RL-RDG achieve relatively high resource utilization compared to SJF-RDG, FIFO-RDG, and DRL-o. DRL-RDG shows a slightly better trend as the number of servers increases, which is related to its better performance in handling complex scenarios. Note that as the number of servers increases, the utilization rate of these algorithms exhibits a slight improvement. This is because more servers mean more fragments can be integrated, and the efficiency of the overall resource pool is enhanced. In summary, the learning-based DRL-RDG and RL-RDG algorithms show significant advantages compared to the traditional SJF-RDG and FIFO-RDG in terms of average task completion time and resource utilization.

5.2. Task Arrival Rate

In this scenario, the number of servers is fixed at 50, while the task arrival rate varies incrementally from 40 to 60. This parameter configuration is intentionally designed to simulate a fixed-scale cluster under dynamically varying task intensities. Figure 4 shows the performance comparison in terms of the task completion time and the resource utilization rate under various task arrival rates.

Table 3. Quantitative comparison of performance metrics between DRL-RDG and RL-RDG.

Task Arrival Rate		40	45	50	55	60
Average Task Completion Time	RL-RDG	5.92	6.95	8.45	10.28	12.50
	DRL-RDG	5.85	6.18	7.86	9.42	11.35
Average CPU Utilization	RL-RDG	78.3	81.5	85.2	86.6	85.9
	DRL-RDG	79.5	84.2	87.8	89.1	92.5
Average Memory Utilization	RL-RDG	79.0	80.8	83.1	87.2	90.7
	DRL-RDG	81.2	83.5	85.7	90.9	92.3

shows the quantitative comparison of performance metrics between DRL-RDG and RL-RDG.

Figure 4a shows that as the task arrival rate increases, the average task completion time of all algorithms shows an upward trend. This is because a higher task arrival rate increases the probability of task queuing, thereby prolonging the average task completion time. Specifically, as the task arrival rate increases, SJF-RDG reduces the average completion time more effectively than FIFO-RDG by prioritizing short-duration tasks. DRL-o exhibits an average task completion time that is comparable to that of SJF-RDG. In contrast to SJF-RDG, RL-RDG and DRL-RDG further decrease the average completion time. This is achieved by comprehensively accounting for the task duration time, resource requirements of tasks, and the long-term implications of the current task-scheduling strategy.

Figure 4b,c show that as the task arrival rate increases, the average CPU and memory utilization of algorithms first rise and then stabilize. In the initial stage of an increasing task arrival rate, more tasks make better use of CPU and memory resources, thereby increasing resource utilization. When the task arrival rate reaches a certain level, the system attains a steady state, leading to stabilized resource utilization. Compared to RL-RDG and DRL-RDG, DRL-o exhibits lower average CPU and memory utilization. Since DRL-o does not perform resource defragmentation, scattered idle resources cannot be consolidated and allocated to tasks effectively. Compared to RL-RDG and DRL-RDG, SJF-RDG lacks long-term strategic optimization and, thus, exhibits more fluctuating utilization patterns.

5.3. Statistical Analysis of Performance Differences

To quantitatively validate the observed performance differences between DRL-RDG and RL-RDG, we conduct Mann–Whitney U tests. The statistical analysis results presented in

Table 4. Mann–Whitney U-test results (p-values) for performance metrics.

Number of Servers	30	40	50	60	70	50	50	50	50
Task Arrival Rate	30	40	50	60	70	40	45	55	60
Average Task Completion Time	0.038	0.025	0.011	0.004	0.047	0.038	0.026	0.021	0.035
Average CPU Utilization	0.032	0.041	0.028	0.015	0.023	0.041	0.033	0.024	0.016
Average Memory Utilization	0.035	0.037	0.031	0.026	0.019	0.039	0.028	0.017	0.014

demonstrate that all p-values are below the 0.05 significance level, confirming the statistical significance of our findings across all evaluation metrics and system scales. The statistical results confirm the performance advantages of our proposed DRL-RDG approach.

6. Conclusions

This paper investigates the joint optimization of resource defragmentation and task scheduling in queuing cloud systems. We first formulate the problem to minimize the average task completion time and maximize resource utilization, then transform it into an online decision problem. In online scenarios, decisions must be made in real time for dynamic task arrivals. We then propose the DRL-RDG approach, which is a two-layer iterative approach, to address the coupled challenges of online resource fragmentation and the task-scheduling problem. Specifically, DRL-RDG solves the task-scheduling subproblem and resource fragmentation subproblem through the deep Q-network algorithm and RDG. In addition, we propose the RL-RDG approach to highlight and substantiate the advantages of DRL-RDG. Simulation results show that DRL-RDG achieves a low average task completion time and high resource utilization, demonstrating its effectiveness in queuing cloud environments.

This study primarily relies on simulations to establish a controlled and scalable benchmarking environment. While the simulation parameters are designed to reflect realistic scenarios, we acknowledge that validation on a real-world testbed is an essential next step. Our immediate future work includes deploying and evaluating the proposed algorithm in a real application to further verify its practical efficacy.