Task Containerization and Container Placement Optimization for MEC: A Joint Communication and Computing Perspective

Abstract

Containers are used by an increasing number of Internet service providers to deploy their applications in multi-access edge computing (MEC) systems. Although container-based virtualization technologies significantly increase application availability, they may suffer expensive communication overhead and resource use imbalances. However, so far there has been a scarcity of studies to conquer these difficulties. In this paper, we design a workflow-based mathematical model for applications built upon interdependent multitasking composition, formulate a multi-objective combinatorial optimization problem composed of two subproblems—graph partitioning and multi-choice vector bin packing, and propose several joint task-containerization-and -container-placement methods to reduce communication overhead and balance multi-type computing resource utilization. The performance superiority of the proposed algorithms is demonstrated by comparison with the state-of-the-art task and container scheduling schemes.

1. Introduction

The container technique has been increasingly adopted in operating system virtualization by cloud computing and edge computing due to its high convenience and flexibility in deploying diverse applications [1]. The lightweight, scalable, and well-isolated environment provided by containers greatly increases the applications’ portability and availability [2], but at the cost of excessive communication overhead and resource utilization imbalance. Therefore, reducing communication overhead and balancing the usage of multi-type computing resources are critical design challenges for container-based multi-access edge computing (MEC).

Zhang et al. [3] studied the joint task scheduling and containerization in an edge computing system, where the tasks associated with a given application were first scheduled into processors deployed on an edge server, then an appropriate algorithm was invoked to determine the containerization scheme. As a result, multiple containers, each representing a standard unit of software for packaging code and all related dependencies, were created for executing different tasks on individual processors, while considering the time cost of inter-container communications. Workflow is a common model for describing microservice-based application’s tasks executed in containers. Bao et al. [4] presented a microservice-based workflow scheduling algorithm for minimizing the end-to-end delay of a given single application consisting of multiple microservices under a pre-specified budget constraint in a cloud, where the inter-container communication latency was considered a component of the end-to-end delay. A dual-line container placement algorithm was proposed in [5] to decide the container placement positions in a container cluster while being aware of the inter-container communication traffic. We note that all of the above contributions concentrated on reducing communication overhead while ignoring computing resource utilization problems. Naturally, allocating the containers of the same application into the same server is beneficial for reducing the cost of inter-container communications. However, this will cause serious resource utilization imbalance, since the same type of computing resource is usually used intensively by different containers of the same application, e.g., a CPU-intensive application [6]. Hence, without a proper resource-utilization balanced container scheduler, the particular server may suffer exponentially increasing response latency [7], and the total throughput of the system may be significantly reduced.

Therefore, it is important to also consider the balanced use of computing resources when designing scheduling algorithms. Since each task has a different computing resource requirement, Li et al. [8] considered the problem of scheduling microservice tasks of workflow applications to containers configured on on-demand virtual machines (VMs) and suggested a heuristic task scheduling algorithm for guaranteeing that the precedence-constrained or independent tasks are scheduled to available containers. As a benefit, the computing resources were more efficiently utilized. Ye et al. [9] explored the stochastic hybrid workflow scheduling problem to keep the cost of computing resources to the minimum, by jointly scheduling offline and online workflows. Although these contributions considerably improve the utilization efficiency of computing resources, they impose high communication overhead, because the containers of the same application have to exchange control messages and payload data. Therefore, the effectiveness of communications among containers has a significant impact on how well services are provided.

How to keep computing resource utilization balanced while reducing inter-container communication costs remains a grave challenge. In order to reduce the total inter-container communication traffic and increase the utilization efficiency of computing resources, Wu et al. [10] proposed a container placement strategy for containerized data centers, by exploiting the inter-container communication traffic pattern that obeys a Zipf-like distribution. They assumed that the major communication traffic originates from the containers that intensively communicate with each other. Each group of such containers is treated as an individual container block, which can be deployed either on the same server or on different servers, and the authors further assumed that the communication traffic between container blocks is negligible. Unfortunately, this assumption is unreasonable in the context of collaborative MEC servers. In [11], the computing resource requirements of containers are represented by a flow network, and container scheduling is formulated as a minimum-cost flow problem. To properly prioritize the execution of containers subject to a batch of concurrent requests, the proposed approach of [11] considered the average life cycle of containers as well as the affinity between the containers and the servers. Their solution took container affinity into account and placed containers with affinity on the same server. However, it is improper for scheduling container clusters with high dependencies. Lv et al. [6] studied container placement and container reassignment strategies to balance resource use in large-scale Internet data centers, where the initial container distribution is optimized further by reassigning containers among servers. However, as far as a large number of distributed MEC servers are concerned, migrating containers among these servers may incur significant communication overhead.

Furthermore, the widely used container orchestration platforms, such as Docker, only provide simple rule-based scheduling principles. Therefore, there has been a scarcity of efforts that focus on container scheduling algorithms capable of jointly reducing the communication overhead and balancing the computing resource usage in MEC.

Against the above backdrop, in this paper, our novel contributions are summarized as follows.

• We propose a task containerization and container placement optimization framework for applications running on MEC servers from a joint communication and computing perspective. The proposed framework comprises two modules. The task containerization module jointly considers low inter-container communication overhead and balanced multi-type computing resource requirements of containers. The container placement module places instantiated containers to the appropriate MEC servers by considering the balanced usage of the multi-type computing resources on MEC servers. The proposed framework is capable of achieving both low communication overhead and balanced computing resource requirement among containers, as well as balanced computing resource utilization among servers. To the best of our knowledge, our work is the first to investigate the impact of task partitioning, task containerization, and container placement on inter-container communication overhead and resource utilization balancing.
• We offer a workflow modeling method for highly interdependent tasks of an application and propose a mathematical model of the workflow to reflect the interactions among the tasks, the communication overhead, and the computing resources needed by each task. Based on the workflow model of tasks, we present a method for calculating the communication overhead and the utilization efficiency deviation of multi-type computing resources in the MEC-based computing network considered. The proposed model and methods are directly applicable to various workflow-based cloud computing and edge computing platforms.
• We evaluate the proposed task-containerization -and-container-placement algorithms in multiple aspects through extensive experiments, and compare them with state-of-the-art methods. Our experiments show that the proposed methods are capable of reducing the communication overhead by up to 74.10%, decreasing the normalized maximum load by up to 60.24%, improving the CPU utilization efficiency by up to 30.66%, and improving the memory utilization efficiency by up to 40.77% under the considered system configurations.

The rest of this paper is organized as follows. In Section 2, we present the MEC system model. In Section 3, we formulate the problem of jointly reducing the communication overhead and improving the degree of balance for multi-type computing resource utilization, which is solved by the algorithms proposed in Section 4, where the original problem is divided into two subproblems, namely the task containerization problem that is formulated as a graph partitioning problem [12], and the container placement optimization problem that is formulated as a multi-choice vector bin packing problem [13], which is a generalization of the classic vector bin packing problem. In Section 5, the proposed algorithms are compared with state-of-the-art methods by extensive simulations, and our conclusions are drawn in Section 6.

2. System Model

As shown in Figure 1, we consider the use case where Internet service providers set up and execute delay-sensitive applications in a computing network composed of multiple MEC servers. The MEC servers are deployed at the edge of a mobile network and connected to base stations (BSs). The user terminals (UTs) can access diverse delay-sensitive computing-intensive services, such as cloud gaming and cloud virtual reality (VR), provided by the MEC-based computing network through BSs. All the MEC servers are distributively connected to each other by optic fibers, so that their computing power can be shared via cooperation.

The MEC servers are represented by $\mathcal{S}=\{s_1,s_2,\cdots ,s_S\}$, where $S=\left|\mathcal{S}\right|$ is the number of servers. In the entire MEC service process, computing resources $\mathcal{R}=\{r_1,r_2,\cdots ,r_R\}$, such as CPU, memory and disk storage, are involved, where $R=\left|\mathcal{R}\right|$ is the number of resource types. For server $s_i\in \mathcal{S}$, let ${\gamma }_{s_i,r_k}$ denote the capacity of resource $r_k\in \mathcal{R}$. We consider an application composed of multiple interdependent tasks $\mathcal{T}=\{t_1,t_2,\cdots ,t_T\}$, and the number of tasks is $T=\left|\mathcal{T}\right|$. Let the matrix $\mathbf{E}={\left[e_{ij}\right]}_{T\times T}$ represent dependencies between tasks, and we have

$$e_{ij}=\left\{\begin{array}{cc}1,\hfill & \hfill \mathrm{if}\mathrm{task} t_i\mathrm{sends}\mathrm{data}\mathrm{to}\mathrm{task} t_j\hfill \\ 0.\hfill & \hfill \mathrm{otherwise}\hfill \end{array}\right.$$

(1)

The tasks are allocated to the container cluster $\mathcal{C}=\{c_1,c_2,\cdots ,c_C\}$, which comprises $C=\left|\mathcal{C}\right|$ containers. For container $c_i\in \mathcal{C}$, ${\beta }_{c_i,r_k}$ denotes the demand for resource $r_k\in \mathcal{R}$. For each task $t_i$, let vector ${\mathbf{v}}_i=(v_i^{\left(r_1\right)},v_i^{\left(r_2\right)},\cdots ,v_i^{\left(r_R\right)})$ represent the individual volumes of various resources required by task $t_i$. Note that in order to eliminate the difficulties encountered in evaluating the usage of heterogeneous resources, in this paper the volume of a specific type of resource required is normalized as the ratio of this partial volume to the total volume of the available resources of the same type across all the servers considered. The matrix $\mathbf{D}={\left[d_{ij}\right]}_{T\times C}$ represents the tasks deployment, where we have

$$d_{ij}=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}\mathrm{task}{t}_i\mathrm{is}\mathrm{assigned}\mathrm{to}\mathrm{container}{c}_j\\ \hfill 0.& \mathrm{otherwise}\end{array}\right.$$

(2)

The resource $r_k$ demanded by the container $c_j$ is the sum of the resources required by the tasks assigned to $c_j$, namely

$${\beta }_{c_j,r_k}=\sum _{i=1}^T{d}_{ij}{v}_i^{\left(r_k\right)}.$$

(3)

Let the matrix $\mathbf{M}={\left[m_{ij}\right]}_{C\times S}$ denote the container placement strategy, as expressed by

$$m_{ij}=\left\{\begin{array}{cc}\hfill 1,& \mathrm{if}\mathrm{container}{c}_i\mathrm{is}\mathrm{placed}\mathrm{in}\mathrm{server}{s}_j\\ \hfill 0.& \mathrm{otherwise}\end{array}\right.$$

(4)

3. Problem Formulation

3.1. Communication Overhead

We consider the communication overhead incurred by the communications between containers within an MEC-based computing network. Typically, the life cycle of a task includes three stages: receiving data, running algorithms, and sending output data. For example, task $t_i$ first receives the data and stores it in memory, then it runs algorithms upon obtaining all the input information it needs. Finally, when the execution is finished, task $t_i$ sends the output data to related tasks, which takes a transmission time of ${\tau }_{t_i}$. This is also the communication time between containers, once the tasks are assigned to containers.

Let $\mathbf{F}={\left[f_{ij}\right]}_{C\times C}$ denote the container dependency matrix. Upon assuming that the communication time between tasks assigned to the same container is negligible, the element of $\mathbf{F}$ can be expressed as:

$$f_{ij}=\left\{\begin{array}{cc}\hfill {\displaystyle \sum _q^T\sum _p^T}{d}_{pi}{\tau }_{t_p}{e}_{pq}{d}_{qj},& i\ne j\\ \hfill 0.& i=j\end{array}\right.$$

(5)

Similarly, we formulate the server dependency matrix as $\mathbf{G}={\left[g_{ij}\right]}_{S\times S}$, where we have

$$g_{ij}=\left\{\begin{array}{cc}\hfill {\displaystyle \sum _q^C\sum _p^C}{m}_{pi}{f}_{pq}{m}_{qj},& i\ne j\\ \hfill 0.& i=j\end{array}\right.$$

(6)

Upon assuming that the packet size and the number of packets required in each occurrence of communication remain constant, the communication overhead in the MEC-based computing network can be modeled as the ratio of the time spent in sending data between servers to the total communication time, namely

$$C_{\mathrm{OH}}(\mathbf{D},\mathbf{M})=\frac{{\displaystyle \sum _i^S\sum _j^S}{g}_{ij}}{{\sum }_{i=1}^T{\tau }_{t_i}}.$$

(7)

Computing Resource Utilization Balance

The efficiency of servers is often degraded by the unbalanced use of the various computing resources deployed on the servers. The resource $r_k$ used by all the containers residing in the server $s_i$ is quantified by

$${\beta }_{s_i,r_k}(\mathbf{D},\mathbf{M})=\sum _{j=1}^C{m}_{ji}{\beta }_{c_j,r_k}.$$

(8)

Since the total available resource $r_k$ on the server $s_i$ is ${\gamma }_{s_i,r_k}$. Then, $u_{s_i,r_k}(\mathbf{D},\mathbf{M})=\frac{{\beta }_{s_i,r_k}(\mathbf{D},\mathbf{M})}{{\gamma }_{s_i,r_k}}$ represents the utilization efficiency of resource $r_k$ on server $s_i$. Typically, if a server’s consumption of different types of computing resources is relatively balanced, the server is able to host more containers and more applications. Then, the balance degree of multi-type computing resource utilization on server $s_i$ is given by

$$b_i=\sum _{k=1}^R\frac{{\left(u_{s_i,r_k}(\mathbf{D},\mathbf{M})-{\overline{u}}_{s_i}\right)}^2}{R},$$

(9)

where ${\overline{u}}_{s_i}$ is the mean utilization efficiency of all resources on server $s_i$.

This metric reflects the deviation of the utilization efficiency of all resource types on a server from the average resource utilization efficiency. The total resource balance degree of the MEC-based computing network is the sum of the resource balance degree of all servers, and it is defined as

$$B_{\mathrm{tot}}(\mathbf{D},\mathbf{M})=\sum _{i=1}^S{b}_i.$$

(10)

Therefore, the joint optimization of the communication overhead and the degree of resource utilization balance is formulated as a weighted-sum minimization problem of

$$\begin{array}{cc}\hfill \mathsf{P}1:\underset{\mathbf{D},\mathbf{M}}{min}& {\mu }_C{C}_{\mathrm{OH}}(\mathbf{D},\mathbf{M})+{\mu }_B{B}_{\mathrm{tot}}(\mathbf{D},\mathbf{M})\hfill \end{array}$$

(11a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& \mathbf{D}\in {\{0,1\}}^{T\times C},\hfill \end{array}$$

(11b)

$$\begin{array}{cc}\hfill & \mathbf{M}\in {\{0,1\}}^{C\times S},\hfill \end{array}$$

(11c)

$$\begin{array}{cc}\hfill & \sum _{j=1}^C{d}_{ij}=1,\hfill \end{array}$$

(11d)

$$\begin{array}{cc}\hfill & \sum _{j=1}^S{m}_{ij}=1,\hfill \end{array}$$

(11e)

$$\begin{array}{cc}\hfill & 0\le {\beta }_{s_i,r_k}(\mathbf{D},\mathbf{M})\le {\gamma }_{s_i,r_k}.\hfill \end{array}$$

(11f)

In the above optimization problem, ${\tau }_{t_i}\ge 0$, $v_t^{\left(r_k\right)}\ge 0$, $0\le {\mu }_C\le 1$, $0\le {\mu }_B\le 1$, $t_i\in \mathcal{T}$, $s_i\in \mathcal{S}$, $r_k\in \mathcal{R}$, and $c_j\in \mathcal{C}$ are all constants known a priori. Here, ${\tau }_{t_i}\ge 0$ indicates that the transmission time of each task is non-negative; $v_t^{\left(r_k\right)}\ge 0$ indicates that each task has a non-negative demand for each type of resource; while $0\le {\mu }_C\le 1$ and $0\le {\mu }_B\le 1$ specify the value range of the weights corresponding to each of the component objectives. (11b) and (11c) indicate that the optimization variables are matrices composed of binary-value elements. Equations (11d) and (11e) imply that each given task $t_i$ and given container $c_i$ have to be allocated to a single container and a single server, respectively. Furthermore, (11f) represents that the resource $r_k$ used by all the containers residing in the server $s_i$ must not exceed the capacity of the resource $r_k$ on the server $s_i$.

It is worth noting that reducing communication overhead may lead to putting more tasks into fewer containers and packing as many containers as possible into fewer servers. However, this may result in a lower utilization efficiency of computing resources in the servers. In other words, there is a trade-off between the communication overhead $C_{\mathrm{OH}}$ and the computing resource utilization efficiency $B_{\mathrm{tot}}$. We attempt to balance $C_{\mathrm{OH}}$ and $B_{\mathrm{tot}}$ rather than cram as many items as possible in a container or a server.

The problem $\mathsf{P}1$ is essentially a multi-objective combinatorial optimization problem involving two subproblems, namely graph partitioning [12] that is corresponding to task containerization and multi-choice vector bin packing [13] that is corresponding to container placement, as shown in Figure 1. Both of them are known NP-hard problems. Hence, the existence of any polynomial time optimal solution is ruled out unless $\mathsf{P}=\mathsf{NP}$. Furthermore, as mentioned before, the component objectives $C_{\mathrm{OH}}$ and $B_{\mathrm{tot}}$ may conflict with each other, and both of them rely on the optimization variables $\mathbf{D}$ and $\mathbf{M}$. Theoretically, the optimal solution to $\mathsf{P}1$ can be found by brute-force search. However, in practice, there may be a large number of tasks, containers, and servers to deal with. Hence, the computational complexity of obtaining the optimal solution to the joint optimization problem $\mathsf{P}1$ is prohibitive. Alternatively, it is possible to solve $\mathsf{P}1$ more efficiently by using an iterative method, where at each iteration, the objective of $\mathsf{P}1$ is minimized with respect to one of the two matrix variables $\mathbf{D}$ and $\mathbf{M}$ while the other matrix variable is held fixed. Nevertheless, the deployment of this strategy may require a dedicated centralized computing server to run the scheduling algorithms and is not naturally aligned with the physically sequential processes of an MEC-based computing network, where the task containerization takes place first and is then followed by container placement. In other words, in practice, it is meaningful to first obtain $\mathbf{D}$ and then obtain $\mathbf{M}$ relying on the result of $\mathbf{D}$.

With this in mind, we turn our attention to a degenerate version of $\mathsf{P}1$ so that the reformulated problem is more naturally matched to the practical sequential process of deploying an application to servers. As a result, a non-iterative sequential strategy can be invoked, which is expected to find good solutions capable of optimizing both the inter-container communication overhead and the degree of resource utilization balance at the expense of lower computational complexity. More specifically, the goal of task containerization is to minimize the weighted-sum of the inter-container communication overhead and the average deviation of multi-type resource demands by each container. Thus, we have

$$\begin{array}{cc}\hfill \mathsf{P}2:\underset{\mathbf{D}}{min}& {\mu }_C\frac{{\displaystyle \sum _{i=1}^C\sum _{j=1}^C}{f}_{ij}}{{\displaystyle \sum _{i=1}^T}{\tau }_{t_i}}+{\mu }_B{\displaystyle \sum _{k=1}^R}{\displaystyle \sum _{j=1}^C}\frac{{({\beta }_{c_j,r_k}-{\overline{\beta }}_{r_k})}^2}{C}\hfill \end{array}$$

(12a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& \mathbf{D}\in {\{0,1\}}^{T\times C},\hfill \end{array}$$

(12b)

$$\begin{array}{cc}\hfill & {\displaystyle \sum _{j=1}^C}{d}_{ij}=1,\hfill \end{array}$$

(12c)

where ${\overline{\beta }}_{r_k}$ represents the average value of all containers’ demand for resource $r_k$. As pointed out before, dividing tasks into groups and putting each group of tasks into a container can essentially be formulated as a graph partitioning problem [12]. Different from the traditional treatment of graph partitioning problems, in this paper we not only take into account the weights of edges, which represent the communication overhead but also consider the weights of vertices, which represent the resource requirements of a particular task.

Furthermore, we simplify the container placement problem without considering the communication overhead caused by inter-container communications and the communication time between servers in the objective function. This is because the servers’ geographic locations have a significant impact on the communication overhead between servers and it can be difficult to ascertain the servers’ locations. After the task containerization module obtains $\mathbf{D}$, the container placement module is intended to minimize the deviation of the utilization efficiency of various computing resources on the MEC servers, hence the container placement problem is formulated as:

$$\begin{array}{cc}\hfill \mathsf{P}3:\underset{\mathbf{M}}{min}& {\mu }_B\sum _{i=1}^S\sum _{k=1}^R\frac{{(u_{s_i,r_k}\left(\mathbf{M}\right)-{\overline{u}}_{s_i})}^2}{R}\hfill \end{array}$$

(13a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& \mathbf{M}\in {\{0,1\}}^{C\times S},\hfill \end{array}$$

(13b)

$$\begin{array}{cc}\hfill & \sum _{j=1}^S{m}_{ij}=1,\hfill \end{array}$$

(13c)

$$\begin{array}{cc}\hfill & 0\le u_{s_i,r_k}\left(\mathbf{M}\right)\le 1.\hfill \end{array}$$

(13d)

The container placement problem is similar to the multi-choice vector bin packing problem [13]. In contrast to the traditional multidimensional vector bin packing problem, we have a set number of servers and a range of resource capacities for each server. The components of our vector set come from the containers obtained during the task containerization stage. Container values are not 0 or 1, but rather a vector of distinct forms of resource requirements as containers have different types of computing resource requirements. Our objective is more complicated—instead of minimizing the number of servers, placing containers into MEC servers requires matching the resources the containers need with what the servers can provide.

On the basis of making references to classic problems, it is vital to create solutions that are more fitting for our objectives. The fourth chapter will provide detailed explanations of the specific approach.

4. Task Containerization and Container Placement Schemes

As shown in Figure 1, the entire process of performing edge computing in an MEC-based computing network can be sequentially divided into two stages: first allocating tasks to containers, and then placing containers onto servers. To find a high-accuracy approximate solution to the problem (11), we can consider the task containerization and the container placement sequentially.

4.1. Task Containerization Method

A task containerization method is invoked to allocate tasks to containers. First of all, it is necessary to balance all types of resources needed by each container for achieving an equalized consumption of resources on each server. To this end, it is preferable to avoid grouping tasks that have a high demand for the same type of resource together. On the other hand, it is recommended to put tasks that have a large amount of data transmitted between each other into the same container and then place the containers having frequent communications with each other on the same server, in order to reduce the communication overhead.

We use workflow to describe the relationship between different tasks of an application. Workflow is an important concept for describing cloud computing and edge computing applications. It consists of multiple interdependent tasks which are bound together through data or functional dependencies. As shown in Figure 2, an application can be built as a workflow, where each task implements certain functionality and collaborates with the others. In Figure 2 the workflow is modelled, as a directed acyclic graph (DAG) $\mathcal{G}=(\mathcal{T},\mathcal{E})$. Each vertex $t_i\in \mathcal{T}$ represents a task, and the weight vector of vertex $t_i$ is ${\mathrm{v}}_i$, which represents various resources required by $t_i$. The adjacency matrix of the DAG is $\mathbf{E}$ and its elements are defined by (1), where the weights of edges are not considered. The number of edges is denoted as E, i.e., $\left|\mathcal{E}\right|=E$, which equals the number of nonzero elements in $\mathbf{E}$. Furthermore, let $\omega \left(e_{ij}\right)$ denote the weight of $e_{ij}$ and it represents the length of communication time from task i to task j. Then, the elements $e_{ij}$ of $\mathbf{E}$ are updated as $\omega \left(e_{ij}\right)$.

For a cluster of vertices $\mathcal{A}\subseteq \mathcal{T}$, let $\mathcal{E}(\mathcal{A},\mathcal{A})$ denote the set of edges with all vertices in $\mathcal{A}$, and let $\mathcal{E}(\mathcal{A},\mathcal{T}\setminus \mathcal{A})$ represent the collection of edges connecting the two disjoint vertex sets of the graph cut $(\mathcal{A},\mathcal{T}\setminus \mathcal{A})$. Furthermore, define a vertex set $\mathcal{P}=\left({\mathcal{A}}_1,{\mathcal{A}}_2,\cdots ,{\mathcal{A}}_C\right)$ as multiple disjoint vertex sets whose union is $\mathcal{T}$, i.e., we have ${\bigcup }_{i=1}^C{\mathcal{A}}_i=\mathcal{T}$ and ${\mathcal{A}}_i\cap {\mathcal{A}}_j=\varnothing$, $\forall i\ne j$, $C\ge 1$. Furthermore, we denote with $v_{{\mathcal{A}}_i}^{\left(k\right)}$ the demand of resource $r_k$ by the subset ${\mathcal{A}}_i$.

It has been established that the graph partitioning problem is NP-hard. For finding approximate solutions to the graph partitioning problem considered more efficiently, we propose two heuristic algorithms in this paper, i.e., the partitioning with non-critical path based initialization (P-NCPI) and the partitioning with random initialization (P-RI). The major operations of the proposed partitioning algorithms include determining the number of disjoint vertex sets, assigning one vertex to each of the disjoint vertex sets in the initial partition, and setting a scoring function for the remaining vertices. Both vertex weights and edge weights are taken into account in the scoring function, and each vertex is exclusively assigned to one of the disjoint vertex sets based on its score. Typically, these operations can be implemented in an appropriate container orchestration software, such as Kubernetes or KubeEdge. This process guarantees that the vertices are partitioned so that the edge weights between the disjoint vertex sets are minimal and that the vertex weights within the disjoint vertex sets are not excessive.

The NCPI and RI algorithms employed in our initial partition are described as follows.

• NCPI: Tasks are sorted topologically, and the critical path is chosen as the one with the highest weight. The tasks that comprise the critical path are known as essential tasks, and they determine the minimum completion time of an application. Due to the high communication overhead between essential tasks, we first select the required number of non-essential tasks and then assign each non-essential task to a different disjoint vertex set as an initial partition to reduce communication overhead between sets.
• RI: Selecting the desired tasks randomly according to the number of disjoint vertex sets and assigning each of them to a different disjoint vertex set as an initial partition.

To more conveniently describe the process of grouping the remaining vertices, we consider an alternative objective function with two components: the cost of inter-partition edge weights $C_e$ and the cost of intra-partition vertex weights $C_v$. For each given set of vertices $\mathcal{P}=\left({\mathcal{A}}_1,{\mathcal{A}}_2,\cdots ,{\mathcal{A}}_C\right)$ and a specific resource $r_k$, this objective function is defined as:

$$\begin{array}{cc}\hfill f\left(\mathcal{P}\right)& ={\mu }_C{C}_e(\mathsf{\Omega }\left(\mathcal{E}({\mathcal{A}}_1,\mathcal{T}\setminus {\mathcal{A}}_1)\right),\cdots ,\mathsf{\Omega }\left(\mathcal{E}({\mathcal{A}}_C,\mathcal{T}\setminus {\mathcal{A}}_C)\right))\hfill \\ \hfill & +{\mu }_B{C}_v(v_{{\mathcal{A}}_1}^{\left(k\right)},\cdots ,v_{{\mathcal{A}}_C}^{\left(k\right)}),\hfill \end{array}$$

(14)

where $\mathsf{\Omega }\left(\mathcal{E}({\mathcal{A}}_i,\mathcal{T}\setminus {\mathcal{A}}_i)\right)$ represents the sum of weights corresponding to the edges contained in the set $\mathcal{E}({\mathcal{A}}_i,\mathcal{T}\setminus {\mathcal{A}}_i)$, and we have

$$C_e(\mathsf{\Omega }\left(\mathcal{E}({\mathcal{A}}_1,\mathcal{T}\setminus {\mathcal{A}}_1)\right),\cdots ,\mathsf{\Omega }\left(\mathcal{E}({\mathcal{A}}_C,\mathcal{T}\setminus {\mathcal{A}}_C)\right))={\sum }_{i=1}^C\mathsf{\Omega }(\mathcal{E}({\mathcal{A}}_i,\mathcal{T}\setminus {\mathcal{A}}_i)),$$

(15)

$$C_v(v_{{\mathcal{A}}_1}^{\left(k\right)},\cdots ,v_{{\mathcal{A}}_C}^{\left(k\right)})={\sum }_{i=1}^{C}g\left(v_{{\mathcal{A}}_i}^{\left(k\right)}\right),$$

(16)

with $g\left(v_{{\mathcal{A}}_i}^{\left(k\right)}\right)$ representing an increasing convex function of the vertex weights in set ${\mathcal{A}}_i$, and $C_v(·)$ being a linear function for balancing the cost of vertex weights across different sets ${\mathcal{A}}_i$, $i=1,2,\cdots ,C$.

It should be noted that the optimization variable in the objective function (12a) is $\mathbf{D}$, while that in (14) is $\mathcal{P}$. Both $\mathbf{D}$ and $\mathcal{P}$ represent the mapping of tasks to containers, and both the expression ${\sum }_{i=1}^C{\sum }_{j=1}^C{f}_{ij}$ in (12a) and the expression $C_e(·)$ in (14) characterize the communication time between containers. However, the differences are: (1) ${\sum }_{i=1}^C{\sum }_{j=1}^C{f}_{ij}$ in (12a) is further normalized by dividing the total communication time of all the tasks, while $C_e(·)$ is a direct summation of $\mathsf{\Omega }\left(\mathcal{E}({\mathcal{A}}_i,\mathcal{T}\setminus {\mathcal{A}}_i)\right)$, $i=1,2,\cdots ,C$; (2) ${\sum }_{k=1}^R{\sum }_{j=1}^C\frac{1}{C}{({\beta }_{c_j,r_k}-{\overline{\beta }}_{r_k})}^2$ in (12a) is the average deviation of multi-type resource demands by each container, while $C_v(·)$ in (14) is the total demand for computing resources by all the containers. Obviously, when the demands for multi-type computing resources by the individual containers are equal, the average deviation of multi-type resource demands by each container reaches the minimum. Therefore, (12a) and (14) can be seen as different mathematical models for describing the same objective. Compared with (12a), Equation (14) directly expresses the communication time and computing resource requirements as specific analytical expressions of the variables ${\mathcal{A}}_i$, $i=1,2,\cdots ,C$, thus making it more convenient to derive the solution algorithm from the graph partitioning perspective.

According to Equation (14), the task containerization method can be derived by solving the following optimization problem:

$$\begin{array}{cc}\hfill \mathsf{P}4:\underset{\mathcal{P}=({\mathcal{A}}_1,\cdots ,{\mathcal{A}}_C)}{min}f\left(\mathcal{P}\right)& ={\sum }_{i=1}^C{\mu }_C\mathsf{\Omega }\left(\mathcal{E}({\mathcal{A}}_i,\mathcal{T}\setminus {\mathcal{A}}_i)\right)\hfill \\ \hfill & +{\sum }_{i=1}^C{\mu }_{B}g\left(v_{{\mathcal{A}}_i}^{\left(k\right)}\right)\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& {\bigcup }_{i=1}^C{\mathcal{A}}_i=\mathcal{T},\hfill \\ \hfill & {\mathcal{A}}_i\cap {\mathcal{A}}_j=\varnothing ,\forall i\ne j.\hfill \end{array}$$

(17)

For the convex function $g\left(x\right)$, based on [12], we select the family of functions $c\left(x\right)=\alpha x^{\theta }$, where $\alpha >0$ and $\theta \ge 1$. The parameter $\theta$ controls the balance degree of vertex weights between partitioned sets. The greater the value of $\theta$, the greater the cost of imbalanced set weights. $\theta =1$ means that the imbalance between the demand of resource $r_k$ by the set ${\mathcal{A}}_i$, i.e., $v_{{\mathcal{A}}_i}^{\left(k\right)}$, is ignored. We choose $\alpha =E\frac{C^{\theta -1}}{T^{\theta }}$ to serve as a proper scaling factor. The optimization problem $\mathsf{P}4$ is then written as:

$$\begin{array}{cc}\hfill \mathsf{P}5:\underset{\mathcal{P}=({\mathcal{A}}_1,\cdots ,{\mathcal{A}}_C)}{min}f\left(\mathcal{P}\right)& =\frac{{\sum }_{i=1}^C{\mu }_C\mathsf{\Omega }\left(\mathcal{E}({\mathcal{A}}_i,\mathcal{T}\setminus {\mathcal{A}}_i)\right)}{E}\hfill \\ \hfill & +\frac{1}{C}{\sum }_{i=1}^C{\mu }_B{\left(\frac{v_{{\mathcal{A}}_i}^{\left(k\right)}}{\frac{T}{C}}\right)}^{\theta }\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& {\bigcup }_{i=1}^C{\mathcal{A}}_i=\mathcal{T},\hfill \\ \hfill & {\mathcal{A}}_i\cap {\mathcal{A}}_j=\varnothing ,\forall i\ne j.\hfill \end{array}$$

(18)

Furthermore, we define a function h as:

$$\begin{array}{cc}\hfill h\left(\mathcal{P}\right)& ={\sum }_{i=1}^C\left({\mu }_C\mathsf{\Omega }\left(\mathcal{E}({\mathcal{A}}_i,{\mathcal{A}}_i)\right)-{\mu }_{B}g\left(v_{{\mathcal{A}}_i}^{\left(k\right)}\right)\right)\hfill \\ \hfill & ={\sum }_{i=1}^C\left({\mu }_C\mathsf{\Omega }(\mathcal{E}(\mathcal{T},\mathcal{T})-\mathcal{E}({\mathcal{A}}_i,\mathcal{T}\setminus {\mathcal{A}}_i))-{\mu }_{B}g\left(v_{{\mathcal{A}}_i}^{\left(k\right)}\right)\right)\hfill \\ \hfill & =C·{\mu }_C\mathsf{\Omega }(\mathcal{E}(\mathcal{T},\mathcal{T})-f\left(\mathcal{P}\right).\hfill \end{array}$$

(19)

Therefore, the problem of minimizing $f\left(\mathcal{P}\right)$ can be solved by maximizing $h\left(\mathcal{P}\right)$. For any $1\le i<j\le C$, if $h({\mathcal{A}}_1,\cdots ,{\mathcal{A}}_i\cup \left\{t\right\},\cdots ,{\mathcal{A}}_j\setminus \left\{t\right\},\cdots ,{\mathcal{A}}_C)\ge h({\mathcal{A}}_1,\cdots ,{\mathcal{A}}_i\setminus \left\{t\right\},\cdots ,{\mathcal{A}}_j\cup \left\{t\right\},\cdots ,{\mathcal{A}}_C)$, vertex t is assigned to vertex set $A_i$. Based on this condition, we define a difference function

$$\begin{array}{cc}\hfill \mathsf{\Delta }h(t,{\mathcal{A}}_i)=& h({\mathcal{A}}_1,\cdots ,{\mathcal{A}}_i\cup \left\{t\right\},\cdots ,{\mathcal{A}}_j\setminus \left\{t\right\},\cdots ,{\mathcal{A}}_C)\hfill \\ \hfill & -h({\mathcal{A}}_1,\cdots ,{\mathcal{A}}_i,\cdots ,{\mathcal{A}}_j,\cdots ,{\mathcal{A}}_C),\hfill \end{array}$$

(20)

and assign vertex t to vertex set ${\mathcal{A}}_i$ when we have $\mathsf{\Delta }h(t,{\mathcal{A}}_i)\ge \mathsf{\Delta }h(t,{\mathcal{A}}_j)$. Let $N\left(t\right)$ denote the neighbors of vertex t (i.e., the vertices directly connected to vertex t). According to Equation (19), the difference function is rewritten as:

$$\begin{array}{cc}\hfill \mathsf{\Delta }h(t,{\mathcal{A}}_i)=& {\mu }_C\mathsf{\Omega }\left(\mathcal{E}(N\left(t\right)\cap {\mathcal{A}}_i,N\left(t\right)\cap {\mathcal{A}}_i)\right)-{\mu }_C\mathsf{\Omega }\left(\mathcal{E}(N\left(t\right)\cap {\mathcal{A}}_j,N\left(t\right)\cap {\mathcal{A}}_j)\right)\hfill \\ \hfill & +{\mu }_B\left(g\left(v_{{\mathcal{A}}_i}^{\left(k\right)}\right)+g\left(v_{{\mathcal{A}}_j}^{\left(k\right)}\right)-g\left(v_{{\mathcal{A}}_i\cup \left\{t\right\}}^{\left(k\right)}\right)-g\left(v_{{\mathcal{A}}_j\setminus \left\{t\right\}}^{\left(k\right)}\right)\right).\hfill \end{array}$$

(21)

The details of the proposed task containerization method are shown in Algorithm 1. First, tasks are selected by the RI or NCPI algorithm for initial partitioning (Lines 2–35 of Algorithm 1). Once the initial partitioning has been determined, the remaining tasks are partitioned using the scores derived from the above calculations (Lines 36–47 of Algorithm 1), thus ensuring that the communication cost and the multi-type computing resource balancing cost are minimized during the task containerization stage.

In order to measure whether the resources required for each vertex set are balanced, we define the measurement parameter $\lambda$ as the normalized maximum load:

$$\lambda =C·\frac{{max}_j\{{\sum }_{k=1}^R{\beta }_{c_j,r_k}|c_j\in \mathcal{C}\}}{{\sum }_{j=1}^C{\sum }_{k=1}^R{\beta }_{c_j,r_k}}.$$

(22)

The balance degree of the usage of multi-type computing resources improves as $\lambda$ approaches 1.

4.2. Container Placement Method

When tasks are divided into multiple vertex sets, each set is a container, and the resources are available to each container. We conduct a comparative study of container placement methods by using the dot product (DP) algorithm [14] and the first fit decreasing (FFD) algorithm [14] to individually solve the problem $\mathcal{P}3$.

DP considers the demands of multi-type computing resources by each container as well as the capacities of multi-type resources on each server. Any server is listed as a candidate server for a container if it has enough multi-type resources to accommodate the container. For each container $c_i$ and one of its candidate servers $s_j$, the dot product of multi-type resources required by the container and multi-type resources that the server can provide is denoted as ${\mathsf{DP}}_{ij}$. The larger the value of ${\mathsf{DP}}_{ij}$, the more resources server $s_j$ is able to supply to container $c_i$. To improve computing resource utilization, we tend to place the container on the server that has the greatest amount of resources, hence we use ${\mathsf{DP}}_{ij}$ as the matching degree between demand and supply, and for servers that are not candidates, we set the matching degree as 0. Thus we have

$${\mathsf{DP}}_{ij}=\left\{\begin{array}{cc}\hfill \sum _{k=1}^R{\beta }_{c_i,r_k}·{\gamma }_{s_j,r_k},& \mathrm{if}{\beta }_{c_i,r_k}<{\gamma }_{s_j,r_k}\\ \hfill 0.& \mathrm{otherwise}\end{array}\right.$$

(23)

The procedure of the DP algorithm is shown in Algorithm 2.

FFD is a greedy algorithm in which the containers are sorted in decreasing order according to their individual weight that is determined by each container’s demands for multi-type resources, and then containers are placed sequentially in the first server that has sufficient capacity to accommodate them. As the resources required by each container are multidimensional (i.e., multi-type), the definition of dimensions and the associated weight assigned to the vector of resources required by an individual container determine the order in which the containers are placed. Containers have different resource requirements and there are multiple alternative methods for selecting an appropriate weight for each container. For example, one can calculate the weighted sum of multi-type resources required by the container as the basis for the decreasing ordering. In particular, when the weights are equal, containers are decreasingly ordered by evaluating ${\sum }_{k=1}^R{\beta }_{c_j,r_k}$, $c_j\in \mathcal{C}$. Additionally, if the demands of a particular type of resource always dominate the demands of the other types of resources, one can only consider the dominant type of resource to determine the weights of containers and the decreasing order. Then, upon traversing the servers, the containers are sequentially placed on the first server that can satisfy their requirements of resources. The details of FFD are presented in Algorithm 3.

To sum up, the process of the whole task containerization and container placement scheme is as follows: An application’s tasks are first grouped by the P-NCPI or the P-RI algorithm, and then each group of tasks, denoted by ${\mathcal{A}}_i$, is encapsulated into a container $c_i$, which is then placed on the selected server by using the DP or the FFD algorithm. In this manner, the application is eventually executed in a given number of containers and on the selected servers. As a beneficial result, the proposed scheme exhibits minimized inter-container communication cost (some containers are possibly placed on different servers), balanced resource requirements among containers, and balanced resource utilization efficiency among the selected servers.

Algorithm 1 The proposed task containerization algorithm

Input: $\mathcal{G}=(\mathcal{T},\mathcal{E})$, $\mathcal{R}$, C, ${\left\{{\mathbf{v}}_i\right\}}_{i=1}^{\left|\mathcal{T}\right|}$ Output: $\mathcal{P}$, $\mathbf{D}$ 1: Initialization: $\mathcal{P}=\left({\mathcal{A}}_1=\varnothing ,{\mathcal{A}}_2=\varnothing ,\cdots ,{\mathcal{A}}_C=\varnothing \right)$ 2: if RI is the initial partitioning algorithm then 3:    select C tasks in $\mathcal{T}$ randomly 4:    allocate each of the C tasks exclusively to all ${\mathcal{A}}_i$, $i=1,2,\cdots ,C$ 5: end if 6: if NCPI is the initial partitioning algorithm then 7:    $\mathbf{z}\leftarrow$ sort vertices in $\mathcal{G}$ topologically 8:    if $\mathcal{G}$ has a loop then 9:      raise Error 10:    end if 11:    Generate a $\left|\mathbf{z}\right|$-dimensional null vector $\mathbf{h}$ 12:    for $i=1$ to $\left|\mathcal{T}\right|$ do 13:      ${\mathcal{N}}_{z_i}\leftarrow$ the indices of the adjacency vertices of $z_i$ 14:      for $j=1$ to $|{\mathcal{N}}_{z_i}|$ do 15:         if $h_{{\mathcal{N}}_{z_i}\left[j\right]}<h_{{\mathcal{N}}_{z_i}\left[j\right]}+\omega \left(e_{{\mathcal{N}}_{z_i}\left[j\right],z_i}\right)$ then 16:           $h_{{\mathcal{N}}_{z_i}\left[j\right]}\leftarrow h_{{\mathcal{N}}_{z_i}\left[j\right]}+\omega \left(e_{{\mathcal{N}}_{z_i}\left[j\right],z_i}\right)$ 17:         end if 18:      end for 19:    end for 20:    Initialize a vector $\mathbf{u}$ with all elements being $max\left\{h_i\right|h_i\in \mathbf{h}\}$ 21:    Reverse the elements order of $\mathbf{z}$ 22:    Repeat Lines 12-19 to obtain $\mathbf{u}$ 23:    for $i=1$ to $\left|\mathcal{T}\right|$ do 24:      ${\mathcal{N}}_i\leftarrow$ the indices of the adjacency vertices of vertex i 25:      for $j=1$ to $|{\mathcal{N}}_i|$ do 26:         $l\leftarrow h_i$ 27:         $v\leftarrow u_i-\omega \left(e_{{\mathcal{N}}_i\left[j\right],i}\right)$ 28:         if $l==v$ then 29:           include $e_{{\mathcal{N}}_i\left[j\right],i}$ into the critical path 30:         end if 31:      end for 32:    end for 33:    Select C tasks on the non-critical path 34:    Allocate each of the C tasks exclusively to all ${\mathcal{A}}_i$, $i=1,2,\cdots ,C$ 35: end if 36: Put the remaining $\left|\mathcal{T}\right|-C$ tasks into the set $\mathcal{Y}$ 37: for $j=1$ to $\left|\mathcal{T}\right|-C$ do 38:    score $\leftarrow 0$ 39:    for $i=1$ to C do 40:      calculate $\mathsf{\Delta }h(\mathcal{Y}\left[j\right],{\mathcal{A}}_i)$ by Equation (21) 41:      if $\mathsf{\Delta }h(\mathcal{Y}\left[j\right],{\mathcal{A}}_i)>$ score then 42:         score $\leftarrow \mathsf{\Delta }h(\mathcal{Y}\left[j\right],{\mathcal{A}}_i)$ 43:         $k\leftarrow i$ 44:      end if 45:    end for 46:    Put task $\mathcal{Y}\left[j\right]$ into ${\mathcal{A}}_k$ 47: end for 48: Generate $\mathbf{D}$ according to $\mathcal{P}$

Algorithm 2 The DP algorithm

Input: $\left|\mathcal{T}\right|$, $\mathcal{C}$, $\mathcal{S}$, $\mathcal{R}$, ${\left\{{\mathbf{v}}_i\right\}}_{i=1}^{\left|\mathcal{T}\right|}$, $\mathbf{D}$ Output: $\mathbf{M}$ 1: Initialization: Generate a null matrix $\mathbf{M}={\mathbf{0}}_{\left|\mathcal{C}\right|\times \left|\mathcal{S}\right|}$ 2: for $i=1$ to $\left|\mathcal{C}\right|$ do 3:    for $j=1$ to $\left|\mathcal{S}\right|$ do 4:      if server $s_j$ can provide all the resources required by container $c_i$ then 5:         ${\mathsf{DP}}_{ij}\leftarrow$ calculate the dot product of multi-type resources required by container $c_i$ and multi-type resources that server $s_j$ can provide 6:      else 7:         ${\mathsf{DP}}_{ij}\leftarrow 0$ 8:      end if 9:    end for 10:    Select the server $s_j$ with the largest ${\mathsf{DP}}_{ij}$ to accommodate the container $c_i$ 11:    $m_{ij}\leftarrow 1$ 12: end for

Algorithm 3 The FFD algorithm

Input: $\left|\mathcal{T}\right|$, $\mathcal{C}$, $\mathcal{S}$, $\mathcal{R}$, ${\left\{{\mathbf{v}}_i\right\}}_{i=1}^{\left|\mathcal{T}\right|}$, $\mathbf{D}$ Output: $\mathbf{M}$ 1: Initialization: Generate a null matrix $\mathbf{M}={\mathbf{0}}_{\left|\mathcal{C}\right|\times \left|\mathcal{S}\right|}$ 2: Sorts the containers in decreasing order by evaluating ${\sum }_{k=1}^R{\beta }_{c_j,r_k}$ of each container $c_j$ 3: for $i=1$ to $\left|\mathcal{C}\right|$ do 4:    for $j=1$ to $\left|\mathcal{S}\right|$ do 5:      if server $s_j$ can provide all the resources required by container $c_i$ then 6:         select server $s_j$ to place container $c_i$ 7:         $m_{ij}\leftarrow 1$ 8:      end if 9:      if $m_{ij}==1$ then 10:         break 11:      end if 12:    end for 13: end for

4.3. Analysis of Algorithm Complexity

The task containerization method groups the tasks of the given application into multiple vertex sets, each of which is encapsulated into a single container. Then the generated containers are placed on the selected appropriate servers by using the container placement method.

More specifically, two initial partitioning algorithms, i.e., P-NCPI and P-RI, are individually employed in the task containerization method. P-RI initializes a given number of vertex sets, which are supposed to represent the task partitioning, by using the same number of randomly selected tasks (see Lines 2-5 of Algorithm 1). The time complexity of P-RI is $O\left(T^2+2T\right)=O\left(C\right)+O\left(C\right)+O\left(T-C\right)+O\left(T(T-C)\right)+O\left(T-C\right)+O\left(TC\right)$, and the number of floating point operations is given by $1+C+C+T-C+T-C+2T(T-C)+T-C+TC=2T^2-TC+3T-C+1$. P-NCPI employs the non-critical path based initialization to conduct the graph partitioning (see Lines 6-35 of Algorithm 1). Its time complexity is $O\left(4T^2+2T\right)=O\left(3T^2\right)+O\left(2C\right)+O\left(T-C\right)+O\left(T(T-C)\right)+O\left(T-C\right)+O\left(TC\right)$, and the number of floating point operations is given by $1+T^2+1+1+T+T^2+1+T+T^2+C+C+T-C+T-C+2T(T-C)+T-C+TC=5T^2-TC+5T-C+4$.

As far as the container placement method is concerned, two bin packing algorithms are used, namely DP (see Algorithm 2) and FFD (see Algorithm 3). The time complexity of DP is $O\left(CS\right)$ and the number of floating point operations is $1+CS+2C$. The time complexity of FFD is $O\left(CR+CS+C\right)$, and the number of floating point operations is $1+CR+2CS+C$.

5. Evaluation

In this section, we present simulation results to demonstrate the effectiveness of our proposed task containerization and container placement methods. The experiments were carried out on a MacBook Pro with a 4-core CPU, 8GB of RAM, and the macOS Monterey operating system. All the algorithms presented in this paper were implemented by Python 3.9.7. For the P-RI algorithm and the K-means algorithm, we took the average of 10 sets of experimental results to obtain statistically reliable results. Moreover, all the results were calculated relying on the experiments of 10 application programs, each of which is represented by a workflow composed of a different number of tasks: 5, 9, 17, 24, 30, 47, 57, 63, 91 and 93.

We considered the CPU and memory resources of a computing network comprising 10 MEC servers. The server configurations are listed in

Table 1. Configuration of MEC servers.

Server	CPU (%)	Memory (%)
0	7	7
1	9	8
2	10	8
3	12	11
4	6	11
5	12	14
6	14	8
7	10	11
8	9	14
9	11	8

, where the volume of each type of computing resources is normalized. We set the parameters ${\mu }_C$ as 0.5, ${\mu }_B$ as 0.5, and $\theta$ as 1.5. The used data regarding the application tasks were obtained from the Alibaba Cloud platform [15]. Each container hosts at least one process, and each task is set up as a separate process. In general, resources allocated to a container are more than the container needs [16]. We ignored the discrepancy between the volume of resources allocated to the container and the volume of resources demanded by the container for the convenience of statistics and visualization.

The balance degree of utilizing computing resources is a critical performance indicator for MEC. To avoid the problem of long request delay caused by insufficient resources of the servers, the utilization of multi-type computing resources on the individual servers has to be balanced. In Figure 3, we compared the normalized maximum load $\lambda$ (defined by Equation (22)) of the proposed task partitioning algorithms and of the benchmarking K-means clustering method. Specifically, we observe in Figure 3 that the normalized maximum load $\lambda$ of the K-means method [17] varies greatly between 1.6 and 4. Unlike those of the K-means method, the values of $\lambda$ for P-NCPI and P-RI remain between 1.25 and 1, indicating that the resource requirements by each group of tasks are more balanced after conducting task partitioning with our proposed algorithms (e.g., decreasing the normalized maximum load by up to 60.24% on average when using P-NCPI).

In Figure 4 and Figure 5, we compared the proposed task-containerization-and-container-placement methods with the state-of-the-art strategy “Spread” [16] employed by the container orchestration tool Docker swarm, in terms of the average CPU and the memory (i.e., MEM) utilization efficiencies in the MEC-based computing network. Specifically, the average CPU utilization efficiency of the MEC-based computing network is defined as

$${\lambda }_{\mathrm{CPU}}=\frac{{\sum }_n^{N_{\mathrm{S}}}{\eta }_n}{N_{\mathrm{S}}},$$

(24)

where $N_{\mathrm{S}}$ represents the number of servers occupied by the application, ${\eta }_n$ denotes the CPU utilization efficiency of server n. Similarly, the average MEM utilization efficiency of the MEC-based computing network is defined as

$${\lambda }_{\mathrm{MEM}}=\frac{{\sum }_n^{N_{\mathrm{S}}}{\zeta }_n}{N_{\mathrm{S}}},$$

(25)

where ${\zeta }_n$ represents the MEM utilization efficiency of server n.

We see from Figure 4 and Figure 5 that the proposed P-NCPI-FFD algorithm improves the average CPU utilization efficiency by 30.66% and the average MEM utilization efficiency by 40.77%, compared with the Spread algorithm. In general, the proposed P-NCPI-FFD algorithm achieves the highest CPU and MEM utilization efficiency among all algorithms considered. The FFD algorithm attains higher resources utilization efficiency than the DP algorithm. This phenomenon can be explained as follows. The DP algorithm focuses on finding out the largest sum product of a container’s multi-type resource requirements and the volume of multi-type resources that a server can provide, whereas FFD puts as many containers having higher resource requirements as possible on the minimum possible number of servers. Therefore, when using FFD, ${\eta }_n$ and ${\zeta }_n$ in Equations (24) and (25) are larger, while $N_{\mathrm{S}}$ is smaller, than those when employing DP. In addition, it is observed that the impact of the P-NCPI and P-RI algorithms on the results of CPU and MEM utilization is trivial.

For a given application, the metric defined by Equation (10) can reflect the extent to which different resources are used in a balanced manner. According to Figure 6, in most cases evaluated, our proposed algorithms outperform the Spread algorithm in terms of the balance degree of utilizing multi-type computing resources. Although the DP algorithm is generally inferior to the FFD algorithm in terms of the CPU and MEM utilization efficiencies (as shown by Figure 4 and Figure 5), it performs better than FFD in terms of the balance degree of utilizing multi-type computing resources across the MEC-based computing network. The P-NCPI and P-RI algorithms, which are invoked for the initial task partitioning, do not make much difference in this respect.

The running time results of all the task-containerization-and-container-placement algorithms considered are provided in Figure 7. It is observed that the running time of the P-RI algorithm is shorter than that of the P-NCPI algorithm. Additionally, when the number of tasks exceeds a particular value, the DP algorithm has a shorter running time than the FFD algorithm. Among the four algorithms we proposed, the P-RI-DP algorithm has the shortest running time, and in this regard, there is very little difference between the P-RI-DP algorithm and the Spread algorithm. Notably, the Spread algorithm always performs best in terms of the running time.

Finally, the results of the communication overhead for various task-containerization-and-container-placement algorithms are shown in Figure 8. We can see that when compared with the benchmarking Spread algorithm, the P-NCPI-FFD algorithm saves 74.10% of the communication overhead, while the P-RI-FFD algorithm saves 59.32%. The communication overhead of FFD is lower than that of DP, because it does not have to consider the balance between the resource demand of containers and the resource supply of servers.

To sum up, our proposed task-containerization-and-container-placement algorithms outperform the existing algorithms in general. In particular, all of our proposed algorithms exhibit a more balanced resource requirements by containers, the P-NCPI-FFD algorithm achieves the highest CPU and MEM utilization efficiencies, the P-NCPI-DP and P-RI-DP algorithms achieve a higher degree of balance in utilizing multi-type computing resources, while the P-NCPI-FFD and P-RI-FFD algorithms perform better in terms of communication overhead. Additionally, different initial task partitioning algorithms, such as P-NCPI and P-RI, do not make much difference in terms of the above performance metrics.

6. Conclusions

In this paper we have proposed four task-containerization-and-container-placement algorithms to jointly reduce the inter-container communication overhead and balance the multi-type computing resource utilization in the MEC-based computing network. We establish a workflow model to reflect the interactions between interdependent tasks. As a beneficial result, the inter-task communication overhead, the inter-container communication overhead, as well as the computing resources required by each group of tasks encapsulated in a container, are conveniently characterized. Furthermore, two task containerization algorithms have been designed to reduce the inter-container communication overhead while balancing the multi-type computing resource requirements of containers. Then we proposed a pair of container placement algorithms for optimizing the utilization of multi-type computing resources across MEC servers. Extensive simulation results demonstrated that our proposed methods are capable of reducing the inter-container communication overhead by up to 74.10%, reducing the normalized maximum load by up to 60.24%, improving the CPU utilization efficiency by up to 30.66%, and improving the memory utilization efficiency by up to 40.77% in the MEC-based computing network considered.