4.1. Task Containerization Method
A task containerization method is invoked to allocate tasks to containers. First of all, it is necessary to balance all types of resources needed by each container for achieving an equalized consumption of resources on each server. To this end, it is preferable to avoid grouping tasks that have a high demand for the same type of resource together. On the other hand, it is recommended to put tasks that have a large amount of data transmitted between each other into the same container and then place the containers having frequent communications with each other on the same server, in order to reduce the communication overhead.
We use workflow to describe the relationship between different tasks of an application. Workflow is an important concept for describing cloud computing and edge computing applications. It consists of multiple interdependent tasks which are bound together through data or functional dependencies. As shown in
Figure 2, an application can be built as a workflow, where each task implements certain functionality and collaborates with the others. In
Figure 2 the workflow is modelled, as a directed acyclic graph (DAG)
$\mathcal{G}=(\mathcal{T},\mathcal{E})$. Each vertex
${t}_{i}\in \mathcal{T}$ represents a task, and the weight vector of vertex
${t}_{i}$ is
${\mathrm{v}}_{i}$, which represents various resources required by
${t}_{i}$. The adjacency matrix of the DAG is
$\mathbf{E}$ and its elements are defined by (
1), where the weights of edges are not considered. The number of edges is denoted as
E, i.e.,
$\left\mathcal{E}\right=E$, which equals the number of nonzero elements in
$\mathbf{E}$. Furthermore, let
$\omega \left({e}_{ij}\right)$ denote the weight of
${e}_{ij}$ and it represents the length of communication time from task
i to task
j. Then, the elements
${e}_{ij}$ of
$\mathbf{E}$ are updated as
$\omega \left({e}_{ij}\right)$.
For a cluster of vertices $\mathcal{A}\subseteq \mathcal{T}$, let $\mathcal{E}(\mathcal{A},\mathcal{A})$ denote the set of edges with all vertices in $\mathcal{A}$, and let $\mathcal{E}(\mathcal{A},\mathcal{T}\setminus \mathcal{A})$ represent the collection of edges connecting the two disjoint vertex sets of the graph cut $(\mathcal{A},\mathcal{T}\setminus \mathcal{A})$. Furthermore, define a vertex set $\mathcal{P}=\left({\mathcal{A}}_{1},{\mathcal{A}}_{2},\cdots ,{\mathcal{A}}_{C}\right)$ as multiple disjoint vertex sets whose union is $\mathcal{T}$, i.e., we have ${\bigcup}_{i=1}^{C}{\mathcal{A}}_{i}=\mathcal{T}$ and ${\mathcal{A}}_{i}\cap {\mathcal{A}}_{j}=\varnothing $, $\forall i\ne j$, $C\ge 1$. Furthermore, we denote with ${v}_{{\mathcal{A}}_{i}}^{\left(k\right)}$ the demand of resource ${r}_{k}$ by the subset ${\mathcal{A}}_{i}$.
It has been established that the graph partitioning problem is NPhard. For finding approximate solutions to the graph partitioning problem considered more efficiently, we propose two heuristic algorithms in this paper, i.e., the partitioning with noncritical path based initialization (PNCPI) and the partitioning with random initialization (PRI). The major operations of the proposed partitioning algorithms include determining the number of disjoint vertex sets, assigning one vertex to each of the disjoint vertex sets in the initial partition, and setting a scoring function for the remaining vertices. Both vertex weights and edge weights are taken into account in the scoring function, and each vertex is exclusively assigned to one of the disjoint vertex sets based on its score. Typically, these operations can be implemented in an appropriate container orchestration software, such as Kubernetes or KubeEdge. This process guarantees that the vertices are partitioned so that the edge weights between the disjoint vertex sets are minimal and that the vertex weights within the disjoint vertex sets are not excessive.
The NCPI and RI algorithms employed in our initial partition are described as follows.
NCPI: Tasks are sorted topologically, and the critical path is chosen as the one with the highest weight. The tasks that comprise the critical path are known as essential tasks, and they determine the minimum completion time of an application. Due to the high communication overhead between essential tasks, we first select the required number of nonessential tasks and then assign each nonessential task to a different disjoint vertex set as an initial partition to reduce communication overhead between sets.
RI: Selecting the desired tasks randomly according to the number of disjoint vertex sets and assigning each of them to a different disjoint vertex set as an initial partition.
To more conveniently describe the process of grouping the remaining vertices, we consider an alternative objective function with two components: the cost of interpartition edge weights
${C}_{e}$ and the cost of intrapartition vertex weights
${C}_{v}$. For each given set of vertices
$\mathcal{P}=\left({\mathcal{A}}_{1},{\mathcal{A}}_{2},\cdots ,{\mathcal{A}}_{C}\right)$ and a specific resource
${r}_{k}$, this objective function is defined as:
where
$\mathsf{\Omega}\left(\mathcal{E}({\mathcal{A}}_{i},\mathcal{T}\setminus {\mathcal{A}}_{i})\right)$ represents the sum of weights corresponding to the edges contained in the set
$\mathcal{E}({\mathcal{A}}_{i},\mathcal{T}\setminus {\mathcal{A}}_{i})$, and we have
with
$g\left({v}_{{\mathcal{A}}_{i}}^{\left(k\right)}\right)$ representing an increasing convex function of the vertex weights in set
${\mathcal{A}}_{i}$, and
${C}_{v}(\xb7)$ being a linear function for balancing the cost of vertex weights across different sets
${\mathcal{A}}_{i}$,
$i=1,2,\cdots ,C$.
It should be noted that the optimization variable in the objective function (
12a) is
$\mathbf{D}$, while that in (
14) is
$\mathcal{P}$. Both
$\mathbf{D}$ and
$\mathcal{P}$ represent the mapping of tasks to containers, and both the expression
${\sum}_{i=1}^{C}{\sum}_{j=1}^{C}{f}_{ij}$ in (
12a) and the expression
${C}_{e}(\xb7)$ in (
14) characterize the communication time between containers. However, the differences are: (1)
${\sum}_{i=1}^{C}{\sum}_{j=1}^{C}{f}_{ij}$ in (
12a) is further normalized by dividing the total communication time of all the tasks, while
${C}_{e}(\xb7)$ is a direct summation of
$\mathsf{\Omega}\left(\mathcal{E}({\mathcal{A}}_{i},\mathcal{T}\setminus {\mathcal{A}}_{i})\right)$,
$i=1,2,\cdots ,C$; (2)
${\sum}_{k=1}^{R}{\sum}_{j=1}^{C}\frac{1}{C}{({\beta}_{{c}_{j},{r}_{k}}{\overline{\beta}}_{{r}_{k}})}^{2}$ in (
12a) is the average deviation of multitype resource demands by each container, while
${C}_{v}(\xb7)$ in (
14) is the total demand for computing resources by all the containers. Obviously, when the demands for multitype computing resources by the individual containers are equal, the average deviation of multitype resource demands by each container reaches the minimum. Therefore, (
12a) and (
14) can be seen as different mathematical models for describing the same objective. Compared with (
12a), Equation (
14) directly expresses the communication time and computing resource requirements as specific analytical expressions of the variables
${\mathcal{A}}_{i}$,
$i=1,2,\cdots ,C$, thus making it more convenient to derive the solution algorithm from the graph partitioning perspective.
According to Equation (
14), the task containerization method can be derived by solving the following optimization problem:
For the convex function
$g\left(x\right)$, based on [
12], we select the family of functions
$c\left(x\right)=\alpha {x}^{\theta}$, where
$\alpha >0$ and
$\theta \ge 1$. The parameter
$\theta $ controls the balance degree of vertex weights between partitioned sets. The greater the value of
$\theta $, the greater the cost of imbalanced set weights.
$\theta =1$ means that the imbalance between the demand of resource
${r}_{k}$ by the set
${\mathcal{A}}_{i}$, i.e.,
${v}_{{\mathcal{A}}_{i}}^{\left(k\right)}$, is ignored. We choose
$\alpha =E\frac{{C}^{\theta 1}}{{T}^{\theta}}$ to serve as a proper scaling factor. The optimization problem
$\mathsf{P}4$ is then written as:
Furthermore, we define a function
h as:
Therefore, the problem of minimizing
$f\left(\mathcal{P}\right)$ can be solved by maximizing
$h\left(\mathcal{P}\right)$. For any
$1\le i<j\le C$, if
$h({\mathcal{A}}_{1},\cdots ,{\mathcal{A}}_{i}\cup \left\{t\right\},\cdots ,{\mathcal{A}}_{j}\setminus \left\{t\right\},\cdots ,{\mathcal{A}}_{C})\ge h({\mathcal{A}}_{1},\cdots ,{\mathcal{A}}_{i}\setminus \left\{t\right\},\cdots ,{\mathcal{A}}_{j}\cup \left\{t\right\},\cdots ,{\mathcal{A}}_{C})$, vertex
t is assigned to vertex set
${A}_{i}$. Based on this condition, we define a difference function
and assign vertex
t to vertex set
${\mathcal{A}}_{i}$ when we have
$\mathsf{\Delta}h(t,{\mathcal{A}}_{i})\ge \mathsf{\Delta}h(t,{\mathcal{A}}_{j})$. Let
$N\left(t\right)$ denote the neighbors of vertex
t (i.e., the vertices directly connected to vertex
t). According to Equation (
19), the difference function is rewritten as:
The details of the proposed task containerization method are shown in Algorithm 1. First, tasks are selected by the RI or NCPI algorithm for initial partitioning (Lines 2–35 of Algorithm 1). Once the initial partitioning has been determined, the remaining tasks are partitioned using the scores derived from the above calculations (Lines 36–47 of Algorithm 1), thus ensuring that the communication cost and the multitype computing resource balancing cost are minimized during the task containerization stage.
In order to measure whether the resources required for each vertex set are balanced, we define the measurement parameter
$\lambda $ as the normalized maximum load:
The balance degree of the usage of multitype computing resources improves as
$\lambda $ approaches 1.
4.2. Container Placement Method
When tasks are divided into multiple vertex sets, each set is a container, and the resources are available to each container. We conduct a comparative study of container placement methods by using the dot product (DP) algorithm [
14] and the first fit decreasing (FFD) algorithm [
14] to individually solve the problem
$\mathcal{P}3$.
DP considers the demands of multitype computing resources by each container as well as the capacities of multitype resources on each server. Any server is listed as a candidate server for a container if it has enough multitype resources to accommodate the container. For each container
${c}_{i}$ and one of its candidate servers
${s}_{j}$, the dot product of multitype resources required by the container and multitype resources that the server can provide is denoted as
${\mathsf{DP}}_{ij}$. The larger the value of
${\mathsf{DP}}_{ij}$, the more resources server
${s}_{j}$ is able to supply to container
${c}_{i}$. To improve computing resource utilization, we tend to place the container on the server that has the greatest amount of resources, hence we use
${\mathsf{DP}}_{ij}$ as the matching degree between demand and supply, and for servers that are not candidates, we set the matching degree as 0. Thus we have
The procedure of the DP algorithm is shown in Algorithm 2.
FFD is a greedy algorithm in which the containers are sorted in decreasing order according to their individual weight that is determined by each container’s demands for multitype resources, and then containers are placed sequentially in the first server that has sufficient capacity to accommodate them. As the resources required by each container are multidimensional (i.e., multitype), the definition of dimensions and the associated weight assigned to the vector of resources required by an individual container determine the order in which the containers are placed. Containers have different resource requirements and there are multiple alternative methods for selecting an appropriate weight for each container. For example, one can calculate the weighted sum of multitype resources required by the container as the basis for the decreasing ordering. In particular, when the weights are equal, containers are decreasingly ordered by evaluating ${\sum}_{k=1}^{R}{\beta}_{{c}_{j},{r}_{k}}$, ${c}_{j}\in \mathcal{C}$. Additionally, if the demands of a particular type of resource always dominate the demands of the other types of resources, one can only consider the dominant type of resource to determine the weights of containers and the decreasing order. Then, upon traversing the servers, the containers are sequentially placed on the first server that can satisfy their requirements of resources. The details of FFD are presented in Algorithm 3.
To sum up, the process of the whole task containerization and container placement scheme is as follows: An application’s tasks are first grouped by the PNCPI or the PRI algorithm, and then each group of tasks, denoted by
${\mathcal{A}}_{i}$, is encapsulated into a container
${c}_{i}$, which is then placed on the selected server by using the DP or the FFD algorithm. In this manner, the application is eventually executed in a given number of containers and on the selected servers. As a beneficial result, the proposed scheme exhibits minimized intercontainer communication cost (some containers are possibly placed on different servers), balanced resource requirements among containers, and balanced resource utilization efficiency among the selected servers.
Algorithm 1 The proposed task containerization algorithm 
Input: $\mathcal{G}=(\mathcal{T},\mathcal{E})$, $\mathcal{R}$, C, ${\left\{{\mathbf{v}}_{i}\right\}}_{i=1}^{\left\mathcal{T}\right}$ Output: $\mathcal{P}$, $\mathbf{D}$
 1:
Initialization: $\mathcal{P}=\left({\mathcal{A}}_{1}=\varnothing ,{\mathcal{A}}_{2}=\varnothing ,\cdots ,{\mathcal{A}}_{C}=\varnothing \right)$  2:
if RI is the initial partitioning algorithm then  3:
select C tasks in $\mathcal{T}$ randomly  4:
allocate each of the C tasks exclusively to all ${\mathcal{A}}_{i}$, $i=1,2,\cdots ,C$  5:
end if  6:
if NCPI is the initial partitioning algorithm then  7:
$\mathbf{z}\leftarrow $ sort vertices in $\mathcal{G}$ topologically  8:
if $\mathcal{G}$ has a loop then  9:
raise Error  10:
end if  11:
Generate a $\left\mathbf{z}\right$dimensional null vector $\mathbf{h}$  12:
for $i=1$ to $\left\mathcal{T}\right$ do  13:
${\mathcal{N}}_{{z}_{i}}\leftarrow $ the indices of the adjacency vertices of ${z}_{i}$  14:
for $j=1$ to ${\mathcal{N}}_{{z}_{i}}$ do  15:
if ${h}_{{\mathcal{N}}_{{z}_{i}}\left[j\right]}<{h}_{{\mathcal{N}}_{{z}_{i}}\left[j\right]}+\omega \left({e}_{{\mathcal{N}}_{{z}_{i}}\left[j\right],{z}_{i}}\right)$ then  16:
${h}_{{\mathcal{N}}_{{z}_{i}}\left[j\right]}\leftarrow {h}_{{\mathcal{N}}_{{z}_{i}}\left[j\right]}+\omega \left({e}_{{\mathcal{N}}_{{z}_{i}}\left[j\right],{z}_{i}}\right)$  17:
end if  18:
end for  19:
end for  20:
Initialize a vector $\mathbf{u}$ with all elements being $max\left\{{h}_{i}\right{h}_{i}\in \mathbf{h}\}$  21:
Reverse the elements order of $\mathbf{z}$  22:
Repeat Lines 1219 to obtain $\mathbf{u}$  23:
for $i=1$ to $\left\mathcal{T}\right$ do  24:
${\mathcal{N}}_{i}\leftarrow $ the indices of the adjacency vertices of vertex i  25:
for $j=1$ to ${\mathcal{N}}_{i}$ do  26:
$l\leftarrow {h}_{i}$  27:
$v\leftarrow {u}_{i}\omega \left({e}_{{\mathcal{N}}_{i}\left[j\right],i}\right)$  28:
if $l==v$ then  29:
include ${e}_{{\mathcal{N}}_{i}\left[j\right],i}$ into the critical path  30:
end if  31:
end for  32:
end for  33:
Select C tasks on the noncritical path  34:
Allocate each of the C tasks exclusively to all ${\mathcal{A}}_{i}$, $i=1,2,\cdots ,C$  35:
end if  36:
Put the remaining $\left\mathcal{T}\rightC$ tasks into the set $\mathcal{Y}$  37:
for $j=1$ to $\left\mathcal{T}\rightC$ do  38:
score $\leftarrow 0$  39:
for $i=1$ to C do  40:
calculate $\mathsf{\Delta}h(\mathcal{Y}\left[j\right],{\mathcal{A}}_{i})$ by Equation ( 21)  41:
if $\mathsf{\Delta}h(\mathcal{Y}\left[j\right],{\mathcal{A}}_{i})>$ score then  42:
score $\leftarrow \mathsf{\Delta}h(\mathcal{Y}\left[j\right],{\mathcal{A}}_{i})$  43:
$k\leftarrow i$  44:
end if  45:
end for  46:
Put task $\mathcal{Y}\left[j\right]$ into ${\mathcal{A}}_{k}$  47:
end for  48:
Generate $\mathbf{D}$ according to $\mathcal{P}$

Algorithm 2 The DP algorithm 
Input: $\left\mathcal{T}\right$, $\mathcal{C}$, $\mathcal{S}$, $\mathcal{R}$, ${\left\{{\mathbf{v}}_{i}\right\}}_{i=1}^{\left\mathcal{T}\right}$, $\mathbf{D}$ Output: $\mathbf{M}$
 1:
Initialization: Generate a null matrix $\mathbf{M}={\mathbf{0}}_{\left\mathcal{C}\right\times \left\mathcal{S}\right}$  2:
for $i=1$ to $\left\mathcal{C}\right$ do  3:
for $j=1$ to $\left\mathcal{S}\right$ do  4:
if server ${s}_{j}$ can provide all the resources required by container ${c}_{i}$ then  5:
${\mathsf{DP}}_{ij}\leftarrow $ calculate the dot product of multitype resources required by container ${c}_{i}$ and multitype resources that server ${s}_{j}$ can provide  6:
else  7:
${\mathsf{DP}}_{ij}\leftarrow 0$  8:
end if  9:
end for  10:
Select the server ${s}_{j}$ with the largest ${\mathsf{DP}}_{ij}$ to accommodate the container ${c}_{i}$  11:
${m}_{ij}\leftarrow 1$  12:
end for

Algorithm 3 The FFD algorithm 
Input: $\left\mathcal{T}\right$, $\mathcal{C}$, $\mathcal{S}$, $\mathcal{R}$, ${\left\{{\mathbf{v}}_{i}\right\}}_{i=1}^{\left\mathcal{T}\right}$, $\mathbf{D}$ Output: $\mathbf{M}$
 1:
Initialization: Generate a null matrix $\mathbf{M}={\mathbf{0}}_{\left\mathcal{C}\right\times \left\mathcal{S}\right}$  2:
Sorts the containers in decreasing order by evaluating ${\sum}_{k=1}^{R}{\beta}_{{c}_{j},{r}_{k}}$ of each container ${c}_{j}$  3:
for $i=1$ to $\left\mathcal{C}\right$ do  4:
for $j=1$ to $\left\mathcal{S}\right$ do  5:
if server ${s}_{j}$ can provide all the resources required by container ${c}_{i}$ then  6:
select server ${s}_{j}$ to place container ${c}_{i}$  7:
${m}_{ij}\leftarrow 1$  8:
end if  9:
if ${m}_{ij}==1$ then  10:
break  11:
end if  12:
end for  13:
end for

4.3. Analysis of Algorithm Complexity
The task containerization method groups the tasks of the given application into multiple vertex sets, each of which is encapsulated into a single container. Then the generated containers are placed on the selected appropriate servers by using the container placement method.
More specifically, two initial partitioning algorithms, i.e., PNCPI and PRI, are individually employed in the task containerization method. PRI initializes a given number of vertex sets, which are supposed to represent the task partitioning, by using the same number of randomly selected tasks (see Lines 25 of Algorithm 1). The time complexity of PRI is $O\left(\right)open="("\; close=")">{T}^{2}+2T+O\left(\right)open="("\; close=")">T(TC)$, and the number of floating point operations is given by $1+C+C+TC+TC+2T(TC)+TC+TC=2{T}^{2}TC+3TC+1$. PNCPI employs the noncritical path based initialization to conduct the graph partitioning (see Lines 635 of Algorithm 1). Its time complexity is $O\left(\right)open="("\; close=")">4{T}^{2}+2T+O\left(\right)open="("\; close=")">2C$, and the number of floating point operations is given by $1+{T}^{2}+1+1+T+{T}^{2}+1+T+{T}^{2}+C+C+TC+TC+2T(TC)+TC+TC=5{T}^{2}TC+5TC+4$.
As far as the container placement method is concerned, two bin packing algorithms are used, namely DP (see Algorithm 2) and FFD (see Algorithm 3). The time complexity of DP is $O\left(\right)open="("\; close=")">CS$ and the number of floating point operations is $1+CS+2C$. The time complexity of FFD is $O\left(\right)open="("\; close=")">CR+CS+C$, and the number of floating point operations is $1+CR+2CS+C$.