Elastic Scheduling of Scientiﬁc Workﬂows under Deadline Constraints in Cloud Computing Environments

: Scientiﬁc workﬂow applications are collections of several structured activities and ﬁne-grained computational tasks. Scientiﬁc workﬂow scheduling in cloud computing is a challenging research topic due to its distinctive features. In cloud environments, it has become critical to perform efﬁcient task scheduling resulting in reduced scheduling overhead, minimized cost and maximized resource utilization while still meeting the user-speciﬁed overall deadline. This paper proposes a strategy, Dynamic Scheduling of Bag of Tasks based workﬂows (DSB), for scheduling scientiﬁc workﬂows with the aim to minimize ﬁnancial cost of leasing Virtual Machines (VMs) under a user-deﬁned deadline constraint. The proposed model groups the workﬂow into Bag of Tasks (BoTs) based on data dependency and priority constraints and thereafter optimizes the allocation and scheduling of BoTs on elastic, heterogeneous and dynamically provisioned cloud resources called VMs in order to attain the proposed method’s objectives. The proposed approach considers pay-as-you-go Infrastructure as a Service (IaaS) clouds having inherent features such as elasticity, abundance, heterogeneity and VM provisioning delays. A trace-based simulation using benchmark scientiﬁc workﬂows representing real world applications, demonstrates a signiﬁcant reduction in workﬂow computation cost while the workﬂow deadline is met. The results validate that the proposed model produces better success rates to meet deadlines and cost efﬁciencies in comparison to adapted state-of-the-art algorithms for similar problems.


Introduction
Various scientific domains such as biology, medicine, planetary science, astronomy, physics, bioinformatics and environmental science, often involves the use of simulations of large-scale complex applications for validating behavior of different real-world activities.Most of such scientific applications are constructed as workflows containing a set of computational tasks linked via control and data dependencies.The workflow is partitioned into multiple tasks which may have different sizes and require different running times ranging from tens of seconds to multiple minutes [1].Workflow scheduling is a process of mapping workflow tasks to processing resources called Virtual Machines (VMs) and managing their execution while satisfying all dependencies, constraints and objective functions.It is well-known that workflow scheduling problems are Nondeterministic Polynomial time (NP)-complete [2], so finding the perfect solution in polynomial time is not viable in all cases.Efficiently executing such workflows within a reasonable amount of time usually require massive storage and large-scale distributed computing infrastructures such as cluster, grid, or cloud.Among such infrastructures, cloud computing has emerged as an economical and scalable distributed computing environment for the design and execution of large and complex scientific workflow applications [3].However, cloud environments may incur significant computing overheads that can adversely affect the overall performance and costs of the workflow execution [4].In real cloud environment, some factors can cause delays.The overall workflow execution time and scheduling time overheads may include the additional times for queuing delay, workflow engine delay, tasks grouping delay, data transfer overhead, resource allocation, resource preparation, VM provisioning and deprovisioning delays, task runtime variance, VM boot time, VM idle times and other unexpected delays in real environments.IaaS providers make no guarantees on the value of these delays and it can have higher or lower variability and impact on the execution of a workflow.Data communication overheads may be critical in case of data-intensive workflows.For instance, the execution of two dependent tasks on one less powerful resource but with zero or negligible transfer overhead would be more efficient than the execution on two more powerful but separate resources with significant transfer overhead.Therefore, in order to minimize the overheads incurred, the most challenging problems in workflow schedulers are the minimization of execution time and monetary budget of workflow execution and capability of dynamic adaptation to any unexpected delays.
The majority of existing strategies which focus on both of these objectives simultaneously, lack in consideration of one or more crucial aspects of workflow scheduling.For example, dynamic provisioning of resources is not considered in [5][6][7][8], scalability in terms of large number of tasks is not considered in [9], heterogeneity of resources is not considered in [8,10], resource auto-scaling is not considered in [11], data dependencies are not considered in [12] and task clustering technique in [5] is not fully autonomous.Moreover, unlike multiple independent BoTs or single task-based workflows, the concept of using multiple connected and constrained BoTs for reducing the data transfer time is not considered in most existing scheduling algorithms [13,14].
To address the limitations of previous research, we propose a dynamic and scalable cost-efficient deadline constrained algorithm for scheduling workflows using dynamic provisioning of resources in a single cloud environment, using CPLEX solver (https://www.ibm.com/analytics/data-science/prescriptive-analytics/cplex-optimizer) and Mixed Integer Programming (MIP).We call our algorithm as Dynamic Scheduling of Bag of Tasks based workflows (DSB).The main contributions of this study are summarized as follows.
(1) Grouping homogeneous tasks with the same depth into BoTs according to their priorities and dependency constraints in order to minimize queuing overheads and then distributing the overall workflow deadline over different BoTs.(2) Implementation of a technique for scheduling of tasks on dynamic and scalable set of VMs in order to optimize cost while satisfying their deadlines.(3) The proposed algorithm involves dynamic provisioning of resources as MIP problem by the use of IBM ILOG CPLEX.(4) Extensive simulations with results for real world scientific applications.
The rest of the paper is organized as follows.Section 2 provides an overview of the related work.Section 3 presents the scientific workflow application model, the cloud resource model and the workflow execution model assumed in this work.Section 4 provides the proposed scheduling algorithm.Section 5 presents results and finally Section 6 concludes the paper.

Related Work
In the last two decades, several research studies have been conducted that address the problem of workflow scheduling due to its NP-completeness.The Quality of Service (QoS)-constrained workflow scheduling algorithms tries to maximize the performance while meeting other user-defined QoS constraints, for example, cost minimization of a workflow execution under deadline constraints, as in IC-PCP and IC-PCP2 [15], URH [16], DPDS [17] and EPSM [18].[19] considered an application model similar to the one addressed in this paper and proposed a scheduling algorithm for optimizing a workflow's execution time (that is, makespan) under a budget constraint.They proposed resource provisioning plan for a subset of the workflow tasks as a mixed integer linear programming (MILP) model.However, their solution is tailored for fine-grained billing periods such as per-minute billing.Moreover, the algorithm's objective is to minimize the makespan instead of meeting a deadline constraint.Despite these differences, we consider their work relevant as the authors not only consider dynamic resource provisioning and delays but also consider several VM instance types with different characteristics and prices.

Rodriguez and Buyya
Dziok et al. [20] presented an adaptive algorithm and used the MIP approach to schedule workflows in IaaS clouds for optimizing cost under a deadline constraint.However, they do not consider reusing of already assigned VMs and also do not consider data transfer time.
Malawski et al. [8] presented a dynamic resource provisioning and scheduling algorithm called DPDS.It schedules workflows under given deadline and budget constraints along with the information about resource utilization for VM provisioning and scheduling.However, it attempts to maximize the number of completed workflows rather than minimizing the rental cost of a single BoT workflow.Moreover, it supports single instance type and do not consider resource heterogeneity which is in contrast to the current IaaS cloud model that offer a wide variety of instance types and dynamic provisioning and scheduling of resources.
Task granularity has been addressed in several research studies for reducing the impact of overheads that may arise when executing scientific workflows in distributed environments, such as the cloud [4,12,21].Task grouping methods reduce computational granularity by reducing the number of computational activities by grouping fine-grained tasks into course-grained tasks.These techniques attempt to minimize queuing times and scheduling overheads when resources are limited.However, task granularity may limit the degree of inherent parallelism, therefore it must be done optimally.
Mao and Humphrey [9] proposed an auto-scaling mechanism that schedules workflow jobs in order to minimize the monetary cost while meeting the application deadlines on clouds.A static resource provisioning plan is made based on a global optimization heuristic and then adapted to dynamic changes by running the global optimization algorithm every time a task is scheduled.However, they do not consider the different priorities of each job and considered dynamic and unpredictable workloads of workflow.Furthermore, the high computational overhead in Scaling-Consolidation-Scheduling (SCS) hinders its scalability in terms of the number of tasks in the workflow and they do not provide a near-optimal solution.
Malawski et al. [22] explicate a deadline and budget constraint scheduling algorithm which tries to maximize the amount of work completed.The scheduling algorithm proposed by Byun et al. [10] estimated the minimum number of resources needed to execute the workflow in a cost-effective way.But, they considered a single type of VM and fails to consider the heterogeneous nature of clouds.
The proposed Dynamic Scheduling of Bag of Tasks based workflows (DSB) algorithm covers all of these deficits and presents a dynamic and scalable cost-efficient deadline constrained algorithm for scheduling workflows using dynamic provisioning of resources in a single cloud environment.

System Model
This section presents the application model, cloud resource model and the overall workflow model for the execution of our proposed method for scheduling of scientific workflows in the cloud.The parameters and their semantics used throughout this paper are summarized in Table 1.
Virtual entry node v exit

Virtual exit node v run
Running task e ij An edge such that e ij ∈ E between the tasks v i and v j e p,i

Inward edges of task v i e i,c
Outward edges of task

Scientific Workflow Application Model
The standard way to represent a BoT-based workflow is a Directed Acyclic Graph (DAG) W = (V, E, B) in which V = {v 1 , v 2 , . . . ,v n } is the set of vertices representing n different tasks of the workflow, E = e ij (1 ≤ i ≤ n, 1 ≤ j ≤ n, i = j is the set of directed edges between the vertices, representing data or control dependencies between the tasks v i and v j , indicating a task v j cannot start its execution before v i finishes and sends all the needed output data to task v j and BoT = {bot 1 , bot 2 , . . . ,bot u } is the set of BoTs, in which bot q is the set of q th BoT.Each task has an estimated runtime RT i which denotes the precedence constraint assigned to task v i .Also, each edge e ij has a communication time CT ij which denotes the amount of data needed to be communicated and the precedence constraint from tasks v i and v j , if they are executed on different resources.It is predetermined and known a priori.In this case, task v i is considered one of the immediate predecessor of task v j and task v j is considered one of the immediate successor of task v i .The predecessor task of v i is denoted as pred(v i ) and the successor task as succ(v i ).Task v i can have multiple predecessors and multiple successor tasks.A task is considered as a ready task when all its predecessors have finished execution.Any task without a predecessor task is called as the entry task v entry and a task with no successor task is called the exit task v exit .Since the proposed method needs a graph with individual entry and exit nodes, virtual entry and exit nodes denoted by v entry and v exit respectively, with zero execution time and zero transmission time have been inserted into the DAG.
The crux of the proposed research is scheduling a BoT-based workflow based on the deadline constraint while minimizing the budget of the application.The proposed approach, called DSB, considers pay-as-you-go IaaS cloud model having inherent features such as elasticity, abundance, heterogeneity, dynamic provision of resources, interval-based pricing model and VM provisioning delays.A user-defined deadline is submitted initially with the workflow DAG.

Cloud Resource Model
The IaaS cloud model, adopted in this work, offers virtualized resources containing various instance types at different costs.In this study, the cloud resource model is based on Amazon Elastic Compute Cloud (Amazon EC2), where VM instances are dynamically provisioned on-demand for executing scientific workflow applications.Under this pay-per-use interval-based model, the IaaS providers charge customers for used hours of VM instances and each partial hour consumed is billed as a full hour.We assume that the users have access to unlimited number of instances represented by a set of heterogeneous virtual machines, V M = {m 1 , m 2 , m 3 , . . . ,m k } where m r ∈ V M|1 ≤ r ≤ k and r is an index of the instance type having varied prices and configurations.

Workflow Execution Model
The workflow execution model used in the proposed method is shown in Figure 1.The scientific workflow is described as DAG where the nodes denote individual executional tasks and the edges denote control and data constraints (i.e. the set of parameters) between the tasks.The output data in the form of files by one task is used as input data for another task.In the model used in this study, the scientific workflow is abstract, i.e. the workflow is unaware of the details of physical locations of the executables and the data.This model fits a number of workflow management systems (WMSs).IaaS providers allow WMSs to access to infinite pool of VMs for lease.For instance, Pegasus (https://pegasus.isi.edu/) and Galaxy [23,24].An introduction of the main components of the workflow execution environment are given below, details of which may be found in [25].

1.
Workflow submission: The user submits the workflow application to the WMS for scheduling of the workflow tasks.The WMS resides on the host machine which may be a user's laptop or a community resource.

2.
Target execution environment: It can be a local machine, like the user's laptop, or a virtual environment such as the cloud or a remote physical cluster or a grid [26].On the basis of the available resources, the WMS maps given abstract workflow into an executable workflow and execute them.Moreover, it monitors the execution and manages the input data, intermediate files and output files of the workflow tasks.An overview of the main components of the target execution environment are discussed below: 1.
Workflow mapper: The workflow mapper produces an executable workflow on the basis of the abstract workflow submitted by the user.It identifies the appropriate data and the software and hardware resources required for the execution.Moreover, the mapper can restructure the workflow for performance optimization.

2.
Clustering engine: To reduce system overheads, one or more small tasks are clustered into single execution unit called job in WMS.

3.
Workflow engine: The workflow engine submits the jobs defined by the workflow in order of their dependency constraints.Thereafter, the jobs are submitted to the local workflow scheduling queue.4.
Local workflow scheduler and local queue: The local workflow scheduler manages and supervises individual workflow jobs on local and remote resources.The elapsed time between the submission of a job to the job scheduler and its execution in a remote compute node (potentially on cloud) is denoted as the queue delay. 5.
Remote execution engine: It manages the execution of clustered jobs on one or more remote compute nodes.1. Workflow submission: The user submits the workflow application to the WMS for scheduling of the workflow tasks.The WMS resides on the host machine which may be a user's laptop or a community resource.2. Target execution environment: It can be a local machine, like the user's laptop, or a virtual environment such as the cloud or a remote physical cluster or a grid [26].On the basis of the available resources, the WMS maps given abstract workflow into an executable workflow and execute them.Moreover, it monitors the execution and manages the input data, intermediate files and output files of the workflow tasks.An overview of the main components of the target execution environment are discussed below: 1. Workflow mapper: The workflow mapper produces an executable workflow on the basis of the abstract workflow submitted by the user.It identifies the appropriate data and the software and hardware resources required for the execution.Moreover, the mapper can restructure the workflow for performance optimization.

Assumptions
The current study assumes that the workflow application executes in a single cloud data center so that one possible source of execution delay and data transfer cost between data centers is eliminated.The average bandwidth between the VM instances is assumed to be fixed during the execution of W. The estimated runtime of task on a VM instance, denoted as RT r i is known to the scheduler in advance.It is obtained by Equation (1).The runtime of virtual entry and exit nodes is zero.Moreover, the tasks do not have the ability to checkpoint their work or get preemption.In other words, a task has to be restarted in case of its failure.Furthermore, we assume that for every VM type, the processing capacity is furnished either from the IaaS provider or can be estimated based on the work of [27].We assume a VM boot time of 97 s based on the measurements presented by Mao and Humphrey [9] for EC2 cloud.We adopted a performance degradation model based on results achieved in Amazon EC2 clouds by Jackson et al. [28].Therefore, we consider the loss in VM performance based on a normal distribution with mean of 15% and standard deviation of 10% in our proposed approach as well.
Similarly, the amount of data needed to be communicated between the tasks of a workflow are considered to be known in advance.The communication time for the tasks allocated to the same VM is considered to be zero.Moreover, data communication cost between resources are assumed to be zero, since in IaaS clouds, data communication inside a data center is free.Furthermore, the cloud datacenter is considered to have unlimited resources, so resource contention is almost negligible.Additionally, we assume that all VMs have enough memory to run any task of the workflow.

Problem Statement
The scheduling problem addressed in the present study can be defined as dynamically scaling a set of VMs and assigning each task to a given VM in order to minimize the cost of the application while fulfilling the deadline constraint.

Basic Definitions
Definition 1.Estimated runtime of task v i on a VM m r , denoted by RT r i , is computed using Equation (1).
where l i is the service length of task v i measured in Floating Point Operations (FLOP), p r is the Central Processing Unit (CPU) processing capacity of VM m r in Million Floating Point Operations Per Second (MFLOPS) based on the number of EC2 compute units and pv r is the VM performance variability which represent the uncertainties such as potential variability or degradation in CPU performance in real cloud environments.
Definition 2. Communication time of edge e ij between tasks v i and v j on VM m r , denoted by CT r ij , is computed using Equation (2).
where s ij is the size of data needed to be communicated from tasks v i and v j in Megabytes (MB) and b r is the bandwidth capacity of VM m r in Megabytes Per Second (MBps).
Definition 3. Execution time of task v i on a VM, denoted by ET r i , is computed using Equation (3).
In IaaS cloud, provisioning delay in VM allocation, that is d r can be caused by multiple factors such as, data center location, VM setup time, software setup time, VM migration time, quantity of simultaneous VM provisioning requests and variations in VM types, Operating Systems (OS) and time zones.Definition 4. Start time of running task v i on VM m r , denoted by ST r i , is computed as in Equation (4).
where slt r is start leasing time of VM m r and ∑ v p ∈pred(v i ) ET r p is the total execution time of assigned tasks to VM m r .Definition 5. Execution cost of task v i on VM m r , denoted by EC r i , is computed as in Equation (5).
where h r represents used billing interval of VM m r , c r represents the cost per interval unit of VM m r , s ij is the size of the data to be transferred from v i to v j and tc ij represents the data transfer cost (per MB) of communicating data from task v i to v j .It becomes zero, if the tasks are scheduled on VMs within the same data center.So, we also do not consider this price when calculating the workflow's execution cost.
Definition 6.The upward rank for a task v i is the length of critical/longest path from task v i to the exit task (v exit ), including the execution time of v i .Thus, the priority of each task is defined as given by Topcuoglu et al. [29].
Definition 7. The Degree of Dependency of each task v i , denoted as DD i , is provided by Equation (7).
where ∑ v p ∈pred(v i ) e p,i is the sum of all inward edges of task v i and ∑ v p ∈pred(v i ) e p,i is the sum of all outward edges of task v i .
Definition 8. Level of a task v j in the workflow, denoted as lv v j , is defined as the maximum number of edges from the task v entry to v j .All the tasks in a BoT have same level.It is computed by Equation ( 8).Level of v entry is assumed to be zero.
Definition 9. Finish Time of task v i on VM m r is computed as in Equation (9).
Definition 10.Makespan (that is, schedule length) of workflow W is calculated by Equation (9).
In other words, the overall makespan of the workflow is computed as the elapsed time between v entry and v exit .Definition 11.Available spare time of a workflow is the amount of elapsed time between the user-defined deadline and the makespan, as computed by Equation (11).
Definition 12. Available spare time of a level is provided by Equation (12).
where RT lv is the runtime of all tasks in the corresponding level lv and RT W is the runtime of all tasks of the workflow W.

Definition 13.
Available spare time of a level is distributed among the tasks in the corresponding level, as provided by Equation ( 13).
AS i = count lv n × AS lv (13) where count lv is the number of all tasks in the corresponding level lv and n is the number of all tasks in the workflow W.
Definition 14. Sub-deadline of a task is the maximum limit of finish time of the corresponding task.
The sub-deadline of each task v i is calculated by recursive traversal of the workflow DAG downwards, starting from v entry , as provided by Equation ( 14).
Definition 15.Sub-deadline of a BoT is initialized as Definition 16.Earliest Finish Time of a task v i on VM, denoted as EFT r i , is given by Equation ( 16).
where min RT r i is the minimum finish time of task v i over all instances and SD p is the estimated sub-deadline for predecessor of v i .

Proposed Algorithm
The main objective of the proposed DSB technique for scheduling scientific workflows in cloud computing is achieving high success rates with lower pay-per-use cost while satisfying the deadline constraint.The proposed DSB includes five distinct steps, namely task prioritization, task grouping, deadline distribution, task selection and elastic resource provisioning phase as presented next.

Task Prioritization
In the first step which is called task prioritization, we propose a task ordering strategy for optimizing task assignment.First, the tasks of the workflow are assigned priorities in order to guarantee task dependencies.For assigning priorities to the workflow tasks, their upward ranks [29] are calculated by Equation (6).The priority of tasks calculated using this technique has several benefits, which make it a worthy candidate for the prioritizing phase.Its implementation is simple.It considers the execution time of the workflow tasks, the transfer time between each pair of the tasks and the critical path of the workflow.Besides, this technique ensures that every task of the workflow has a higher priority over its successor tasks.It is calculated recursively for each task, beginning from the ending task to the first task of the workflow.The degree of dependency of each task is calculated by Equation (7).The tasks in the workflow having higher number of inward and outward edges have higher priority than others for scheduling them on the available VMs.Finally, the task scheduling sequence after the first phase, contain tasks sorted in descending order of their priorities.By assigning a priority to each task of the workflow, this method creates tasks with higher priorities earlier than the other tasks.

Task Grouping
Subsequently, in the second step, the tasks are combined into BoTs by applying horizontal grouping so that the tasks in each BoT have the same functionality and the same level (that is, depth) in the workflow DAG and are guaranteed to be ready for processing in parallel [19].Figure 2 shows our example benchmark scientific workflows after applying horizontal grouping.The homogeneous tasks are represented by same color in Figure 2.Each BoT can comprise a single task or multiple tasks.Tasks within each BoT are homogeneous and share the same single immediate predecessor, have identical data distribution structure and have same type in terms of data size, input/output size and computational cost.Table 2 shows the published details of these real-world workflows including the DAG, the average data size and the reference runtime of tasks based on Xeon@2.33GHz CPUs (Compute Units ∼ = 8) [25,30,31].has a higher priority over its successor tasks.It is calculated recursively for each task, beginning from the ending task to the first task of the workflow.The degree of dependency of each task is calculated by Equation (7).The tasks in the workflow having higher number of inward and outward edges have higher priority than others for scheduling them on the available VMs.Finally, the task scheduling sequence after the first phase, contain tasks sorted in descending order of their priorities.By assigning a priority to each task of the workflow, this method creates tasks with higher priorities earlier than the other tasks.

Task Grouping
Subsequently, in the second step, the tasks are combined into BoTs by applying horizontal grouping so that the tasks in each BoT have the same functionality and the same level (that is, depth) in the workflow DAG and are guaranteed to be ready for processing in parallel [19].Figure 2 shows our example benchmark scientific workflows after applying horizontal grouping.The homogeneous tasks are represented by same color in Figure 2.Each BoT can comprise a single task or multiple tasks.Tasks within each BoT are homogeneous and share the same single immediate predecessor, have identical data distribution structure and have same type in terms of data size, input/output size and computational cost.Table 2 shows the published details of these real-world workflows including the DAG, the average data size and the reference runtime of tasks based on Xeon@2.33GHz CPUs (Compute Units ≅ 8) [25,30,31].(e) SIPHT    The third step of the proposed algorithm is called deadline distribution.First, the cheapest VM type (m sel ) is determined among all the available VM instances such that, if assigned to all the workflow tasks, their estimated makespan (calculated by using Equation ( 10)) would be lower than the user-defined deadline (DL W ). If the estimated makespan exceeds deadline, then the next cheapest VM type (m sel ) is selected until the estimated execution time of the workflow on m sel is lower than DL W .The elapsed time between the deadline and the estimated makespan, denoted as the available spare time AS W is calculated according to Equation (11).AS W is distributed proportionally over all levels of the workflow on the basis of runtime of tasks according to Equation (12).Then, AS lv distributed among all tasks of each level proportional to the number of tasks in the corresponding level according to Equation (13).Afterwards, an estimated sub-deadline SD i is computed for each task of the workflow under the determined VM type m sel , by using Equation ( 14).This sub-deadline will be a guiding factor for taking decisions at runtime about whether to reuse existing VM or rent a new one.It is larger for the BoTs with longer running tasks.The sub-deadline of each BoT is assumed to be the maximum finish time of its tasks corresponding to m sel (Equation ( 15)).In other words, the tasks within a BoT are assigned the same sub-deadline which would be equal to the estimated start time of the successor task.The algorithm for distributing deadlines to the BoTs is given in Algorithm 1.

Task Selection
The set of tasks ready for execution are put in the execution queue.A task is considered as ready after all of its predecessors are already scheduled.The tasks in each BoT can be executed in parallel because they have no dependency constraints as well as sequentially to improve the utilization of leased billing intervals.However, there may exist dependency constraints between BoTs at the upper or lower levels.In order to exploit parallelism of such BoTs, the currently running tasks can send the data to their dependent tasks as soon as it is prepared, so that the dependent tasks at the consequent level can start their execution.Consequently, more tasks will execute in less time resulting in faster schedules.All the tasks in a BoT are homogeneous and share the same predecessor, therefore they become ready at the same time and have identical VM performance requirements.

Elastic Resource Provisioning
The resource provisioning phase is based on the optimization of a QoS metric cost and used to dynamically adjust the number of required VM instances to ensure the completion of workflow within the deadline.Its primary purpose is to prioritize the reuse of already leased VMs by utilizing idle times on previous rented intervals when possible instead of renting a new VM instance.When all the tasks in a BoT are ready for execution, they are put in the execution queue from the priority queue.Tasks in a BoT can be scheduled one by one to an elastic number of VMs for parallel, or serial executions.Scheduling plan is created based on tasks in the same BoT to decide the number and type of VM instances that can be launched to schedule the BoT.The plan needs to be made once for all the tasks in each BoT using MIP.The MIP problem was formulated to provide an estimate of the number and types of VMs that can optimize total cost of the workflow execution with the given deadline.It can be formally modeled as follows: ∑ m r ∈V M,u∈m r y r,u = n r,u bot , ∀bot ∈ BoT (20) FT r sink ≤ DL W (22) Constraint ( 18) is a binary variable and ensures that each task is assigned to one and only one VM type.Constraint (19) ensures the precedence constraint.Constraint (20) ensures that the number of tasks of a BoT assigned to the u th VM of type r process all the tasks.Constraint (21) guarantees that the sum of sub-deadlines of all BoTs is not greater than the workflow deadline.Constraint (22) ensures that the deadline is met.
The problem is solved in CPLEX solver and a scheduling plan SP r q = (m r , n r q , k m r bot q ) is obtained for each BoT to obtain VMs that can process the BoTs while meeting the deadline and with minimum cost.n r q corresponds to the maximum number of tasks in the BoT bot q that can be executed by the VM type m r without exceeding the deadline and k m r bot q is the number of VMs of type m r to use.The VM selection for the task v i ∈ BoT requires the estimation of its actual execution time which is computed by taking the sum of execution time and provisioning delay.Then each ready task is checked to available times of rented intervals on existing VMs.The execution cost is calculated by Equation ( 5).Since, each partial hour consumed is billed as a full hour, therefore the execution cost of other tasks submitted to the same VM during the available partial hour is zero.If such VMs exist that incur no execution cost for the remaining intervals as well as satisfy the sub-deadline, then the scheduler selects the VM m sel , that can execute the running task with the estimated earliest finish time (according to Equation ( 16)).The number of tasks allotted to current billing interval of m sel depends on the number of tasks that can meet their sub-deadline.It can improve resource utilization by minimizing wastage of already leased intervals without affecting the execution time of the workflow.The total execution time of assigned tasks to VM m r is updated dynamically after every task assignment.If no such VMs exist with remaining available billing intervals, all the tasks in the BoT from the execution queue are scheduled to subsequent billing interval of existing VMs only if the BoT still meets the sub-deadline.If none of the existing active VMs can satisfy the BoT sub-deadline, the resource scaling-up strategy will be triggered to create a new VM type with the minimum execution cost among others and that can meet the BoT's sub-deadline.In extreme cases in which no such VM type exist that can complete the tasks within deadline, a fastest VM type can be rented.Whenever a BoT completes its execution, the sub-deadline of all the remaining BoTs is updated dynamically.Since all the tasks within a BoT are homogeneous, so they have the same VM type preference.Therefore, the current VM type m sel can be assigned to all tasks in BoT.The algorithms for this phase are given in Algorithms 2-4.In order to adapt to unexpected delays such as variations in task runtime estimations, network congestion and resource provisioning delays, the sub-deadline of remaining workflow tasks is adjusted whenever a task finishes either earlier or later than expected.In this way, if a task finishes earlier, the remaining tasks will have more time to run and hence they can either be assigned to a cheaper VM or delayed, to be scheduled in subsequent cycles.If a task finishes later than expected, adjusting the deadline of the remaining tasks may prevent the deadline from being missed.The VM scaling-down strategy is used by the scheduler during execution to shut down any leased VMs that are idle for a long time and approaching their subsequent billing interval.An overview of resource scaling down technique is shown in Algorithm 6.
The algorithm of the proposed method is shown in Algorithm 5.

Computational Complexity
Consider n as the total number of workflow tasks, m as the number of BoTs, k as the total number of available VMs, d as the number of dependency constraints and e as the number of directed edges.The time complexity of this scheduling algorithm requires the computation of some basic operations.Calculating priority, dependency constraints and sub-deadline of all tasks have complexity O(n.k).Sorting groups based on their priority O(m).Computing start time, completion time and solving the MIP for all VMs O(k).Mapping tasks on VMs O(n.k).The total time is O(n.k + n(m + k + nk)), therefore complexity of the proposed algorithm is O n 2 .k .

Experiment Environment
WorkflowSim is a Java-based open source discrete event workflow engine that has been used to model a cloud execution environment for executing scientific workflow applications [32].This trace-based framework is an extension of CloudSim [33].WorkflowSim offers support for workflow DAGs and provides execution environment for workflow level resource provisioning, resource management, task clustering and task scheduling.For our experiments, WorkflowSim was adopted to evaluate the performance of the proposed method on real traces under dynamically provisioned on-demand cloud VMs and a pay-per-use model derived from Amazon EC2 pricing model (http://aws.amazon.com/ec2).The correctness of WorkflowSim has been proved in [32].Moreover, it supports the analysis of various scheduling overheads and failures.
An IaaS provider offering a single data center and six types of VMs was modelled.Table 3 lists the configurations of the VM types based on EC2.The workflow generator toolkit, Pegasus, was used to generate synthetic workflows of various sizes for each workflow application in terms of total number of nodes.Five real-world scientific applications were chosen, namely: Cybershake (data-intensive, memory-intensive, resource-intensive), Epigenomics (CPU-bound), LIGO Inspiral Analysis (memory-intensive, resource-intensive), Montage (I/O bound) and SIPHT [30].

Performance Metric
The goal of scheduling algorithm considered here is to find a schedule map in such a way that the cost is optimized under user-defined deadline constraint.The performance metrics used to evaluate the proposed DSB algorithm are given below.

Normalized Deadline (ND)
To investigate the quality of results and for the purpose of comparison, deadline is normalized, called ND (Normalized Deadline).MP W is the minimum makespan or schedule length of the workflow DAG.It can be obtained by computing the makespan of the workflow on fastest VM types with a maximum level of parallelism and ignoring delays.With these values, the normalized user-defined deadline constraint is computed by Equation ( 23): For successful schedule map that meet the deadline constraint, the value of ND could not be less than 1.

Improvement Rate (IR)
It is defined as the percentage improvement obtained in the performance of the proposed algorithm in the overall execution of workflow DAG with respect to other algorithms.Reduction in the cost of the proposed algorithm over other considered algorithms can be calculated by IR(%).

Success Rate (SR)
It is defined as the success rate of finding a cost optimized schedule while satisfying the user-defined deadline, as given by Equation ( 25

Evaluation Results
We implemented WRPS [34], SCS [9] and HEFT [29] algorithms.Then, we evaluated these scheduling methods and our proposed DSB scheduling algorithm on a standard and real set of scientific workflow applications.Finally, we compared their results with our scheduling model.
The workflows were executed by changing the deadline.The experiments were repeated 30 times for each workload instance and the average value of output metrics are reported in this section.The effect of varying deadline over the cost was evaluated, as shown in Figures 3 and 4. The proposed DSB scheduling algorithm successfully scheduled maximum number of tasks while meeting the deadline constraint with the average success rate of 97.93% compared to the success rates of WRPS, SCS and HEFT which are 92.68%,82.86% and 66.93% respectively.Figures 3 and 4 shows the average cost of Montage and CyberShake workflow respectively for 1000 tasks.The structure of Montage and CyberShake consists of many small-sized tasks in each level with almost same priorities, dependency constraints and deadline.It can be seen that the average cost of the proposed DSB is lower because the workflow tasks are ordered on the basis of their priorities, dependency constraints and sub-deadlines and afterwards partitioned into horizontal BoTs.The cost of SCS and HEFT are comparatively high because these methods take longer to process tasks of large size.Similarly, it can be seen that the cost of the proposed DSB was comparatively lower than the other algorithms which shows that the proposed DSB exhibits better performance than its counterparts.Figure 5 shows that SCS and HEFT missed the deadline when it is minimum for executing Epigenomics workflow, while in all other cases, the deadline was successfully met.The proposed method not only execute maximum number of tasks successfully without violating the deadline but also achieves lowest cost for Epigenomics workflows.Figure 6 illustrates that the proposed DSB shows best performance under hard deadline.This is due to the fact that LIGO Inspiral Analysis workflow consists of blocks of tasks and each block create specific results on a portion of data.These results must be created with minimum makespan.Our proposed method can execute the blocks in each level in parallel by partitioning them in horizontal groups while taking care of the deadline and dependency constraints.The SIPHT workflow has different parts and their intermediate output is required as input to the last node to achieve the final output.Our proposed algorithm prioritizes reuse of rented intervals and schedule the different parts of the SIPHT by considering priority and sub-deadline of each task and map it with the VM that leads to minimum cost of the task on the assigned VM, as shown in Figure 7.The SIPHT workflow has different parts and their intermediate output is required as input to the last node to achieve the final output.Our proposed algorithm prioritizes reuse of rented intervals and schedule the different parts of the SIPHT by considering priority and sub-deadline of each task and map it with the VM that leads to minimum cost of the task on the assigned VM, as shown in Figure 7.The SIPHT workflow has different parts and their intermediate output is required as input to the last node to achieve the final output.Our proposed algorithm prioritizes reuse of rented intervals and schedule the different parts of the SIPHT by considering priority and sub-deadline of each task and map it with the VM that leads to minimum cost of the task on the assigned VM, as shown in Figure 7.It can be seen from the results that the proposed method DSB shows significant improvement in cost when the deadline was minimum.Moreover, it showed better resource management by exploiting efficient level of parallelism.
Figure 8 shows the normalized deadline obtained for each workflow with different scheduling algorithms.In the proposed DSB, the greater value of ND represents that it could achieve an efficient schedule map with lowest cost under the user-defined deadline constraint.Its value less than 1 denotes that the algorithm was unable to generate a schedule map while satisfying the deadline constraint.Moreover, the proposed DSB achieves lower cost with workflows having longer deadlines because they are scheduled on least expensive VMs.It can be seen from the results that the proposed method DSB shows significant improvement in cost when the deadline was minimum.Moreover, it showed better resource management by exploiting efficient level of parallelism.
Figure 8 shows the normalized deadline obtained for each workflow with different scheduling algorithms.In the proposed DSB, the greater value of ND represents that it could achieve an efficient schedule map with lowest cost under the user-defined deadline constraint.Its value less than 1 denotes that the algorithm was unable to generate a schedule map while satisfying the deadline constraint.Moreover, the proposed DSB achieves lower cost with workflows having longer deadlines because they are scheduled on least expensive VMs.It can be seen from the results that the proposed method DSB shows significant improvement in cost when the deadline was minimum.Moreover, it showed better resource management by exploiting efficient level of parallelism.
Figure 8 shows the normalized deadline obtained for each workflow with different scheduling algorithms.In the proposed DSB, the greater value of ND represents that it could achieve an efficient schedule map with lowest cost under the user-defined deadline constraint.Its value less than 1 denotes that the algorithm was unable to generate a schedule map while satisfying the deadline constraint.Moreover, the proposed DSB achieves lower cost with workflows having longer deadlines because they are scheduled on least expensive VMs.
It can be seen from the results that the proposed method DSB shows significant improvement in cost when the deadline was minimum.Moreover, it showed better resource management by exploiting efficient level of parallelism.
Figure 8 shows the normalized deadline obtained for each workflow with different scheduling algorithms.In the proposed DSB, the greater value of ND represents that it could achieve an efficient schedule map with lowest cost under the user-defined deadline constraint.Its value less than 1 denotes that the algorithm was unable to generate a schedule map while satisfying the deadline constraint.Moreover, the proposed DSB achieves lower cost with workflows having longer deadlines because they are scheduled on least expensive VMs.The average success rate achieved for the workflows are shown in Figure 9.It can be seen that the performance of the proposed DSB is better in terms of cost achieved under user-defined deadline as compared to the other baseline algorithms.The average success rate achieved for the workflows are shown in Figure 9.It can be seen that the performance of the proposed DSB is better in terms of cost achieved under user-defined deadline as compared to the other baseline algorithms.

Sensitivity of Overheads, VM Performance Variations and Task Failures
In real cloud environment, unexpected delays, VM provisioning delays, inaccuracies in task runtime estimations and VM performance variations can cause failures.The sensitivity of our mechanism to overheads, delays and performance variations was evaluated.It was modeled using a normal distribution around the real value with 30% provisioning delay and task failure probability.The loss in VM performance was considered based on a normal distribution with mean of 15% and standard deviation of 10%.The relative performance of our method is given in Figure 10.It shows four curves, each representing (i) accurate estimations and no delays and failures; (ii) inaccurate estimations of VM performance variation; (iii) inaccurate estimations of provisioning delays and (iv) task failures respectively.These results demonstrate the algorithm's good sensitivity to inaccurate estimations.It is also found that the proposed mechanism is successful in meeting its deadline constraint in most of the cases.However, task failure has a larger impact on performance due to its direct effect on overall makespan.

Sensitivity of Overheads, VM Performance Variations and Task Failures
In real cloud environment, unexpected delays, VM provisioning delays, inaccuracies in task runtime estimations and VM performance variations can cause failures.The sensitivity of our mechanism to overheads, delays and performance variations was evaluated.It was modeled using a normal distribution around the real value with 30% provisioning delay and task failure probability.The loss in VM performance was considered based on a normal distribution with mean of 15% and standard deviation of 10%.The relative performance of our method is given in Figure 10.It shows four curves, each representing (i) accurate estimations and no delays and failures; (ii) inaccurate estimations of VM performance variation; (iii) inaccurate estimations of provisioning delays and (iv) task failures respectively.These results demonstrate the algorithm's good sensitivity to inaccurate estimations.It is also found that the proposed mechanism is successful in meeting its deadline constraint in most of the cases.However, task failure has a larger impact on performance due to its direct effect on overall makespan.
runtime estimations and VM performance variations can cause failures.The sensitivity of our mechanism to overheads, delays and performance variations was evaluated.It was modeled using a normal distribution around the real value with 30% provisioning delay and task failure probability.The loss in VM performance was considered based on a normal distribution with mean of 15% and standard deviation of 10%.The relative performance of our method is given in Figure 10.It shows four curves, each representing (i) accurate estimations and no delays and failures; (ii) inaccurate estimations of VM performance variation; (iii) inaccurate estimations of provisioning delays and (iv) task failures respectively.These results demonstrate the algorithm's good sensitivity to inaccurate estimations.It is also found that the proposed mechanism is successful in meeting its deadline constraint in most of the cases.However, task failure has a larger impact on performance due to its direct effect on overall makespan.

Analysis of Variance (ANOVA) Test
In this section, the statistical significance of the obtained experimental results is checked by the one-way ANOVA test [35].Table 4 shows that most of the variation in the obtained results is the variation between and not the variation within.It is evident that the ANOVA test is statistically

Analysis of Variance (ANOVA) Test
In this section, the statistical significance of the obtained experimental results is checked by the one-way ANOVA test [35].Table 4 shows that most of the variation in the obtained results is the variation between and not the variation within.It is evident that the ANOVA test is statistically significant due to the greater F-statistic and lower p-value.In other words, there is statistically significant difference between the two algorithms.Therefore, the null hypothesis (H 0 ) which states that the mean of all the algorithms are equal, can be rejected.Note: SS = Sum of Squares, MS = Mean Sum of Squares, df = Degree of Freedom.

Conclusions
In this paper, a BoT based workflow scheduling algorithm has been proposed for the dynamic and elastic provisioning of VM instances, that considers resource renting cost minimization constrained to user-defined deadline.The proposed model groups the workflow into BoTs based on data dependency and priority constraints and thereafter optimizes the allocation and scheduling of BoTs on elastic, heterogeneous and dynamically provisioned cloud resources called VMs in order to attain the proposed method's objectives.The proposed approach considers pay-as-you-go IaaS clouds having inherent features such as elasticity, abundance, heterogeneity, VM performance variation and VM provisioning delays.Mathematical model was successfully used for the dynamic provisioning of resources.The traces for the experiments were taken from real-world scientific workflow applications.Experimental results demonstrate that our proposed DSB increases the chance of deadline being satisfied and minimizes the execution cost compared to other approaches for the real-world scientific workflow applications considered.
The future work is intended to investigate more accurate models to predict potential failures, uncertainties and performance variations of time critical applications in the real IaaS environment.Another future direction is to extend the proposed model to implement fault tolerant energy efficient elastic resource provisioning and scheduling.Moreover, future research will also evaluate the proposed model on VMs with different lengths of pricing intervals.Finally, the algorithms will be implemented in a workflow execution engine for their effective utilization in real life.

2 .
Clustering engine: To reduce system overheads, one or more small tasks are clustered into single execution unit called job in WMS. 3. Workflow engine: The workflow engine submits the jobs defined by the workflow in order of their dependency constraints.Thereafter, the jobs are submitted to the local workflow scheduling queue.4. Local workflow scheduler and local queue: The local workflow scheduler manages and supervises individual workflow jobs on local and remote resources.The elapsed time
): SR = Number o f simulation runs that success f ully meet deadline Total number o f simulation runs × 100 (25)

Future
Internet 2018, 10, x FOR PEER REVIEW 18 of 23

Figure 3 .
Figure 3. Average cost of Montage under varied normalized deadlines.

Figure 4 .
Figure 4. Average cost of CyberShake under varied normalized deadline.

Figure 3 .
Figure 3. Average cost of Montage under varied normalized deadlines.

Figure 3 .
Figure 3. Average cost of Montage under varied normalized deadlines.

Figure 4 .
Figure 4. Average cost of CyberShake under varied normalized deadline.Figure 4. Average cost of CyberShake under varied normalized deadline.

Figure 4 .
Figure 4. Average cost of CyberShake under varied normalized deadline.Figure 4. Average cost of CyberShake under varied normalized deadline.

Figure 4 .
Figure 4. Average cost of CyberShake under varied normalized deadline.

Figure 5 .
Figure 5. Average cost of Epigenomics under varied normalized deadline.Figure 5. Average cost of Epigenomics under varied normalized deadline.

Figure 5 . 23 Figure 6 .
Figure 5. Average cost of Epigenomics under varied normalized deadline.Figure 5. Average cost of Epigenomics under varied normalized deadline.Future Internet 2018, 10, x FOR PEER REVIEW 19 of 23

Figure 7 .
Figure 7. Average cost of SIPHT under varied normalized deadline.

Figure 7 .
Figure 7. Average cost of SIPHT under varied normalized deadline.

Figure 7 .
Figure 7. Average cost of SIPHT under varied normalized deadline.

Figure 8 .
Figure 8. Normalized Deadline (ND) for each workflow with different methods.Figure 8. Normalized Deadline (ND) for each workflow with different methods.

Figure 8 .
Figure 8. Normalized Deadline (ND) for each workflow with different methods.Figure 8. Normalized Deadline (ND) for each workflow with different methods.

Future
Internet 2018, 10, x FOR PEER REVIEW 20 of 23

Figure 10 .
Figure 10.Sensitivity to inaccurate estimations and task execution failure.

Figure 10 .
Figure 10.Sensitivity to inaccurate estimations and task execution failure.
Communication time of edge e ij from tasks v i and v j on a VM m r ET r

Table 2 .
Characteristics of the benchmark workflows.CU: Compute Unit.

Table 2 .
Characteristics of the benchmark workflows.CU: Compute Unit.

Algorithm 1 Deadline Distribution procedure DD
(Workflow W, Deadline DL W , Runtime of tasks on VMs RT r i ) 1: m sel ← Find cheapest VM type such that MP W < DL W 2: if m sel = null then 3: while m sel = null do 4: m sel ← Next cheapest VM type such that MP W < DL W Distribute AS W proportionally over all tasks in each level lv //according to Equation (13) 10: Calculate estimated sub-deadline SD i for each task //according to Equation (14) 11: Update sub-deadline of each BoT as the maximum finish time of its tasks ← m r ∈ M sel such that EFT sel run is minimum 14 if m sel = null then 15 m sel ← m r ∈ M sel such that ET sel run is maximum 16 end if 17 if newV M = true then 18 Lease new interval m sel and schedule task v run on it 19 ExeQ new ← ExeQ new − {v run } 20 ST sel ← CurrentTime() 21 end if q .count(tasks)= 1 then 16 v run ← bot q 17 m sel ← select cheapest VM that can run task within deadline 18 Schedule v run to m sel 19 ExeQ ← ExeQ − {v run } 20 CPT sel ← CPT sel + ET run

Algorithm 5 The Proposed DSB Input:
Workflow W, Deadline DL W , Runtime of tasks on VMs RT r

Table 4 .
One-way ANOVA test result.