Next Article in Journal
Past, Present, and Future of Regeneration Therapy in Oral and Periodontal Tissue: A Review
Previous Article in Journal
Workpiece Pose Optimization for Milling with Flexible-Joint Robots to Improve Quasi-Static Performance
Article

SLA-Based Adaptation Schemes in Distributed Stream Processing Engines

1
Division of Computer Science and Engineering, Hanyang University, Seoul 133-791, Korea
2
Department of Computer Engineering, Hongik University, Seoul 121-791, Korea
3
School of Computing and Communications, Lancaster University, Lancaster, UK
*
Author to whom correspondence should be addressed.
This article is a re-written and extended version of “An Adaptive SLA-Based Data Flow Mechanism for Stream Processing Engines” presented at ICTC 2017, Jeju Island, South Korea on 18 October 2017.
Appl. Sci. 2019, 9(6), 1045; https://doi.org/10.3390/app9061045
Received: 20 January 2019 / Revised: 23 February 2019 / Accepted: 8 March 2019 / Published: 13 March 2019
(This article belongs to the Section Computing and Artificial Intelligence)
With the upswing in the volume of data, information online, and magnanimous cloud applications, big data analytics becomes mainstream in the research communities in the industry as well as in the scholarly world. This prompted the emergence and development of real-time distributed stream processing frameworks, such as Flink, Storm, Spark, and Samza. These frameworks endorse complex queries on streaming data to be distributed across multiple worker nodes in a cluster. Few of these stream processing frameworks provides fundamental support for controlling the latency and throughput of the system as well as the correctness of the results. However, none has the ability to handle them on the fly at runtime. We present a well-informed and efficient adaptive watermarking and dynamic buffering timeout mechanism for the distributed streaming frameworks. It is designed to increase the overall throughput of the system by making the watermarks adaptive towards the stream of incoming workload, and scale the buffering timeout dynamically for each task tracker on the fly while maintaining the Service Level Agreement (SLA)-based end-to-end latency of the system. This work focuses on tuning the parameters of the system (such as window correctness, buffering timeout, and so on) based on the prediction of incoming workloads and assesses whether a given workload will breach an SLA using output metrics including latency, throughput, and correctness of both intermediate and final results. We used Apache Flink as our testbed distributed processing engine for this work. However, the proposed mechanism can be applied to other streaming frameworks as well. Our results on the testbed model indicate that the proposed system outperforms the status quo of stream processing. With the inclusion of learning models like naïve Bayes, multilayer perceptron (MLP), and sequential minimal optimization (SMO)., the system shows more progress in terms of keeping the SLA intact as well as quality of service (QoS). View Full-Text
Keywords: big data; distributed computing; modern stream processing engine; SLA; watermarking; cloud computing big data; distributed computing; modern stream processing engine; SLA; watermarking; cloud computing
Show Figures

Figure 1

MDPI and ACS Style

Hanif, M.; Kim, E.; Helal, S.; Lee, C. SLA-Based Adaptation Schemes in Distributed Stream Processing Engines. Appl. Sci. 2019, 9, 1045. https://doi.org/10.3390/app9061045

AMA Style

Hanif M, Kim E, Helal S, Lee C. SLA-Based Adaptation Schemes in Distributed Stream Processing Engines. Applied Sciences. 2019; 9(6):1045. https://doi.org/10.3390/app9061045

Chicago/Turabian Style

Hanif, Muhammad, Eunsam Kim, Sumi Helal, and Choonhwa Lee. 2019. "SLA-Based Adaptation Schemes in Distributed Stream Processing Engines" Applied Sciences 9, no. 6: 1045. https://doi.org/10.3390/app9061045

Find Other Styles
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Access Map by Country/Region

1
Back to TopTop