Next Article in Journal / Special Issue
Real-Time Fuzzy Data Processing Based on a Computational Library of Analytic Models
Previous Article in Journal / Special Issue
Television Rating Control in the Multichannel Environment Using Trend Fuzzy Knowledge Bases and Monitoring Results
Article Menu

Export Article

Open AccessArticle

Multi-Agent Big-Data Lambda Architecture Model for E-Commerce Analytics

1
Department of Computer Science, The University of Liverpool, Liverpool L69 7ZX, UK
2
Department of Computer Science, Xi’an Jiaotong-Liverpool University, Wuzhong 215123, China
*
Author to whom correspondence should be addressed.
This paper is an extended version of our paper published in the IEEE Second International Conference on Data Stream Mining Processing (DSMP), Lviv, Ukraine, 21–25 August 2018.
Received: 5 November 2018 / Revised: 27 November 2018 / Accepted: 27 November 2018 / Published: 1 December 2018
(This article belongs to the Special Issue Data Stream Mining and Processing)
  |  
PDF [1910 KB, uploaded 11 December 2018]
  |     |  

Abstract

We study big-data hybrid-data-processing lambda architecture, which consolidates low-latency real-time frameworks with high-throughput Hadoop-batch frameworks over a massively distributed setup. In particular, real-time and batch-processing engines act as autonomous multi-agent systems in collaboration. We propose a Multi-Agent Lambda Architecture (MALA) for e-commerce data analytics. We address the high-latency problem of Hadoop MapReduce jobs by simultaneous processing at the speed layer to the requests which require a quick turnaround time. At the same time, the batch layer in parallel provides comprehensive coverage of data by intelligent blending of stream and historical data through the weighted voting method. The cold-start problem of streaming services is addressed through the initial offset from historical batch data. Challenges of high-velocity data ingestion is resolved with distributed message queues. A proposed multi-agent decision-maker component is placed at the MALA stack as the gateway of the data pipeline. We prove efficiency of our batch model by implementing an array of features for an e-commerce site. The novelty of the model and its key significance is a scheme for multi-agent interaction between batch and real-time agents to produce deeper insights at low latency and at significantly lower costs. Hence, the proposed system is highly appealing for applications involving big data and caters to high-velocity streaming ingestion and a massive data pool. View Full-Text
Keywords: Lambda Architecture; e-commerce analytics; real-time data analytics; real-time data ingestion; real-time machine leaning; recommender engine; online k-means clustering Lambda Architecture; e-commerce analytics; real-time data analytics; real-time data ingestion; real-time machine leaning; recommender engine; online k-means clustering
Figures

Figure 1

This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited (CC BY 4.0).
SciFeed

Share & Cite This Article

MDPI and ACS Style

Pal, G.; Li, G.; Atkinson, K. Multi-Agent Big-Data Lambda Architecture Model for E-Commerce Analytics. Data 2018, 3, 58.

Show more citation formats Show less citations formats

Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Metrics

Article Access Statistics

1

Comments

[Return to top]
Data EISSN 2306-5729 Published by MDPI AG, Basel, Switzerland RSS E-Mail Table of Contents Alert
Back to Top