A Capillary Computing Architecture for Dynamic Internet of Things: Orchestration of Microservices from Edge Devices to Fog and Cloud Providers
Abstract
:1. Introduction
- What potential monitoring metrics can impact the performance of emerging smart IoT applications, and how can these quality parameters be monitored, exchanged and used?
- What architectural approach is needed to be able to offload requests from Edge to Fog or Cloud under a varied number of workloads in which IoT devices dynamically move from one geographic location to another?
- From the interoperability viewpoint, what are the main existing open standards which can be used to design a capillary distributed computing architecture that is able to support IoT applications?
2. Background and Related Works
2.1. The Microservices Architecture
2.2. Container-Based Microservices
2.3. Edge/Fog State-of-The-Art Review
- Using a static amount of Edge and Fog resources as a complete replacement to Cloud infrastructures: The best practice in such a solution is exploiting general-purpose Edge nodes that can be involved in heterogeneous types of computation and data analytics. To this end, the LightKone project [28] is recently aimed at moving computation and data storage completely out of the Cloud and directly on the extreme Edge of the network. As another example, the main objective of Open Edge Computing (OEC) project [29] is enabling all nearby Edge components such as Wi-Fi access points, DSL-boxes, base stations to offer computing and storage resources through standardized, open mechanisms to any types of applications. The OEC solution enables an improvement in customer experience through low latency interaction with compute and storage resources just one hop away from the end-users. Similarly, ParaDrop [30] is also proposed that only considers using Edge and Fog nodes as the replacement to the centralized Cloud. Such projects focus mainly on offering distributed orchestration frameworks through which Edge and Fog resources can be assigned to running services. Such solutions can be also extended to exploit Edge-based distributed caching policies [31], which may increase the chance to react to runtime fluctuations in the workload before a performance issue arises. However, achieving the goal to have all Edge and Fog nodes as general-purpose computation and storage resources is still difficult. This is because, at first, such static Edge resources cannot be virtualized as they have physical items such as attached antennas, and hence they may not be scalable enough to handle increasing workloads at runtime.
- Discovering other available Edge and Fog computing resources during runtime, when the workload increases and elasticity must be achieved: Di Salle et al. [32] proposed a new software architecture to develop IoT systems of new generation. This architecture is highly capable of connecting smart devices with each other in order to combine services provided not only from the Cloud and network resources, but also by Things themselves. Exploiting the Edge of the network requires discovery methods to find available Edge and Fog nodes which can be leveraged in a distributed computing environment [33,34,35]. Resource discovery methods employed in the Cloud environment cannot be useful in this context for the discovery of Edge and Fog nodes. This is because Edge nodes usually are in a private network, and hence organizations should discuss regulations for using their own resources with those who may exploit these devices [36]. In this regard, Zenith [37] as a resource allocation model allows Edge and Fog infrastructure providers to establish resource sharing contracts with service providers at runtime. Consideration of multi-tenancy issues on Edge and Fog nodes comes first in such environments in which the Edge and Fog resource provider and each of customers should have different views on these infrastructures [38]. Besides that, it is not an easy task to convince other entities to make use of their own resource in order to enhance computing or storage capabilities at runtime [39]. It should be also noted that volunteer resources are generally less reliable and less predictable since they may leave the execution environment at any time.
- Replicating services in all Edge devices, Fog nodes and Cloud resources to address reliability and resilience problems: This approach includes replicated services running not only on Edge and Fog resources close to the users, but also on Cloud infrastructure [40,41,42,43] which means resources are wasted when the workload drops. In this context, if Edge and Fog resources allocated to process incoming requests are overloaded, and hence these nodes are no longer capable of improving the application QoS, additional arrived requests will be sent to the application server running on the Cloud. However, replication of servers comes with its own technical challenges. For example, temporary inconsistencies among storage or computing replicas are required to be taken into account. Moreover, different organizations have various regulations of using computing and storage infrastructures such as legislation on the geographic location of service instances or data storage servers [44].
3. Architecture and Design
4. Quality of Service Parameters for Autonomic Orchestration
4.1. Network-Related Parameters
4.1.1. Static Network-Related Parameters
4.1.2. Dynamic Network-Related Parameters
4.2. Infrastructure-Related Parameters
4.2.1. Static Infrastructure-Related Parameters
4.2.2. Dynamic Infrastructure-Related Parameters
5. Empirical Evaluation
5.1. Use Case
5.2. Experimental Design
5.3. Experimental Setup
5.4. Experimental Results
6. Conclusions
Author Contributions
Funding
Conflicts of Interest
References
- Suárez-Albela, M.; Fernández-Caramés, T.M.; Fraga-Lamas, P.; Castedo, L. A Practical Evaluation of a High-Security Energy-Efficient Gateway for IoT Fog Computing Applications. Sensors 2017, 17, 1978. [Google Scholar] [CrossRef] [PubMed]
- Song, F.; Ai, Z.; Li, J.; Pau, G.; Collotta, M.; You, I.; Zhang, H. Smart Collaborative Caching for Information-Centric IoT in Fog Computing. Sensors 2017, 17, 2512. [Google Scholar] [CrossRef] [PubMed]
- Taherizadeh, S.; Jones, A.; Taylor, I.; Zhao, Z.; Stankovski, V. Monitoring self-adaptive applications within edge computing frameworks: A state-of-the-art review. J. Syst. Softw. 2018, 136, 19–38. [Google Scholar] [CrossRef]
- Raspberry Pi 3 Model B. Available online: https://www.raspberrypi.org/products/raspberry-pi-3-model-b/ (accessed on 8 August 2018).
- Arduino. Available online: https://www.arduino.cc/ (accessed on 8 August 2018).
- RancherOS. Available online: https://rancher.com/rancher-os/ (accessed on 8 August 2018).
- CoreOS. Available online: https://coreos.com/ (accessed on 8 August 2018).
- Jarwar, M.A.; Kibria, M.G.; Ali, S.; Chong, I. Microservices in Web Objects Enabled IoT Environment for Enhancing Reusability. Sensors 2018, 18, 352. [Google Scholar] [CrossRef] [PubMed]
- Docker. Available online: https://www.docker.com/ (accessed on 8 August 2018).
- Kubernetes. Available online: https://kubernetes.io/ (accessed on 8 August 2018).
- OpenShift Origin. Available online: https://www.openshift.org/ (accessed on 8 August 2018).
- Swarm. Available online: https://docs.docker.com/swarm/ (accessed on 8 August 2018).
- Taherizadeh, S.; Novak, B.; Komatar, M.; Grobelnik, M. Real-Time Data-Intensive Telematics Functionalities at the Extreme Edge of the Network: Experience with the PrEstoCloud Project. In Proceedings of the 42nd Annual IEEE International Conference on Computers, Software and Applications (COMPSAC 2018), Tokyo, Japan, 23–27 July 2018. [Google Scholar] [CrossRef]
- Bradeško, L.; Herga, Z.; Fabijan, S.; Senožetnik, M. The Deliverable D7.3—Integrated Car2X Pilot; The OPTIMUM Project: Brussels, Belgium, 2018. [Google Scholar]
- Di-Martino, B.; Petcu, D.; Cossu, R.; Goncalves, P.; Máhr, T.; Loichate, M. Building a mosaic of clouds. In Proceedings of the Euro-Par Parallel Processing Workshop, Ischia, Italy, 31 August–3 September 2010; Springer: Berlin/Heidelberg, Germany, 2010; pp. 571–578. [Google Scholar]
- Juju. Available online: https://jujucharms.com/ (accessed on 8 August 2018).
- The SWITCH Project. Available online: http://www.switchproject.eu/ (accessed on 8 August 2018).
- Patel, A.; Daftedar, M.; Shalan, M.; El-Kharashi, M.W. Embedded hypervisor Xvisor: A comparative analysis. In Proceedings of the 23rd Euro-Micro International Conference on Parallel, Distributed and Network-Based Processing (PDP 2015), Turku, Finland, 4–6 March 2015. [Google Scholar]
- Trofimov, V.V.; Kiyaev, V.I.; Gazul, S.M. Use of virtualization and container technology for information infrastructure generation. In Proceedings of the 2017 IEEE International Conference on Soft Computing and Measurements (SCM), St. Petersburg, Russia, 24–26 May 2017; IEEE: New York, NY, USA, 2017; pp. 788–791. [Google Scholar]
- Bellavista, P.; Giannelli, C.; Lanzone, S.; Riberto, G.; Stefanelli, C.; Tortonesi, M. A Middleware Solution for Wireless IoT Applications in Sparse Smart Cities. Sensors 2017, 17, 2525. [Google Scholar] [CrossRef] [PubMed]
- Liang, H.; Xing, T.; Cai, L.X.; Huang, D.; Peng, D.; Liu, Y. Adaptive computing resource allocation for mobile cloud computing. Int. J. Distrib. Sens. Netw. 2013, 9. [Google Scholar] [CrossRef]
- Taherizadeh, S.; Stankovski, V. Dynamic multi-level auto-scaling rules for containerized applications. Comput. J. 2018. [Google Scholar] [CrossRef]
- Google Container Engine. Available online: https://Cloud.google.com/container-engine/ (accessed on 8 August 2018).
- Amazon EC2 Container Service. Available online: https://aws.amazon.com/ecs/ (accessed on 8 August 2018).
- Taherizadeh, S.; Stankovski, V. Auto-scaling applications in Edge computing: Taxonomy and challenges. In Proceedings of the International Conference on Big Data and Internet of Thing, London, UK, 20–22 December 2017; ACM: New York, NY, USA, 2017; pp. 158–163. [Google Scholar] [CrossRef]
- Apache Mesos. Available online: http://mesos.apache.org/ (accessed on 8 August 2018).
- Alam, K.M.; Saini, M.; El Saddik, A.E. Workload Model Based Dynamic Adaptation of Social Internet of Vehicles. Sensors 2015, 15, 23262–23285. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- The LightKone Project. Available online: https://www.lightkone.eu/ (accessed on 8 August 2018).
- The Open Edge Computing Project. Available online: http://openEdgecomputing.org/ (accessed on 8 August 2018).
- Liu, P.; Willis, D.; Banerjee, S. Paradrop: Enabling lightweight multi-tenancy at the network’s extreme edge. In Proceedings of the IEEE/ACM Symposium on Edge Computing (SEC), Washington, DC, USA, 27–28 October 2016. [Google Scholar]
- Bhardwaj, K.; Agrawal, P.; Gavrilovska, A.; Schwan, K. Appsachet: Distributed app delivery from the Edge Cloud. In Proceedings of the International Conference on Mobile Computing, Applications, and Services, Berlin, Germany, 12–13 November 2015; Springer: Cham, Switzerland, 2015; pp. 89–106. [Google Scholar]
- Di-Salle, A.; Gallo, F.; Pompilio, C. Composition of Advanced (µ) Services for the Next Generation of the Internet of Things. In Proceedings of the International Conferences on Software Technologies: Applications and Foundations, Vienna Austria, 4–8 July 2016; Springer: Cham, Switzerland, 2016; pp. 436–444. [Google Scholar]
- Sahni, Y.; Cao, J.; Zhang, S.; Yang, L. Edge Mesh: A new paradigm to enable distributed intelligence in Internet of Things. IEEE Access 2017, 5, 16441–16458. [Google Scholar] [CrossRef]
- Ryden, M.; Oh, K.; Chandra, A.; Weissman, J. Nebula: Distributed Edge Cloud for data intensive computing. In Proceedings of the 2014 IEEE International Conference on Cloud Engineering (IC2E), Boston, MA, USA, 10–14 March 2014; pp. 57–66. [Google Scholar]
- Amento, B.B.; Balasubramanian, B.; Hall, R.J.; Joshi, K.; Jung, G.; Purdy, K.H. FocusStack: Orchestrating Edge Clouds Using Location-Based Focus of Attention. In Proceedings of the IEEE/ACM Symposium on Edge Computing, Washington, DC, USA, 27–28 October 2016; pp. 179–191. [Google Scholar]
- Varghese, B.; Wang, N.; Barbhuiya, S.; Kilpatrick, P.; Nikolopoulos, D.S. Challenges and opportunities in edge computing. In Proceedings of the IEEE International Conference on Smart Cloud (SmartCloud), New York, NY, USA, 18–20 November 2016; pp. 20–26. [Google Scholar]
- Xu, J.; Palanisamy, B.; Ludwig, H.; Wang, Q. Zenith: Utility-aware resource allocation for Edge computing. In Proceedings of the 2017 IEEE International Conference on Edge Computing (EDGE), Honolulu, HI, USA, 25–30 June 2017; IEEE: New York, NY, USA, 2017; pp. 47–54. [Google Scholar]
- Samaniego, M.; Deters, R. Supporting IoT Multi-Tenancy on Edge Devices. In Proceedings of the 2016 IEEE International Conference on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData), Chengdu, China, 15–18 December 2016; IEEE: New York, NY, USA, 2016; pp. 66–73. [Google Scholar]
- Shiraz, M.; Gani, A. A Lightweight Active Service Migration Framework for Computational Offloading in Mobile Cloud Computing. J. Supercomput. 2014, 68, 978–995. [Google Scholar] [CrossRef]
- Hu, W.; Gao, Y.; Ha, K.; Wang, J.; Amos, B.; Chen, Z.; Pillai, P.; Satyanarayanan, M. Quantifying the impact of Edge computing on mobile applications. In Proceedings of the 7th ACM SIGOPS Asia-Pacific Workshop on Systems, Hong Kong, China, 4–5 August 2016; ACM: New York, NY, USA, 2016. [Google Scholar]
- Xu, X.; Fu, S.; Cai, Q.; Tian, W.; Liu, W.; Dou, W.; Sun, X.; Liu, A.X. Dynamic Resource Allocation for Load Balancing in Fog Environment. Wirel. Commun. Mob. Comput. 2018. [Google Scholar] [CrossRef]
- Yu, Y. Mobile Edge computing towards 5G: Vision, recent progress, and open challenges. China Commun. 2016, 13, 89–99. [Google Scholar] [CrossRef]
- Jararweh, Y.; Ababneh, F.; Khreishah, A.; Dosari, F. Scalable Cloudlet-based mobile computing model. Procedia Comput. Sci. 2014, 34, 434–441. [Google Scholar] [CrossRef]
- Gholami, A. Security and Privacy of Sensitive Data in Cloud Computing. Ph.D. Thesis, KTH Royal Institute of Technology, Stockholm, Sweden, 2016. [Google Scholar]
- Rutten, E.; Marchand, N.; Simon, D. Feedback control as MAPE-K loop in autonomic computing. In Software Engineering for Self-Adaptive Systems III Assurances; Springer: Cham, Switzerland, 2017; pp. 349–373. [Google Scholar]
- Arcaini, P.; Riccobene, E.; Scandurra, P. Modeling and analyzing MAPE-K feedback loops for self-adaptation. In Proceedings of the 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems, Firenze, Italy, 18–19 May 2015; IEEE: New York, NY, USA, 2015; pp. 13–23. [Google Scholar] [CrossRef]
- The Developed Monitoring System. Available online: https://github.com/salmant/ASAP/tree/master/SWITCH-Monitoring-System (accessed on 8 August 2018).
- StatsD Protocol. Available online: https://github.com/etsy/statsd/wiki (accessed on 8 August 2018).
- Apache Cassandra. Available online: http://cassandra.apache.org/ (accessed on 8 August 2018).
- Han, R.; Guo, L.; Ghanem, M.M.; Guo, Y. Lightweight resource scaling for Cloud applications. In Proceedings of the 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid 2012), Ottawa, ON, Canada, 13–16 May 2012; pp. 644–651. [Google Scholar] [CrossRef]
- Lu, Z.; Wu, J.; Bao, J.; Hung, P.C. OCReM: Openstack-based Cloud datacentre resource monitoring and management scheme. Int. J. High Perform. Comput. Netw. 2016, 9, 31–44. [Google Scholar] [CrossRef]
- Singhi, G.; Tiwari, D. A load balancing approach for increasing the resource utilization by minimizing the number of active servers. Int. J. Comput. Secur. Source Code Anal. 2017, 3, 11–15. [Google Scholar]
- Monil, M.A.H.; Rahman, R.M. Implementation of modified overload detection technique with VM selection strategies based on heuristics and migration control. In Proceedings of the 2015 IEEE/ACIS 14th International Conference on Computer and Information Science (ICIS), Las Vegas, NV, USA, 28 June–1 July 2015; pp. 223–227. [Google Scholar] [CrossRef]
- Alonso, A.; Aguado, I.; Salvachua, J.; Rodriguez, P. A metric to estimate resource use in Cloud-based videoconferencing distributed systems. In Proceedings of the 2016 IEEE 4th International Conference on Future Internet of Things and Cloud (FiCloud), Vienna, Austria, 22–24 August 2016; pp. 25–32. [Google Scholar] [CrossRef]
- Alonso, A.; Aguado, I.; Salvachua, J.; Rodriguez, P. A methodology for designing and evaluating Cloud scheduling strategies in distributed videoconferencing systems. IEEE Trans. Multimedia 2017, 19, 2282–2292. [Google Scholar] [CrossRef]
- The SWITCH Alarm-Trigger Component. Available online: https://github.com/salmant/ASAP/tree/master/SWITCH-Alarm-Trigger (accessed on 8 August 2018).
- Valenzuela-Escárcega, M.A.; Hahn-Powell, G.; Surdeanu, M. Odin’s Runes: A Rule Language for Information Extraction. In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC 2016), Portorož, Slovenia, 23–28 May 2016; pp. 322–329. [Google Scholar]
- Mukib, A.; Shahidul, M.; Mia, S. Energy Efficient Routing Protocol for Sensing Data. Int. J. Comput. Appl. 2017, 170. [Google Scholar] [CrossRef]
- Docker Remote API. Available online: https://docs.docker.com/engine/api/v1.21/ (accessed on 8 August 2018).
- Java Code to Implement the On/Offloading Procedure. Available online: https://github.com/salmant/PrEstoCloud/blob/master/OnOffloading/On_Offloading_Client.java (accessed on 8 August 2018).
- Zhao, S.; Yu, L.; Cheng, B. A Real-Time Web of Things Framework with Customizable Openness Considering Legacy Devices. Sensors 2016, 16, 1596. [Google Scholar] [CrossRef] [PubMed]
- Stankovski, V.; Trnkoczy, J.; Taherizadeh, S.; Cagelu, M. Implementing time-critical functionalities with a distributed adaptive container architecture. In Proceedings of the 18th International Conference on Information Integration and Web-Based Applications and Services, Singapore, 28–30 November 2016; ACM: New York, NY, USA, 2016; pp. 453–457. [Google Scholar]
- OPTIMUM Project. Available online: www.optimumproject.eu/ (accessed on 8 August 2018).
- Adria Mobil Company. Available online: www.adria-mobil.com/ (accessed on 8 August 2018).
- VESNA Platform. Available online: http://sensorlab.ijs.si/hardware.html (accessed on 8 August 2018).
- Lavassani, M.; Forsström, S.; Jennehag, U.; Zhang, T. Combining Fog Computing with Sensor Mote Machine Learning for Industrial IoT. Sensors 2018, 18, 1532. [Google Scholar] [CrossRef] [PubMed]
- Huang, Q.; Yang, Y.; Shi, Y. SmartVeh: Secure and Efficient Message Access Control and Authentication for Vehicular Cloud Computing. Sensors 2018, 18, 666. [Google Scholar] [CrossRef] [PubMed]
Feature | Containers | VMs |
---|---|---|
Requirement | Container engine e.g., Docker | Hypervisor e.g., Xvisor |
Weight | Lightweight | Heavyweight |
Size | Small | Large |
Boot Time | Fast | Slow |
QoS Parameter’s Type | Static/Dynamic | Name | Abbreviation | Measurement Unit |
---|---|---|---|---|
Network-related | Static | Number of Hops | NH | # |
Network Bandwidth | NB | MBps | ||
Dynamic | Packet Loss | PL | % | |
Network Throughput | NT | MBps | ||
Network Delay | ND | Ms | ||
Network Jitter | NJ | Ms | ||
Infrastructure-related | Static | Number of CPUs | NC | # |
Amount of Memory | AM | MB | ||
Size of Disk space | SD | GB | ||
Dynamic | Percentage of CPU | PC | % | |
Percentage of Memory | PM | % | ||
Free Disk | FD | MB |
Feature | Fog Nodes | Cloud Node | Dedicated JSI Server |
---|---|---|---|
OS | Ubuntu 14.04.5 LTS | Ubuntu 16.04.4 LTS | Ubuntu 16.04.4 LTS |
CPU(s) | 1 | 1 | 4 |
CPU MHz | 2397.222 | 2399.998 | 2659.998 |
Cache size | 4096 KB | 16,384 KB | 6144 KB |
Memory | 4096 MB | 4096 MB | 16,384 MB |
Disk | 10 GB | 10 GB | 80 GB |
Bandwidth | 1000 MBps | 1000 MBps | 1000 MBps |
Provider | ARNES | UL | JSI |
Important Properties Achieved During Execution | Edge | Edge-Fog | Edge-Cloud |
---|---|---|---|
Average response time offered before on/offloading event | 35.60 | 35.51 | 35.44 |
Average response time offered after on/offloading event | none | 948.31 | 3011.69 |
Median response time obtained before on/offloading event | 35.25 | 35.25 | 35 |
Median response time obtained after on/offloading event | none | 948.25 | 3010.5 |
99th percentile of the response time provided before on/offloading event | 42.25 | 42.75 | 44.25 |
99th percentile of the response time provided after on/offloading event | none | 954.60 | 3901.52 |
Standard deviation of response time achieved before on/offloading event | 3.32 | 2.92 | 3.22 |
Standard deviation of response time achieved after on/offloading event | none | 2.62 | 364.50 |
Number of detected events | 6 | 6 | 6 |
© 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Taherizadeh, S.; Stankovski, V.; Grobelnik, M. A Capillary Computing Architecture for Dynamic Internet of Things: Orchestration of Microservices from Edge Devices to Fog and Cloud Providers. Sensors 2018, 18, 2938. https://doi.org/10.3390/s18092938
Taherizadeh S, Stankovski V, Grobelnik M. A Capillary Computing Architecture for Dynamic Internet of Things: Orchestration of Microservices from Edge Devices to Fog and Cloud Providers. Sensors. 2018; 18(9):2938. https://doi.org/10.3390/s18092938
Chicago/Turabian StyleTaherizadeh, Salman, Vlado Stankovski, and Marko Grobelnik. 2018. "A Capillary Computing Architecture for Dynamic Internet of Things: Orchestration of Microservices from Edge Devices to Fog and Cloud Providers" Sensors 18, no. 9: 2938. https://doi.org/10.3390/s18092938