The Internet of Things (IoT) is expected to consist in a worldwide network comprising, by 2020, more than 50 billions devices. This gigantic number of pervasively deployed devices will (i) enable new forms of interaction between things and people and (ii) foster novel applications, denoted as Smart-X Applications. The IoT will be characterized by the extreme heterogeneity of the involved devices, which will range from highly constrained Class 0, Class 1 and Class 2 devices [1
] (operating in Low power and Lossy Networks, IP(V6) LLNs), to smartphones, traditional hosts, and the Cloud. IP has been foreseen as the driver for a successful growth of the IoT [2
]. An IP-based protocol stack will also allow to maximize the reutilization of the experience derived from the development of Internet architectures and protocols. IP will, in fact, enable maximum interoperability among devices and integration with the existing Internet. Access to LLNs typically occurs through a border router, a network element with multiple network interfaces: one towards the LLN and the others toward the Internet using different protocols, either wired (e.g., IEEE 802.3) or wireless (e.g., IEEE 802.11, low power Wi-Fi, IEEE 802.15.4, BLE). Due to its less limited capabilities, the border router acts as an IPv6 network coordinator for the LLN, typically being the root of a IPv6 Routing Protocol for Low-Power and Lossy Networks (RPL) tree [3
The work carried out by standardization organizations, such as the Internet Engineering Task Force (IETF), and several research projects has focused on defining mechanisms to bring IP connections to constrained devices and to design IoT-oriented application layer protocols, such as the Constrained Application Protocol (CoAP) [4
]. CoAP is a lightweight, UDP-based, binary protocol designed to bring the REpresentational State Transfer (REST) paradigm to the IoT, targeting constrained applications. According to the request/response model of the REST paradigm, smart objects expose their resources using CoAP. REST has been selected as a reference communication paradigm for IoT, as it minimizes the coupling between client and server applications and promotes long-term evolution of applications, which is particularly desirable in IoT scenarios, where devices are expected to be deployed and remain operational for years.
At the application layer, other protocols have been designed for efficient communications in IoT and Machine-to-Machine (M2M) scenarios. Message Queue Telemetry Transport (MQTT) is a lightweight publish/subscribe (pub/sub) protocol over TCP/IP [5
]. MQTT uses topics to “tag” messages, which are sent by publishers to a broker that, according to topic subscriptions, forwards messages to subscribers. Unlike CoAP, MQTT is not suited to highly constrained devices operating in LLNs: TCP connection establishment is a heavy operation, that might not be feasible for devices with limited resources and operating over lossy links. In order to overcome this pitfall, an extension of MQTT for Sensor Network (MQTT-SN) has been introduced. The pub/sub model implemented by MQTT enables reliable communications and minimizes network bandwidth, which make it perfectly suitable for fast transmissions among non-constrained devices.
Billions of IoT sensing/actuating devices will generate an unprecedented amount of data, which will need to be managed properly in order to provide highly available and robust services. Although IP-based IoT would make it possible to address smart objects directly, the following potential drawbacks exist: (i) in order to extend their battery lifetime, smart objects may be duty-cycled devices and, thus not always accessible; (ii) smart objects might be unable, to handle a large number of concurrent requests, thus leading to service disruption and becoming possible targets for Denial-of-Service (DoS) attacks; (iii) under some circumstances (for instance, to minimize memory footprint when the available in-device memory is critically low), smart objects may play the role of clients rather than servers.
In all the above cases, the presence of an intermediate network element operating at the application layer, which may typically coincide with the border router, is desirable. Such a node could integrate several functionalities which may reduce the processing load on smart objects and help overcome some of the problems discussed above (e.g., caching). This resourceful node, denoted as “IoT Hub” and shown in Figure 1
, helps moving some of the processing load towards the edge of the network, following the emerging paradigm of “Fog Computing” [6
Fog Computing, also referred to as Edge Computing, is a novel paradigm, which has been introduced to meet several requirements such as mobility support, low latency, and location awareness [6
]. Fog Computing aims at distributing and moving some Cloud-based computation and storage to the edge of the network. While Cloud-only architectures can provide a solution to scalability and flexibility issues by distributing resources among multiple servers, this approach presents some weaknesses, such as: (i) latency; (ii) availability/dependence on Internet connectivity for operations; (iii) flexible networking; (iv) quality of service/experience; (v) security and privacy. Due to its benefits over Cloud-based architectures, especially if time is a critical issue or Internet connectivity is poor or absent, Fog Computing is expected to play a key role in the deployment of IoT applications. The Fog is not intended to replace the Cloud, but rather to complement it, in order to provide location-aware and real-time services, thus enabling new applications that could have not been deployed otherwise. Characteristics features of Fog Computing are the following:
Geographically distributed, in contrast with a fully Centralized solution associated with Cloud-based architectures;
Low latency and high performance to support real-time applications;
Support for mobility.
According to the principles of Fog Computing and the fact that it will represents the reference architecture for IoT, the IoT Hub represents a critical building block for the deployment and manageability of complex IoT scenarios. The IoT Hub will allow Smart Objects to interoperate at a local level and to enable the design of new applications and interaction patterns of the Edge (e.g., analytics, machine learning, and human-to-machine interactions). In this context and architectural vision, the role of the IoT Hub becomes central in the foreseen IoT architecture, since it should process all requests providing the following functionalities and benefits:
Independence from Internet Connectivity allows to guarantee continuous service operation at the local level with no disruption in case of unreachability of Cloud. In case Internet Access is temporarily unavailable, the IoT Hub will keep the state of all Smart Objects and will automatically synchronize with the Cloud as soon as the connection resumes;
Controlled and secure access to things. Direct access to Smart Object occurs only at the local level and may be filtered by the IoT Hub Border Router: the IoT Hub is the gateway/bridge between one or more constrained networks (e.g., IEEE 802.15.4);
Service and Resource Discovery: the IoT Hub is able to discover which SOs are available in the network and, subsequently, to discover the resources that they host;
Resource Directory (RD): the IoT Hub maintains a list of all CoAP resources available in the constrained networks;
Origin Server (OS): it provides a CoAP server where resources are hosted or are to be created;
Protocol Translation: (i) CoAP-to-CoAP (C2C) Proxy: it provides proxying capabilities for CoAP requests coming from external clients that should reach internal constrained nodes; and (ii) HTTP-to-CoAP (H2C) Proxy: it provides HTTP-to-CoAP cross-proxying (i.e., protocol translation) in order to enable access to CoAP resources by HTTP clients;
Cache: in order to avoid unnecessary load on SOs and to minimize latencies, a cache is kept with the representation of most recently accessed resources.
In order to increase the robustness of an IoT Hub-oriented architecture, in this work we propose a Cloud-supported replication mechanism for IoT Hubs in order to efficiently manage CoAP resources in a scalable and secure way. The proposed mechanism relies on the possibility to exploit Cloud platforms to clone and virtualize IoT Hubs. Replicas are full copies of IoT Hubs, thus implementing all their functionalities, and can thus be accessed on behalf of actual physical nodes. The proposed approach introduces several benefits: (i) a common Cloud-based interface for accessing available resources; (ii) the behavior, the logic and the implementation of the IoT Hub are hidden from communication with the aim to protect the IoT Hub and the smart objects behind it; (iii) the introduction of Cloud balancing policies in order to scale up or down the number of replicas according to needs, depending on the number of incoming/estimated requests and managed resources.
This work focuses on the definition, implementation, and evaluation of all functionalities (standardization, communication, synchronization, and caching) that the IoT Hub will provide in order, on the one hand to tackle the complexity of handling heterogenous Smart Objects making them available for applications running on the Edge and, on the other hand to provide an efficient synchronization with the Cloud for remote interactions.
The rest of this paper is organized as follows. In Section 2
, an overview of related work, with focus on Fog Computing principles and its integration with the IoT is presented. Section 3
describes the proposed Cloud-based resource management architecture. In Section 4
, an extensive experimental analysis of the proposed architecture is presented and the corresponding performance is investigated in a meaningful scenario. Finally, in Section 5
we draw our conclusions.
2. Related Work
The role of Cloud Computing in the IoT is gaining greater and greater attention. Most research has been so far smart object-driven, focusing mainly on the definition of IP-based, low-power, and efficient communication protocols and mechanisms. Many of these aspects have now been significantly addressed. Several IoT solutions have been deployed and brought to the market in several application scenarios, from Home Automation to Smart Cities. Most of these fragmented and vertical solutions rely on the Cloud, in order to provide a centralized access to services exploiting data that are sent uplink from deployed sensors to Cloud storage. Typically, mobile apps “consuming” such services are made available to end-users. However, this approach, which is expedient to disseminate the concept of IoT, does not fully exploit the potential of the Cloud.
As billions of smart objects are expected to be deployed pervasively, efficient data processing has highlighted the need to rely on the Cloud. The Cloud of Things (CoT) refers to the interaction between IoT and the Cloud [7
]. In [8
], an architecture for integrating Cloud/IoT is proposed, based on a network element, denoted as “Smart Gateway”, which is intended to act as intermediary between heterogeneous networks and the Cloud. The role of the Smart Gateway is similar to that of the IoT Hub, in terms of supporting several heterogeneous networks. However, the role of the Cloud is mainly envisioned as a data storage and aggregator, which can be used by end-users to access data. According to this approach, data are sent uplink, making it impossible to directly address and act on smart objects, as the IoT is supposed to do. At the opposite, in the current paper we envision that the Cloud, by hosting replicas of the IoT Hub, is also used as an enabler for direct and efficient access to resources, while providing desirable features, such as: seamless access by external clients; security; and high availability. In [9
] the concept of Smart Service Proxy is presented and described. It propose an interesting solution combining the use of IoT protocols and technologies with Semantic Web solutions. The proxy provides a unified access to resources/nodes through Proxying and Caching solution and introduce a dedicated Directory Service dedicated for the management of object representations, queries and ontology.
Fog Computing brings a new approach to Internet access networks by making computation, storage, and networking resources available at the edge of access networks. This improves the performance, by minimizing latency and availability, since resources are accessible even if Internet access is not available [10
]. Fog-based solutions aim at introducing an intermediate architectural layer where resources and applications are made available in the proximity of end devices, thus avoiding continuous access to the Cloud.
Fog-based access networks are based on the presence of highly specialized nodes, denoted as Fog Nodes, able to run distributed applications at the edge of the network. In particular, the deployment of computing resources on Internet access networks allows to dinamically activate Virtual Machines (VMs) dynamically on Fog Nodes. For this reason, the cloning and synchronization techniques of VMs at the core of this work fit perfectly into Fog-based infrastructures, as will be discussed in more detail in Section 3
. The proposed architecture can protect local resources by providing remote access to their replicas in a transparent way. Local resources are kept synchronized by multiple clones of the same machine, thus achieving a high level of reliability and load balancing. Smart management of the activation/deactivation of replicas and choice of the most appropriate Fog Node to run the clone allows to optimize the usage of CPU and memory available on the infrastructure, according to the specific real-time resources requirements by running applications.
The adoption of Virtualization Technologies for Wireless Sensor Networks has been studied and evaluated in several aspects during last years. In [11
] the researchers present the implementation of a testbed where physical, simulated, and emulated sensor nodes cooperate and interact in real time extending the experimentation capabilities of a deployment. With the arrival of the IoT revolution, this technologies have been enriched with new features and functionalities in order to augment the proposed platforms and solutions. For example the authors of [12
] introduce an interesting and innovative platform supporting different aspects of Wireless Sensor Network virtualization in order to handle the heterogenous requirements typical of IoT applications.
] the authors present a scalable virtual sensor framework to support the build of logical data flows related to either physical sensors or custom virtual sensors. Although the presented approach is interesting, the presented technology is not applied on the Edge to handle synchronization between local and remote environment. The authors of [14
] propose the use Virtual IoT Devices on the Edge for local data processing, management of physical devices, and quick actuation. The Cloud is envisioned only as a way to access the Edge infrastructure remotely but without the introduction of Gateway and/or Objects replicas and the related synchronization procedures.
A suitable lightweight alternative to VMs is represented by containers, which provide a more flexible environment for “disposable applications”, like the IoT Hub. Container platforms like Docker [15
] are gaining increasing attention also for edge and Fog computing applications. In this challenging scenario, the possibility of moving from centralized to decentralized paradigm to offload the processing to the edge reducing application response time and improving overall user experience will play a fundamental role in Internet of Things. In [18
], the authors present how a container-based architecture could be efficiently used for dynamic networking application. In [20
], an interesting comparison about existing lightweight and hypervisor-based approaches for edge computing and efficient networking is presented. Furthermore, in [21
] a novel approach for the application of a lightweight virtualization technology (such as Docker) to constrained devices with a negligible overhead is presented. In this work, a containerized version of the IoT Hub, based on Docker, will be considered.
The authors of [22
] present the use of virtualization techniques in the context of IoT Gateway management. In particular, they introduce the concept of Gateway-as-a-Service, a lightweight instance that can be shared between different users thanks to the use of virtualization techniques. This interesting work is focused on the adoption of virtualization techniques (e.g., Virtual Machines and Containers) on the Edge but does not take into account the virtualization and synchronization of gateways and Smart Objects with the Cloud and how this procedure can be seamless for users and services both on local and remote environments.
The purpose of the work is to propose an architecture based of a locally deployed network element (the IoT Hub) for the seamless and ubiquitous access to IoT resources: (i) within the same local network where Smart Objects are deployed; or (ii) externally through a Cloud service with high availability (Smart Object virtual replicas). This work provides the details also about the synchronization mechanisms between the state of resources maintained by the IoT Hub and their Cloud replicas. This solution will enables remote access to resources in networks in a fully transparent and standardized way. In order to achieve our goals, a new application layer for IoT networks is designed and developed. In particular, we focus on the design of an architecture that allows the access by external clients, by virtualizing the functionalities of an IoT network. The details of the IoT network, such as its location or its actual implementation, are kept hidden from users and external clients willing to access resources. In other words, resource access by remote clients will be mediated by the Cloud platform, which provides a standard and secure front-end.
3.1. The IoT Hub
As anticipated in Section 1
, the proposed architecture is based on a network element component, denoted as “IoT Hub”. The IoT Hub does not have the same strict requirements on energy consumption and processing capabilities as other smart objects and is thus expedient to provide relevant features to a constrained network. The IoT Hub is placed at the edge of the constrained network and plays a fundamental role by implementing the following functions, as summarized in the functional plane, mapped in the protocol stack, shown in Figure 2
Service and Resource Discovery: the IoT Hub is able to discover which SOs are available in the network and, subsequently, to discover the resources that they host;
LoWPAN Border Router: at the network layer, the IoT Hub is the gateway between one or more constrained networks (e.g., IEEE 802.15.4) it belongs to (through some radio interfaces it is equipped with).
CoAP/CoAP (C2C) Proxy: at the application layer, it provides proxying capabilities for CoAP requests coming from external clients that should reach internal constrained nodes.
HTTP/CoAP (H2C) Proxy: at the application layer, it provides cross-proxying (protocol translation) between HTTP and CoAP in order to let external HTTP clients access CoAP resources hosted by smart objects.
Resource Directory: the IoT Hub maintains a list of all CoAP resources available in the constrained network. These resources may have been gathered through several mechanisms, such as those described in [23
Cache: in order to avoid unnecessary load on smart objects and to minimize latencies, a cache is kept with the representation of most recently accessed resources.
Replica Manager: a software module responsible for the coordination and the synchronization between the IoT Hub and its replicas.
Moreover, due to the constrained nature of smart objects, they cannot typically implement strong security mechanisms and access policies, for instance, those related to authorization, whereby the IoT Hub may act as a filter for incoming requests in order to limit access to specific resources. Because of all the functionalities outlined above, the IoT Hub may suffer from the following critical issues, which may undermine the lifecycle of a constrained IoT network:
the IoT Hub is a bottleneck of the architecture, since all traffic must pass through it even if it is not a communication endpoint;
failure of the IoT Hub would make the resources hosted by smart objects temporarily or permanently unavailable.
It is necessary to relieve the IoT Hub from some of this load in order to guarantee that resources can be accessed with high availability in a secure and seamless way.
In this work, we propose to rely on the Cloud by providing virtual replicas of the IoT Hub. Replicas of the IoT Hub are fully functional clones, which may be used by any external client to access resources in the same way as they would do with the actual IoT Hub. Interacting with resources through the replicas allows to provide an access point different from the real (physical) IoT Hub, thus decoupling the constrained network management function and granting access to external clients. Replicas of the IoT Hub are synchronized through a dedicated MQTT-based protocol, which is used to transfer copies of the resources from the IoT Hub to the replicas in a pub/sub model. Replicas can be instantiated on-the-fly, according to particular needs, thus also providing load balancing (according to policies which may depend on the number of connected clients and smart objects) and acting as recovery facilities, in the presence of temporary failure of the real IoT Hub.
3.2. Operational Scenarios
Resource access through the proposed Cloud-based platform can occur according to the following three operational models implemented by a smart object: (i) CoAP server; (ii) observable CoAP server; and (iii) CoAP client.
3.2.1. Polling Resources
If the smart object is a CoAP server, it can receive requests to access its hosted resources. In this case, shown in Figure 3
, the message flow is as follows: (1) the external client sends a HTTP or CoAP request to the Cloud platform front-end, which (2) forwards the request to one selected replica of the IoT Hub.
The CoAP protocol [4
] defines the Max-Age option to indicate the maximum time a response may be cached before it is considered not fresh. Since Smart Objects, the IoT Hub, and Virtual Replicas all implement CoAP they are able invalidate their caches autonomously based on this information.
If the replica has a matching cached “fresh” resource, it can return it immediately; otherwise, (3) the replica forwards the request to the actual IoT Hub (which is securely connected through a VPN tunnel): if the IoT Hub has a matching cached fresh resource, it can return it immediately; otherwise, (4) it acts as a reverse proxy and forwards a (and, if needed, translated) CoAP request to the CoAP server, which (5) returns the resource. At this point, the returned resource is cached by the IoT Hub, and (6) returned to the replica, which, in turn, (7, 8) sends it back to the client. The resource cached at the IoT Hub is then ( and ) synchronized with its replicas, in order to speed up and efficiently manage subsequent requests targeting the same resource.
3.2.2. Observing Resources
The Observe option [24
] is a CoAP option which allows resource observing. According to this specification, a CoAP client can send a single request, including an Observe option with a value of 0 (register), in order to register its interest in receiving updates related to the targeted resource. The CoAP server sends a response each time the resource value is updated. In this case, multiple responses are sent after a single request. The Observer option allows to implement a notification-based communication model, thus reducing network traffic. The observing scenario is shown in Figure 4
. If the targeted smart object is a CoAP server which implements the Observe option, it can receive requests to access its hosted resources, which may be either observable or not. In the latter case, requests are handled as in the polling case. In the former case, instead, the message flow resembles that of the polling scenario, but (i) the IoT Hub observes the resource and (ii) the external client observes the cached resource on the replica. When the resource is updated, (i) the smart object will send a notification to the IoT Hub; (ii) the IoT Hub will synchronize the resource with its replica; and, finally, (iii) the replica will send a notification to the external observing client.
3.2.3. Pushing Resources
Sometimes, the memory constraints of smart objects make it unfeasible to let them act as CoAP servers. In this case, described in Figure 5
, the smart object acts as a CoAP client and sends CoAP requests (POST and PUT) to the IoT Hub, which plays the role of origin server, by maintaining resources on behalf of the clients. If the smart object is a CoAP client, according to the semantics of CoAP methods, the following message flow takes place: (1) the smart object sends CoAP POST requests to the IoT Hub in order to create resources and CoAP PUT requests in order to change their value. When the IoT Hub receives POST and PUT requests, after properly handling these requests it accordingly stores them and (2,
) synchronizes them on the replica. When (3) an external client sends an HTTP or a CoAP request to the Cloud platform front-end, it (4) the latter forwards the request to one selected replica of the IoT Hub. Since the replica is synchronized with the IoT Hub, (5, 6) the replica can respond immediately with the stored representation of the resource.
3.3. Synchronization Protocol
The diagram presented in Figure 6
illustrates the operations and involved message exchange during the synchronization phases between involved actors. The operations related to the IoT Hub are detailed as follows:
The IoT Hub discovers existing Smart Objects that are deployed in the same network using multiple service discovery mechanisms such as ZeroConf [25
] and UPnP [27
] and the list of all the resources that they host.
The IoT Hub starts observing all the discovered resources, according to [24
] to receive updates related to state changes.
Any time the IoT Hub receives such updates it populates/refreshes its resource cache.
The IoT Hub publishes a message (using an MQTT topic bound to the resource and the IoT Hub) related to the resource state change to the Cloud Broker to let Virtual Replicas update their mirrored state.
The operations related to the Virtual Replicas are detailed as follows:
Upon launch the Virtual Replicas subscribe to the Broker to receive updates for topics related to a target IoT Hub.
Any time the IoT Hub publishes an update for a resource the Virtual Replica will receive a message and will update its state accordingly.
Client applications that wish to interact with resources operate in the following ways:
Local Clients issue CoAP requests directly to the Smart Objects (if it implements CoAP) or to the IoT Hub otherwise.
Remote and External Clients may interact with a Smart Objects only through the Virtual Replica which will be publicly accessible using its REST interface.
The synchronization protocol used in the architecture implements a pub/sub communication model. In fact, communication follows a one-to-many pattern from the IoT Hub to all of its replicas. All messages are sent by the IoT Hub to an MQTT message broker (hosted on the Cloud platform and illustrated in Figure 7
a) using specific MQTT topics (which can be used to selectively target one, many, or all replicas in order to implement unicast, multicast, or broadcast communications respectively) that is managed by the Cloud platform, using a VPN connection, for security and addressing reasons, as shown in Figure 7
a. The IoT Hub and its replicas are all identified by a system-wide identifier, assigned by the designed Cloud platform.
Each IoT Hub includes a Replica Manager (RM), which is a dedicated software module responsible for the synchronization among the IoT Hub and its replicas. The RM is composed by the following items, as shown in Figure 7
A Replica Registry (RR), which contains the list of the identifiers of all the replicas of the IoT Hub.
An MQTT subscriber, which registers to the broker to receive messages related to two topics: (i) its own identifier () and (ii) the identifier of the actual IoT Hub (); these may coincide in the case of the actual IoT Hub.
An MQTT publisher, which publishes messages to the broker, using the method , where t is the topic and m is the message to be published.
The IoT Hub is in charge of keeping full synchronization of its resources with its replicas, in order to ensure that all requests are served in the same way, regardless of the specific replica that was targeted by the client. Synchronization comes into play every time a resource on the IoT Hub changes. This can be caused by different events. At startup, a replica of the IoT Hub needs to synchronize with the actual IoT Hub. The procedure is shown in Figure 8
. The RM of the replica publishes its
to the topic
, in order to inform of its creation (
). At this point, the IoT Hub updates its RR by adding
and then starts publishing to the broker all the resources (R) using the topic
), which guarantees that the new replica will receive the resources. When the synchronization procedure has ended, the replica will be automatically kept synchronized with the IoT Hub during the normal system lifecycle.
When resources are polled for (as will be described in Section 3.2.1
), the IoT Hub might find out that a resource targeted by some requests has changed. A request targeting a resource that either has not been cached or is not considered fresh must be forwarded by the IoT Hub to the smart object. Upon receiving the response from the smart object, after updating its cache and forwarding the response to the requesting replica, the IoT Hub uses the synchronization protocol to publish the updated information to all of its replicas. When observing resources (as described in Section 3.2.2
), the synchronization procedure resembles the same of the polling case. When resources are pushed by smart objects to the IoT Hub (as described in Section 3.2.3
), the IoT Hub uses the synchronization protocol to publish the updated information at all replicas. Note that this synchronization strategy is needed only for those replicas that are part of the request/response loop: in fact, all resources are automatically synchronized with the request-issuing replica by design, since the replica also perfectly reproduces the behavior of the actual IoT Hub.
Pub/Sub approaches for data synchronization may cause inconsistencies between the state of the IoT Hub and Virtual Replicas at some moments. This may be caused by the following:
A lack of Internet connectivity with Cloud Virtual replicas: in this case there is no way to keep the states synchronized until the connection resumes;
In a Pub/Sub architecture messages may be delivered to consumers at different times so that a Client requesting the state of a resource may receive different responses depending on the Virtual Replica serving the request. However, it is import to point out that all replicas are typically being executed in Cloud Environments/Carrier-grade Data Centers which guarantee high availability and reliable and low latency connectivity, thus minimizing the inconsistency time windows.
Our work relies on an Optimistic Locking approach which solves the inconsistency issue by versioning the state of resources and leaving clients responsible for managing race conditions. Consensus Algorithms (e.g., Raft or Paxos [28
]) may introduce additional overhead whose trade offs should be carefully evaluated according to application requirements.