Towards the Internet of Augmented Things: An Open-source Framework to Interconnect IoT Devices and Augmented Reality Systems ^†

Abstract

The latest Augmented Reality (AR) and Mixed Reality (MR) systems are able to provide innovative methods for user interaction, but their full potential can only be achieved when they are able to exchange bidirectional information with the physical world that surround them, including the objects that belong to the Internet of Things (IoT). The problem is that elements like AR display devices or IoT sensors/actuators often use heterogeneous technologies that make it difficult to intercommunicate them in an easy way, thus requiring a high degree of specialization to carry out such a task. This paper presents an open-source framework that eases the integration of AR and IoT devices as well as the transfer of information among them, both in real time and in a dynamic way. The proposed framework makes use of widely used standard protocols and open-source tools like MQTT, HTTPS or Node-RED. In order to illustrate the operation of the framework, this paper presents the implementation of a practical home automation example: an AR/MR application for energy consumption monitoring that allows for using a pair of Microsoft HoloLens smart glasses to interact with smart power outlets.

1. Introduction

Internet of Things (IoT) devices have been already deployed in a relevant amount of practical solutions for different sectors and smart applications [1] and, due to their success, their number is expected to grow significantly in the next years. IoT applications, when coupled with advances in Augmented Reality (AR) and Mixed Reality (MR), have the potential to bring sensing, communication and interaction to a whole new level.

Although AR/MR research was initiated in the 60s, initial prototypes lacked certain features required for being functional on a massive scale. Nevertheless, in the past years, AR/MR capabilities have been enhanced remarkably thanks to improvements in the underlying computing technologies, electronics and connectivity. In particular, the adoption of AR in the industrial sector has increased significantly due to the reduction of its commercialization price, which lead to the concept of Industrial Augmented Reality (IAR) under the Industry 4.0 paradigm [2].

One of the biggest problems faced by the developers of AR applications that interact with real-world elements (e.g., physical things) is the wide range of technologies that need to be managed to perform even the simplest interactions. This is because the most popular AR frameworks use very different development paradigms from those used by IoT devices. In addition, the professional profile of experts working in both sectors tends to be very different.

For an AR application to interact with the physical world, both the AR and the IoT device need to be able to use a shared communication mechanism and to exchange messages using the same language in a way that they can understand each other. This is not straightforward, since some constraints may arise, like the heterogeneity and resource limitations of IoT hardware devices and the development restrictions often imposed by AR frameworks.

Recently, different researchers have addressed the compatibility issues associated with the diversity of protocols, technologies and standards that exist in the IoT field [3]. However, although AR can be appointed as an attractive, convenient and complementary interface for IoT, only a few works focused on the IoT-AR interaction and none of them proposes an IoT-AR framework designed from scratch. For instance, Jo et al. [4] proposed a method in which the preferred AR tracking method is determined by the IoT device. Moreover, the same authors published recently a thorough state-of-the-art review that emphasizes open issues to enhance the IoT-AR integration [5]: distributed and object-centric data management and visualization; access, control and interaction mechanisms with the objects; and seamless interaction and content exchange interoperability. Furthermore, the researchers developed an IoT-AR proof-of-concept prototype for enhanced shopping experiences, but only a few specific details of the IoT-AR integration framework have been provided [6].

To solve the aforementioned issues and to create a ubiquitous, scalable, flexible, interoperable, open-source and easily configurable IoT-AR framework, this article presents an open-source framework that enables the integration of AR systems and IoT devices through the use of standard communication protocols.

2. Design and Implementation of the System

2.1. Communications Architecture

Figure 1 shows the communications architecture of the proposed IoT-AR framework, which is composed of three layers:

• An AR device layer, which is composed of the different AR devices (e.g., smart glasses, smartphones, tablets).
• An IoT-AR layer. It is the most relevant layer and is based on Node-RED [7], which is responsible for interconnecting the different elements of the system. Node-RED is an integration tool based on visual programming and implemented in Node.js. It enables the manipulation of data flows between different components in a very simple way. Node-RED is in charge of translating all the protocols involved in the requests and messages exchanged by the system. In addition, Node-RED allows data flows to change dynamically very easily according to the requirements of the system. Thus, thanks to a structure of nodes and links between them, Node-RED enables modifying system parameters simply by changing connections between two nodes, without the need for adapting the implementation of any part of the system, which adds flexibility and reduces the probability of coding errors. Furthermore, Node-RED also allows for interconnecting data flows of different protocols in a simple and intuitive way without having to know the specific details of each protocol. Specifically, these advantages are used to interact with the IoT Ecosystem (with the help of the Message Queuing Telemetry Transport (MQTT) protocol [8]) and with the AR Devices (through HTTP exchanges).
• The IoT ecosystem layer includes the different IoT devices and their communications protocols. Although the framework could make use of different messaging protocols to communicate Node-RED and the IoT nodes [9], MQTT was chosen because it is widely used for heterogeneous IoT networks, it is lightweight and it makes use of persistent connections to exchange messages among nodes in a seamless way, even if they are located behind a network with Network Address Translation (NAT) or protected with a firewall that prevents incoming connections.

2.2. Implementation

The design described in the previous section was implemented and the developed systems are available along with implementation examples in an Open-Source repository on Github [10]. The main details on such an implementation are given in the next subsections.

2.2.1. AR Device Layer

Currently, the most popular AR devices are smartphones and tablets due to their processing capacity and low cost. This means that AR applications have been usually developed for Android and iOS. However, in the last years, numerous AR specific devices such as smart glasses and Head-Mounted Displays (HMDs) are emerging, so there are multiple alternatives for choosing among hardware devices, supported platforms and software development tools [11].

Microsoft HoloLens smart glasses [12] are currently the commercial device that offers the best AR and MR experience, mainly due to two factors that differentiate them from their competitors: their specific acceleration hardware and their software platform. Unlike other AR/MR devices, Microsoft HoloLens offers a software platform that consists of a Windows operating system completely adapted to AR. This software platform provides developers with tools that greatly ease their work. Nonetheless, low-level actions such as video processing or working with sockets can be difficult as it is not directly supported by the Software Development Kit (SDK), and require specific knowledge. Nevertheless, HTTP communications are provided by the SDK and can be implemented in an easy way.

The AR SDK from HoloLens is composed of different modules (the most relevant ones are depicted in Figure 2) that interact with the hardware to perform all the necessary tasks to provide a good AR experience to the user. Specifically, the Gaze Manager and Gaze Stabilizer monitor the position of the users’ head and the point they are looking at. The Spatial Mapping is in charge of maintaining and updating a 3D scanned model of the environment that surrounds the user so that the virtual elements can interact with it. Finally, when the users make a gesture with their hand, the Gesture Manager module recognizes it and triggers an action related to one of the handler modules. If an action requires communication with the real world, the Service module of the IoT-AR framework is called. Such a module generates an HTTP request that is sent asynchronously to Node-RED and, if it is a query, it waits for the answer and notifies the application by means of a callback.

The Service Module is completely asynchronous to avoid blocking the main interface thread. This was achieved by developing a set of listeners: the elements that consume data from the service API need to subscribe to them in order to receive notifications on the available data.

2.2.2. IoT-AR layer

This layer is composed of an MQTT server, the core of Node-RED and its API REST. It is in charge of routing the messages so that each IoT device is able to receive and send information from and to the appropriate AR device. When an HTTP request arrives to Node-RED from the AR application, it is decoded, and it is decided the IoT device must be forwarded by using a specific header that contains a session token. Then, the request is routed to the corresponding IoT device via a MQTT message that is published on a specific topic.

In the same way, when an IoT device sends information using MQTT, Node-RED saves such an information and updates the AR device that request it when it asks for updates.

2.2.3. IoT Ecosystem

One of the major problems of IoT devices is their limitations in terms of computational power, as well as their usual dependence on batteries. For such reasons, their communications protocols should be lightweight, which enables optimizing the use of resources (e.g., to minimize computational load and storage needs) and energy consumption, as well as to reduce the communications overhead related to the lower layers of the communications stack [13]. In addition, when IoT devices have to receive remote requests, they must be provided with a mechanism that allows them to respond to events fast without needing to expose their ports on the Internet, even when they may be behind a firewall.

Furthermore, the use of standard protocols can be helpful, since they provide scalability, interoperability and functionality to IoT heterogeneous systems. Therefore, the used lightweight protocols have to be flexible, provide support for plug-and-play mechanisms and be compatible with already existing protocols to be implemented into different types of devices (e.g., beacons, wearables, Programmable Logic Controllers (PLCs), gateways or smart garments [14]).

To comply with the above-mentioned characteristics, MQTT was selected for exchanging messages. Such a protocol is also supported by the most common open-source firmwares. Each IoT device subscribes to an MQTT topic and listens permanently to it, waiting for the arrival of new requests. Moreover, an IoT device can initiate the communication to notify status changes or alerts.

3. AR Energy Monitoring and Control: A Practical Use Case

In order to verify the proposed IoT-AR framework, a use case for energy consumption monitoring was devised, where Microsoft HoloLens smart glasses are able to interact with smart power outlets.

Specifically, the developed system is able to show the real-time information obtained from a current sensor connected to a Sonoff Pow socket [15]. In addition, users can interact directly with the power socket through the smart glasses: commands can be sent from the glasses to open or to close an electrical switch connected to the socket.

The dashboard of the developed application is shown in Figure 3. Such a dashboard can be moved by the user, who can place it in any position of the real world. Once the dashboard is placed, the user can interact with the buttons by looking at them and by carrying out predefined HoloLens hand gestures. Such gestures are interpreted by the gesture recognizer module integrated in the AR SDK and are passed to the service module of the framework that is in charge of transmitting the data over the network so that IoT devices can receive commands and perform the actions that the user requests.

4. Conclusions

AR is a powerful technology that enables user interaction in different areas, whose use is expected to grow significantly in the next years. However, its full potential can only be reached if applications are able to interact with the real world in real-time and as seamlessly for the user as possible.

This paper has presented an open-source framework that eases the implementation of communication mechanisms between IoT devices and AR/MR applications. After detailing the proposed design and implementation, the framework utility was demonstrated by developing an AR/MR application for Microsoft HoloLens that enables two-way real time interaction with an intelligent power socket.

Although the presented framework was applied to an energy monitoring solution, it can be easily adapted to a broad range of IoT devices and to other AR/MR-based end-user applications (e.g., preventive maintenance, augmented communication, context-aware applications or enhance localization, among others). As a consequence, further work will be focused on upgrading the IoT-AR framework design and performing additional experiments in terms of usability and performance in industrial scenarios.