This section presents related work in the two research fields that are closely related to the Internet of Tangible Things: The Internet of Things (IoT) and Tangible Interaction. In particular, our initial analysis will focus on the current research on Human-Computer Interaction applied to IoT (Section 3.1
) and current trends of tangible interaction that seems promising for the internet of things (Section 3.2
). In particular, through the analysis of tangible interaction literature, we identify eight tangible interaction properties that are promising for designing the interaction with the Internet of Things. We introduce then the new research field of the Internet of Tangible Things and discuss similar or related work (Section 3.3
3.1. Human-Computer Interaction with IoT
The US National Intelligence Council defined the Internet of Things (IoT) as one of the six most “Disruptive Civil Technologies” [2
]. The European Community is investing 192 million euros in several different domains of IoT research through the Horizon 2020 programme for the years 2014–2017 [14
]. Since the first definition of IoT, different technological platforms generated different related visions of what the IoT is. Atzori et al. identified three main visions: the first one is more “things-oriented” and respond to the initial trend of equipping everyday objects with Radio-Frequency Identification (RFID) tags; the second vision, defined as “Internet-oriented”, relies on smarter objects, capable of fulfilling the full Internet Protocol (IP) stack; the third vision highlights the semantic reasoning that can be performed on IoT data [2
]. The authors suggest that IoT stems from the intersection of these three visions. The Internet of Things is still a very actual research topic and many efforts are being devoted in order to standardize this broad ecosystem of interconnected objects: the Internet Architecture Board (IAB) has recently proposed a document for the definition of interconnected smart objects [15
]. Similarly, in 2012, the ITU has proposed a first Recommendation for the definition of the Internet of Things [16
A literature review of IoT shows that so far, most of the research on the Internet of Things focused on improving the hardware and software architecture that allows IoT objects to better manage power consumption, optimize data collection and sharing as well as seamlessly communicate to each other, often without human intervention [17
A search in the ACM Digital Library with the keywords (+(“human-computer interaction” HCI) +(“internet of things” iot)) produced 152 results, most of them concentrated in the last 3 years (from 2 results in 2010, to 14 results in 2013, up to 49 in 2016).
As is also pointed out by Koreshoff et al. [4
], the human is often neglected in IoT design and only recently has research been conducted on the human-IoT interaction.
It is worth noting that interaction in smart environments and with the ubiquitous computer [18
] has been already explored in depth. However, although these research fields share several challenges with the Internet of Things, the interaction with IoT introduces peculiar challenges, because of the worldwide interconnectivity between the different objects, people and virtual [19
] and physical sensors and because of their autonomous evolution according to exchanged information. In Designing Connected Products [20
], Rowland et al. discuss the main challenges for improving the consumers’ User Experience with IoT products. Rowland suggests that, because of the novelty of IoT in the consumer market, it is difficult to communicate to the user the conceptual model that shows how the device works and the related interaction model [20
]. As a result, many IoT products fail to conquer a broad market, remaining niche products for geeks or experts because of functions that do not appeal to the mainstream user or configuration processes and interaction models that are not straightforward.
In order to design IoT objects that better meet users’ needs and expectations, it is fundamental to also analyse IoT from a human-centred point of view. Koreshoff et al. readapted Atzori et al.’s IoT framework to address relevant research issues for Human-Computer Interaction (HCI) that are still open [4
]. Koreshoff et al. conducted a literature review of HCI research and a review of IoT products [17
]. The authors denote a lack of explicit involvement in HCI for IoT: most academic research focuses on specific application issues or in object interconnectivity and out of the 89 papers reviewed from the CHI conference and the Personal and Ubiquitous Journal, only 5 dealt with user inputs. For the 93 IoT commercial products reviewed, the authors show that most user interfaces are based on smartphone or web apps, or on screens integrated in the object.
Other attempts to promote user-centred design of the interaction with IoT devices can be found in Soro et al.’s article [21
], who also propose an adaptation of Atzori et al.’s framework and in Fauquex et al.’s article [22
], who propose a user-centred methodology for creating “people-aware” IoT applications.
Rowland et al. points out that IoT interaction models often suffer from their underlying technological complexity [20
]. A typical property of IoT objects is to be connected to the Internet and to have a behaviour that can depend from a dynamic network of remote connected entities. Therefore, the interaction designer for IoT should be able to cope with connectivity issues (Is the object still able to work when disconnected from the network? Will it respond to user inputs?) as well as with interaction models that depend on the autonomous communication with other connected objects (Did the object state change because of the sensed local environment or because of the object interaction with the networked objects? Is this object sending data to other objects? Which objects? Which data? For which people?).
Moreover, IoT objects are able to sense the external environment, collect sensitive information about the user and share it through the Internet. Therefore, a reliable control on sensed and shared data is particularly important for the user, who can otherwise lose the trust in the product or service. Bellotti and Sellen addressed these privacy issues with a framework for designing feedback and control in ubiquitous computing environments [23
]. In particular, the framework addresses four system behaviours concerning feedback and control: when and which information is captured, how the information is elaborated, who has access to the information and for which purposes. While Bellotti and Sellen’s framework supposes that actors that can access the information were humans, in IoT we face to a more complex scenario where also other IoT entities (e.g., cloud software or other IoT objects) can access, elaborate and further share this information.
In this article, we argue that tangible interaction can help addressing these challenges. Therefore, the next subsection reviews the perspectives of tangible interaction applied to IoT.
3.2. Tangible Interaction and Perspectives for IoT
We present here an analysis of current research in tangible interaction, conducted to individuate properties that can be useful for designing the interaction with the internet of things. We summarize the results of this analysis in the form of a narrative literature review. In the analysis, we highlight 8 tangible interaction properties (T1 to T8), which are resumed at the end of this subsection. The eight properties will be used as a framework for the meta-analysis of the papers selected in the systematic review and as themes for the design card set presented in Section 7
In Chapter 8 of Designing Connected Products, Charlier analyses how different outputs and inputs can be combined in connected products to design appropriate interfaces and interactions [20
]. Among the different interaction paradigms that can be explored for IoT, Charlier cites Tangible User Interfaces (TUIs) suggesting as advantages the usefulness for learning applications and music and, as main disadvantages, the need of several tokens that can be lost [20
]. A recent research on tangible interaction, published at CHI’2016, suggests that richer interactions, beyond token manipulation, can be designed for IoT [24
Since Fitzmaurice’s Graspable User Interfaces [25
] and Ullmer and Ishii’s TUIs [26
], Tangible Interaction has evolved much, embracing new disciplines and interaction paradigms [27
]. Hornecker and Buur [27
] individuated several advantages of tangible interaction over traditional interfaces: facilitates collaboration thanks to a shared interaction space and multiple access points (T4), exploits the spatiality and the user’s proprioception (T4) as well as the interaction through the full-body (T2), makes use of expressive representations that help cognition (T1). Hoven et al. individuate three main aspects that characterize tangible interaction: the interaction with the physical world, the exploitation of the human skills and digital computation [8
]. Although most digital interactions involve to some extent these aspects, the purpose of interacting with everyday objects exploiting human skills that are often ignored by traditional user interfaces (T2) makes tangible interaction different from other interaction paradigms and particularly interesting for interacting with IoT objects. Hoven et al. individuated particular qualities of tangible interactive systems: they generally offer direct, integrated and meaningful representation and control of the digital information (T1) [8
Since digital information is represented by a physical representation, which can be also used as a user control to manipulate such information, a particular property of Tangible User Interfaces is the persistence of the information even in case of power outage (T3). This property is particularly interesting for IoT, since IoT objects can run out of power or can be disconnected suddenly from the Internet (cf. [20
]). Moreover, the directness of the interaction and the coupling between action and perception [28
] supports immediate interactions that are generally easy to learn and understand (T5). At the same time, objects can embody meaningful concepts and can support mindful interactions that make the user step back and reflect (T7) [10
Indeed, among the different tangible interactive systems, it is possible to distinguish between two main categories: those that support immediacy and those that support mindful interactions. This distinction stems from the Heidegger’s definition of Ready-to-Hand and Present-at-Hand [29
], which has been resumed by Dourish to introduce the principles of embodied interaction [30
]. Tanenbaum et al. [31
] extend this notion and map the two Heideggerian concepts to the Bolter and Grusin’s concepts of transparent immediacy and hypermediation [32
]. Tanenbaum et al. [31
] discuss the typical Heidegger’s example of the hammer, a tool that is typically Ready-to-Hand, until it breaks down and it appears to the user attentions, becoming Present-at-Hand: the hammer can also attract our attention without necessarily break down, for example because a detail of the grip reminds the user of past memories. Therefore, the authors suggest that the objects, besides their functional role, can be categorized according to a semantic line because they can become, present-at-mind. Also Hornecker [33
] argued that tangible interaction has a hybrid nature and can support two different approaches, one that exploits affordances and user’s previous knowledge to provide intuitive and seamless interactions and another that breaks the immediacy typical of direct manipulation in order to support reflection and the understanding of the system.
Tangible Interaction is particularly interesting for supporting peripheral interactions. Bakker and Niemantsverdriet [34
] suggest that to better integrate digital interactions in our daily routines, the designers of interactive systems should exploit the full interaction-attention continuum. Indeed, the system should support both focused interactions that exploits all the users’ cognitive and motor resources and lighter interactions that happen in the users’ peripheral attention, which are more suitable when they are occupied in other tasks. Indeed, an important challenge that interaction designers have to face in ubiquitous computing is the overwhelming quantity of digital information and request for user input that is continuously presented to the user. This often disrupts the attention and concentration of the user, especially at work. Peripheral interaction deals with this problem by designing interactions that can be performed through the user’s peripheral attention, freeing user’s cognitive resources for other tasks (T6) [9
]. Bakker et al. suggests that peripheral interaction is particularly useful for all those situations where the interaction could be integrated into the daily routine, without necessarily being in the centre of the attention [9
]. The designer of peripheral interfaces should also be able to take into account the context of the interaction, as well as the personal preferences of the user [9
]. While direct manipulation in touchscreen has been proved to be particularly intuitive and easy to learn, it requires visual attention and lacks of haptic feedback; tangible manipulation, instead, can be often performed without visual attention, relying on proprioception and haptic feedback for evaluating the result of the interaction [5
The opposite approach to design valuable tangible interactions for long-lasting objects is bringing them to the centre of the user’s attention. In this case, the interaction should engage the user, stimulating reflection or emotional response [34
]. Schmitz suggests that objects with life-like, often unpredictable, behaviours can enhance the longevity of the interaction, especially if coupled with zoomorphic or anthropomorphic affordances (T8) [10
]. As in [10
], Chapman proposes a framework for emotionally durable electronic devices [35
]. The framework suggests that durable devices should have their own consciousness and should not be fully understood by the user (at least in the product exploratory phase), should make the user develop both attachment and detachment to the product and should carry signs of ageing and stories associated to the product itself. The purpose of Chapman framework is coping with the increasing waste of domestic electronic products, which get often abandoned by the user after few usages (T8). The association of stories to physical objects is a point that is often explored in tangible interaction to create attachment to a product. Hoven and Eggen [36
] proposed an extension of Ullmer and Ishii’s [26
] Tangible Interaction Framework to take into account personal objects that are associated to personal memories (T7). Indeed, personal objects facilitate the association with digital media thanks to the pre-existing mental models. Tanenbaum et al. explored the relationship between narrative and objects through the Reading Glove, a wearable device equipped with a RFID reader to support storytelling with RFID tagged objects [31
]. The ultimate challenge would be to combine both interaction paradigms (peripheral interactions and long-lasting focused interactions) in the same interface, allowing the user to switch between the two paradigms according to the context of use.
We resume the 8 properties found in our analysis of tangible interaction research in the following list:
Meaningful representations and controls of the single IoT object connectivity status and IoT object interconnections, as well as of information capture, elaboration and sharing.
Rich interactions that exploit the natural human skills, in particular exploiting haptic and peripheral interactions with IoT objects that are situated in the physical world.
Persistent physical representations that could last in case of power or connectivity outrage, allowing the user to control the state of an IoT object even when no Internet connection is available.
Spatial interactions that support collaborative setups with multiple IoT objects.
Immediacy and intuitiveness of the interaction, facilitating the understanding and control of IoT objects with minimal learning time.
Interactions with IoT objects that are integrated in the daily routines, which free users’ cognitive resources and do not disrupt attention.
Facilitated reflections on IoT object meaning and working principles, as well as support for associating and sharing memories.
Long-lasting interactions with IoT objects exploiting emotional durable designs, to cope with electronic waste due to technological obsolescence.
This analysis of the tangible interaction literature according to its two opposite and complementary approaches, which we identify in this article with the Heideggerian terms of Present-at-Hand and Ready-to-Hand suggests that tangible interaction offers many interesting possibilities for the design of new Human-IoT interfaces. The tangible interaction properties described in this section and summarized in the previous list are promising for providing an easier understanding of the IoT objects and an increased trust in the IoT system. Through the systematic literature review, we aim at evaluating which of these properties have been explored until now and which properties still deserve further investigations.
3.3. Internet of Tangible Things
Tangible interaction in IoT is still mostly unexplored. Embedding the interaction in IoT objects is not a novelty and already in 2010, Kranz et al. [37
] showed several examples in this field. The authors evidenced some challenges that are still actual nowadays, such as the risk of changing the nature of existing objects in order to add interactive capabilities, or vice-versa, the risk of hiding too much the interactive capabilities of the objects to the user eyes. Nevertheless, their analysis did not focus on the important implications of interacting with a network of objects that communicate to each other.
The Web of Things, proposed by Guinard and Trifa [38
] can be considered another close related domain. While in their first idea the Web of Things aimed at building web services for IoT objects, Mayer et al. [39
] proposed a framework that associates semantic interaction primitives to different IoT object characteristics, with the purpose of supporting both web GUI and tangible interfaces, such as physical knobs and sliders. Although this was a valuable effort for bringing back IoT interfaces to the physical world, the richness of tangible interaction was still poorly exploited and the approach was mostly technology-oriented.
The novelty of the field of research explored in this article may be evidenced by the fact that “Internet of Tangible Things” is a new term introduced by Sarah Gallacher [40
] only in January 2016 during the Second European Tangible Interaction Studio [41
], with no previous reference in scientific literature. This term is adopted in this article to promote a shift towards the design of physical interactions with IoT. Indeed, the potential advantages and the related challenges are mostly unexplored until now in current research.
As an outcome of the systematic literature review that will be presented in this article, we can highlight some theoretical work that has been conducted recently in this field. Among the most relevant papers found in the literature, it is worth citing the work of Ambe et al. [42
], who analysed three particular tangible IoT systems for connecting distant people and encourage social connection according to a selection of properties from Hornecker and Buur’s [27
] tangible interaction framework. In particular, they highlighted the importance of supporting user appropriation and personalization of IoT devices, in order to support long-term use of these devices. These properties are without doubt valuable for IoT objects that should foster social connection but might not apply to other categories of IoT objects. For example, in VoxBox [43
], a tangible questionnaire for collecting feedback during public events, appropriation was discouraged. Indeed, the purpose of the tangible questionnaire was to collect valuable feedback with intuitive but meaningful interactions related to users’ event perception. In this case, while interactions should be playful, they should not be transformed just in a game. Instead, a tangible questionnaire should stimulate reflection on the proposed questions and facilitate discussions with other people.
The tangible properties explored or proposed and the approaches for exploring them might vary consistently among different application domains and research fields. In our literature review, we individuated different kinds of papers, from those having a user-centred approach, for example for understanding the importance of personal objects for older adults [44
] and derive guidelines for IoT system implementations, to more technology-driven approaches where the design and development results were not even assessed with users [45
]. The vision might also be slightly different between researchers with a computer science background and those with a product design background. For example, Knutsen et al. [46
] through a review of commercial IoT products analyse the hybrid nature (tangible/intangible) of current IoT products, suggesting the term “Internet of Hybrid Products.” Indeed, some of the commercial IoT objects reviewed presented a mix between a tangible interface and digital services based on smartphone or web apps. Although the idea is closely related to the IoTT concept, the paper of Knutsen et al. [46
] was more focused on product and service design than on interaction design, discussing few of the aforementioned tangible interaction properties. For this reason, this paper was not included in the meta-analysis.
We found in literature interesting attempt to facilitate the interaction with IoT objects through tangible interfaces. Van der Vlist et al. [47
] presented a system that allowed to create connections between IoT objects in the smart home through simple manipulations of tangible tokens that represent the IoT objects and of a central tile that represents the network hub. Domaszewicz et al. [48
] presented a similar system where “an application pill” object allows interconnecting different IoT objects and determining whether an IoT function can be obtained combining the IoT objects that were in the range of the application pill. Ideally, different pills can be used to assess different functions. Those papers were considered as borderline since IoT objects were not augmented with tangible interfaces; instead, an additional tangible object was introduced to manipulate IoT object connections. Although this a valuable first attempt towards tangible interaction with IoT we did not include the papers in the analysis because, according to our eligibility criteria (cf. Section 4
), the physical interaction was not embodied in the IoT object.
Besides all the papers presented in Section 5
, to the best of our knowledge, the only other relevant work that was not included in the literature review, is Physikit [26
], a toolkit for the physical visualization of ambient data collected by an IoT sensor platform. Unfortunately, this article was not detected through our queries in the digital libraries (cf. Section 6.3