A Design Space for Virtuality-Introduced Internet of Things

Abstract

Augmented reality (AR) and virtual reality (VR) technologies have been dramatically expanded in recent years. In the near future, we expect that diverse digital services that employ Internet of Things (IoT) technologies enhanced with AR and VR will become more popular. Advanced information technologies will enable the physical world to be fused with the virtual world. These digital services will be advanced via virtuality, which means that things that do not physically exist make people believe in their existence. We propose a design space for digital services that are enhanced via virtuality based on insights extracted from three case studies that we have developed and from discussions in focus groups that analyze how existing commercial IoT products proposed in a commercial crowdfunding platform, Kickstarter, could be enhanced through virtuality. The derived design space offers three dimensions to design a digital service to fuse IoT technologies with virtuality: (1) Taxonomy of IoT; (2) Visualizing Level, and (3) Virtuality Level. The design space will help IoT-based digital service designers to develop advanced future IoT products that incorporate virtuality.

1. Introduction

Augmented reality (AR) and virtual reality (VR) technology have been advanced in the real world by introducing virtuality, which makes people believe in the existence of things that do not physically exist. VR technology has enabled people to enter and interact with immersive virtual worlds. This technology extends not only to gaming experiences but also various real-life services, such as rehabilitation, education systems for children with autism spectrum disorder and self-confidence services [1,2,3]. In addition, some digital services utilize mobile VR [4]. AR technologies attach information to the real world, and have been studied for a long time; for example, technology-enabled museum guidance displays digital objects in the real world [5], and book-reading experiences are augmented by showing virtual objects in a book [6]. PTC, a company that provides an AR development platform named Vuforia, developed a new platform named ThingWorx Studio, which combines IoT technologies with AR technologies [7]. For example, people can develop their own services to visualize data sensed by IoT devices using this platform.

However, many of the services that are proposed do not seamlessly integrate the real world with the virtual world. In the area of health care research, particularly, it has recently become popular to use VR and AR technologies to develop advanced health services [8,9,10]. The virtual world is considered to divide the two worlds, thus these technologies are usually considered independently. We believe that it is necessary to seamlessly fuse virtual-world and real-world objects with AR and VR technologies in the real world. In this paper, we named the fusion world beyond the boundary of the two worlds “Fusion Reality”. In the near future, VR technologies can be used to create immersive and more realistic virtual objects that can be integrated into the real world through AR technologies, although the two technologies are currently separated. However, Fusion Reality is realized through the true combination of the two technologies.

The following paragraph shows a typical example scenario realized by Fusion Reality:

Naomi wakes up in the morning as usual, then she unmounts her smart glass from a charging spot and wears it. The virtual agent character “Riki” begins talking to her. He looks like a strange monster whose size is small enough to be held by her. The character is not so unusual after wearing a glasses-type display on a daily basis. She is notified of the schedule and the weather today by him, and she is gradually waking up. He tells her, “Today is not hot, but it seems to be rainy; it will last until the evening, so you should bring your umbrella.” She also tells him that she recently had lost her umbrella. He offers to lend his umbrella delightedly. He may feel happy because he can help her. He has an umbrella in the virtual world, and there is a physical umbrella that is the same design as Riki. In addition to the umbrella, there are several of his own things in her house. One example is a ball toy that Naomi does not completely understand. He occasionally plays with it deftly, for example juggling with it. Looking at his playing, Naomi is healed for a while, so she has not regretted buying it for him. She also practices juggling a little, but she stops because she cannot juggle better. Recalling such a thing, she takes breakfast while watching the news displayed on a wall. After breakfast, she dresses in her favorite suit and goes to her office with Riki's umbrella.

In the world that realizes Fusion Reality by coexisting objects and contexts of the physical world and the virtual world, virtuality is either explicitly or implicitly introduced into people’s daily life, and information presentation and notification can be performed through the virtuality. Furthermore, it is possible not only to notify information but also to facilitate communication between people and make the game experience more realistic. In the above scenario, daily life is expanded by a virtual character that does not exist in the real world. By breaking the boundary between the real world and the virtual world, these two worlds are seamlessly connected, and the virtual world can become more real to increase the influence of the virtual world on people. By making things in the real world appear in the virtual world, people can feel virtual-world neighbors in the real world. In the above scenario, the real world and the virtual world share their contexts; thus, the virtual character’s umbrella can be employed by Naomi in the real world, and the toys bought for him can be used in the real world.

To realize Fusion Reality, we aim to introduce virtuality into Internet of Things (IoT) technologies. Currently, most typical IoT products enable data to be acquired and real objects to be manipulated by connecting devices to the Internet. By combining IoT technologies and virtualization technologies such as AR and VR, we expect that Fusion Reality can be achieved. Pokric et al. developed a service to improve urban air quality by combining AR technologies and IoT technologies in the context of a smart city [11]. They developed an AR Genie Platform to easily develop AR services and an ekoNET system, which is an environmental monitoring service. They also developed a serious game to combine AR Genie Platform and ekoNET to increase the awareness of air pollution issues. Atsali et al. described a methodology for the implementation of a service that combines IoT technologies and mixed-reality technologies [12]. The implementation method of the service, which incorporates IoT data into 3D mixed reality, is described. These two projects described some examples that fuse virtuality in the real world, and how to develop them. Although the implementation of these services has become technically feasible, the design space for these services has not been well investigated. For this reason, this paper focuses on providing a design space for the services and products that incorporate virtuality into IoT technologies. This design space provides a basis that enables designers to design virtualized services in better way. People can design services that confer virtuality that is suitable for their purpose based on the design space.

This paper is organized as follows: Section 2 describes the development of existing IoT products and virtualization technologies as a background. In Section 3, we introduce IoT products that are currently proposed on Kickstarter, which is one of the most popular crowdfunding platforms, and three case studies to introduce virtuality. In Section 4, we derive the design space by analyzing the IoT products and case studies shown in Section 3, and discuss the results of focus groups that we conducted to evaluate the effectiveness of the proposed design space. Finally, we discuss future directions in Section 5 and summarize this paper in Section 6.

2. Background

IoT technologies currently have attempted to transform industry in different countries based on that country’s national policy. For example, the German Federal Ministry for Economic Affairs and Energy has established a vision named Industry 4.0, which is an attempt to update the industry to a new stage by applying information technology [13]. The country promotes extensive use of IoT technologies because Industry 4.0 will be able to boost the productivity of companies by 30%, and Germany’s gross value can be boosted cumulatively by 267 billion euros by 2025 [14]. General Electronic (GE) has a similar concept named Industrial Internet, which aims to blend industrial equipment with information technologies. The use of IoT technologies for these industries has generated high expectations, and IoT technologies will gain worldwide attention since the proposal has been rendered by large organizations such as GE and countries such as Germany.

IoT technologies are also predicted to have a substantial impact on agriculture. For example, Konica Minolta and Yammar Heli & Agri made an innovation in Japanese agriculture via IoT technologies [15]. Japanese agricultural workers encounter the emerging problem of aging and a lack of successors; thus, the efficiency of agricultural work is an important issue. Current rice farmers measure leaf colors one leaf at a time. Since this approach is not realistic for all fields, they check 10 to 20 plants for each field and assess the quality of soil by the average value of them. Despite the fact that a mixture of adequate and inadequate nutrient places exists in one field, the farmers uniformly supply fertilizer because they do not know the growing condition of each field. To solve this problem, Yammar Heli focused on the fact that unmanned helicopters are employed by 42% of Japanese farmers, and proposed a new business based on the idea of sensing a leaf color using Konica Minolta’s infrared camera mounted on the unmanned helicopter. However, the rice collapsed and could not be sensed because the unmanned helicopter caused strong winds. Instead, they decided to use a drone to sense the rice, and the data were collected from the camera mounted on the drone. The system can create a leaf color map, and then the fertilizer is supplied from the unmanned helicopter based on the leaf color map. As a result, a dynamic fertilizer supplement was achieved. When the dynamic fertilizer supplement was applied based on the sensing in the field, it caused a significant effect. Because the soil was properly improved, the yield per 10 square meters increased from 7 bales to 11 bales when given the same amount of fertilizer. The harvest of the rice brand “Haenuki” also increased from 6 bales to 10 bales. A significant effect was observed with respect to the protein content, which is an indicator of the quality. Many efforts in agriculture have utilized IoT technologies [16,17], and IoT technologies are now extensively employed in agricultural research [18]. Agriculture is appropriate for using typical IoT technologies that provide a sensor network because environmental monitoring is important for efficient crop production.

IoT technologies are expected to be applied not only in industrial and agricultural sectors but also in our daily lives. The smart home, which contains digitized home appliances that are connected to a network, has been discussed from the first half of 2000 [19] to today [20]. On the business side, the popularity of the smart speaker Amazon Echo is increasing. After Amazon.com released Amazon Echo in 2014, Google Home by Google and HomePod by Apple were subsequently released. Users of these products can access weather and web information by talking to the speaker and the products themselves can connect and operate with other smart appliances. You can ask questions to the products such as “Today’s weather?” or give commands to them such as “Please turn off lights”. When IoT technologies achieve penetration and smart home appliances become the mainstream, these voice-based operations will become popular without using remote controllers every time. In the smart home where wireless sensor networks are constructed, security problems are also common currently; many efforts have been investigated to solve them [21,22].

Substantial advancements in AR technologies have been achieved in recent years. Pokémon Go, which is a game for smartphones that appeared in 2016, has achieved 750 million downloads in the worldwide in June 2017 [23]. Pokémon Go increased user activity more than 25% via game playing [24]. This game utilized a hand-held AR technology; users can feel Pokémon creatures to be lived in the real world. AR technologies that are recognized by this game are predicted to become more popular, and services and games that utilize AR technologies will continue to be expanded in the future. In addition to a hand-held AR technology, see-through head-mounted displays (HMDs) such as Microsoft HoloLens have also been released. The see-through HMD has been investigated since the early 1990s [25,26]. A head-up display (HUD), which projects information on a glass in front of eyes, has also been investigated [27]. Car navigation applications that utilize HUD technologies are being developed by Konica Minolta [28]. Likewise, VR technologies have also been investigated since the early 1990s [29], and commercial products such as Oculus Rift, HTC Vive, and PlayStation VR have been released [30,31,32]. Because these devices have become popular, services that introduce virtuality are increasing. For example, the application named “cluster” enables people to gather and perform activities in a virtual space [33]. Since this application is a platform that can be operated with thousands of users, the users can hold conferences, meetings, concerts and large-scale events in the virtual space. They can virtually meet, talk, enjoy the same contents and communicate with other persons together without physically meeting them. The communication with the virtual objects on a daily basis via the HMD to access the virtual world may be a realistic future concept.

We predict that the IoT technologies introducing virtuality will be advanced, and the trend will continue to advance the direction in the near future. However, these two types of technologies, AR and VR, do not have a close relationship. To achieve Fusion Reality, the real world and the virtual world must become more seamlessly integrated. For this purpose, we need to clarify the technologies to introduce virtually, and a methodology to design the virtual world would be necessary. We predict that the combination of IoT technologies and virtuality technologies will be effective; however, a well-defined design methodology has not been established for designing these services. The paper clarifies the methodology through the analysis of the survey of existing IoT products, the case studies introducing virtuality, and focus groups discussing the IoT products and the case studies.

3. Analysis of Current IoT Products and Services that Introduce Virtuality

3.1. Current IoT Product Development Landscape

Currently, the IoT technologies are beginning to penetrate not only research areas but also business areas. To analyze the current situation of IoT products, we analyzed several IoT products proposed on Kickstarter, which is a popular crowdfunding platform [34]. Kickstarter is also employed as a fundraising platform for diverse hardware developments. By raising funds with Kickstarter, various proposed products have actually been developed, including various IoT products. Thus, we can extract insights about the current trend of IoT technologies to analyze the IoT products in Kickstarter. The survey was conducted on 25 July 2017, and we obtained 142 results from the survey; since four products were not really invested, we finally analyzed 138 products, excluding these four products.

The product that was most funded is “Brixo” [35], where Brixo resembles Lego blocks; it is powered and offers functions as an electronic circuit. Because it can be assembled like Lego, users can easily create original things without programming skills. Brixo consists of connector blocks, trigger blocks, and action blocks. By connecting these blocks like Lego blocks, a light block can be turned on and a motor block can be activated when a signal from a trigger block is received. The creators of Brixo indicate that children can learn electronic circuits by playing with Brixo. A similar product that is considered to be one of the top ten funded products is “MODI” [36]. This product provides functions by connecting various modules with magnets. The difference between Brixo and MODI is that MODI provides IDE-named MODI Studio, which supports both GUI coding and CUI coding. Since it can be controlled by the program that the user develops, MODI Studio enables more advanced customization than Brixo. As a result, MODI seems to be able to create more practical products than Brixo.

The analysis of other products indicates that many products can receive notifications from IoT devices, and are operated by the user, who connects sensor devices to a smartphone application. The fourth most funded product is “WioLink”, which can be attached via sensors and actuators named Grove to a base board and operated by a smartphone [37]. This provides a RESTful API, and the information from the sensors can be checked through the mobile application. As a result of this survey, we divided the analyzed IoT products into three categories: (1) Customizable IoT; (2) Static IoT and (3) Software platform-supporting IoT.

Customizable IoT enables users to customize functions and modules. Brixo and WioLink are classified as Customizable IoT. Another product in this category is “Onion Omega 2” [38]. Onion Omega 2 is a very small Linux machine that enables users to develop python, ruby, C ++, Node JS and PHP applications. Many customizable IoT products are similar to Arduino and Raspberry Pi, which are board-shaped products.

Static IoT is a product that provides specific functions to users who utilize their static functions provided by IoT devices. One example is “miaLinkup” [39]. MiaLinkup is a device that can provide various functions to a car by connecting it to the on-board diagnostics (OBD) port that is installed on the car. MiaLinkup enables door lock and unlock, trunk unlock, remote horn, lights flashing and remote engine starting functions by operating a user’s smartphone. Since a GPS function is installed, the position of the car can also be tracked. Therefore, a route can be subsequently confirmed. MiaLinkup can also sense the fuel and battery level of the car, which informs the user about the amount of fuel and battery through the smartphone application. Since it provides an API named miaAPI, applications can be developed by third parties. Other products for cars also exist. “Raksha SafeDrive” provides specialized services for accidents, including impact detection and emergency services [40,41]. Raksha SafeDrive automatically notifies an accident when one occurs to friends and family, and provides breakdown support and driving assist functions. Since this device is equipped with GPS, it can grasp a route. Another product that is not related to cars is “Autonomous Cultivation Controller” [42]. A user can monitor plant growth data, such as temperature, lux, and pH levels, from a smartphone application. In addition, the user can measure the water level and control pH level from the application.

The number of Customizable IoT projects in our survey was 64, the number of Static IoT was 61, and the number of software platform-supporting IoT was 13. We do not discuss the software platform-supporting IoT because it is not an IoT product but a supporting tool.

Current IoT products provide devices and software to operate sensors and actuators. Although these products contain numerous devices that sense the real world environment, and act on the real world, few products introduce virtuality in the products. In the top 10 products, “Tittle X: The World’s First Ultra Compact IoT Golf Simulator” was a device that introduces virtuality [43]. This product provides devices and applications that enable a user to virtually practice golf. The user can check how a ball flies and the angle of swing by watching the display in the simulation. This product enables access to the virtual world by utilizing the devices and a display. However, the simulation using the display is suggested to have a low immersion feeling [44]. The introduction of AR and VR technologies, such as HMDs, is effective for enhancing the virtuality of these products.

3.2. Case Studies of Digital Services Introducing Virtuality

In this section, we present three case studies of digital services that introduce virtuality and have been developed in our project, then we analyze them to derive the dimensions in the design space for designing IoT products introducing virtuality. The three case studies are Virtual Aquarium [45], HoloMoL [46] and Ambient Bot [47]. These case studies change a user’s behavior, realize human memory augmentation, and provide useful daily information by showing virtual objects to the user in the real world. Since the purpose and method of these case studies are diverse, by analyzing them we can extract common features from them that exploit the design space for designing virtuality.

3.2.1. Virtual Aquarium

Virtual Aquarium was developed to improve user’s dental hygiene; it is set up near a mirror in a washstand [45]. In the aquarium, the behavior of fishes is reflected by the user’s tooth brushing activity, and the fishes are visualized on the display, as shown in Figure 1. The activity is detected by Cookie, which is a wireless sensor device that can sense three-axis acceleration [48]. Each toothbrush that is attached Cookie enables the system to recognize the activity. In addition, an event notification is managed by a context acquisition framework named Prottoy [49]. Tooth brushing patterns are recognized by analyzing the sensed data.

The tooth brushing patterns are reflected in the virtual aquarium. In this prototype, the ideal patterns are as follows: (1) people should brush their teeth at least twice per day; (2) one session should involve at least three minutes of brushing; and (3) brushing should involve the patterns that ensure that their teeth are properly cleaned. The system provides two types of feedback for these ideal patterns: immediate feedback and accumulated feedback. When the user starts tooth brushing, the walls of the virtual aquarium begin to be cleaned by a scrub and the fishes start dancing in a playful manner. This action is termed immediate feedback. When the brushing time is sufficient, the movement of the scrub and the fishes’ dance become more elegant. After brushing is finished, the fishes stop dancing and start normal movement. The movement of the scrub and fishes is designed to give the user some suggestions regarding to a better method of tooth brushing. The fishes’ health is affected by the cleanliness of the aquarium. If the user does not sufficiently brush his/her teeth, the fishes may become ill or die. However, continuous brushing produces eggs laid by the fishes in the aquarium. The incubation ratio of the eggs is not very high. If the user continues to brush at a constant rate, the user can improve the incubation ratio. This type of feedback is termed accumulated feedback. This approach improves long-term user behavior.

An experiment that evaluates the effectiveness of improving the users’ tooth brushing patterns was conducted. It consisted of the following three phases:

• The households are asked to continue their normal activities while daily tooth brushing patterns are recorded by the toothbrush attached the Cookie. Once the patterns have become stable over time, the study progresses to the next phase.
• Virtual Aquarium’s feedback components are introduced to the households’ bathrooms while the tooth brushing patterns continue to be recorded. Once the patterns have stabilized to a pre-determined degree over time, the study progresses to the final phase.
• The feedback components are removed while tooth brushing patterns continue to be recorded. Once the patterns have stabilized to a pre-determined degree over time, the data collection is concluded. Brushing records are analyzed to determine the effect of the feedback on the patterns.

An 8- to 12-day pilot study was conducted. The study participants consisted of three Japanese male participants and four Japanese female participants from three families. The experiment indicated that every participant’s brushing time was less than three minutes at the beginning but increased to more than three minutes. When the feedback system was removed, the brushing time decreased; however, the brushing time of the majority of participants remained long.

3.2.2. HoloMoL

HoloMoL is a Microsoft HoloLens application for augmenting human memorization that applies the Method of Loci (MoL) technique [46]. The MoL is a memorization method that relates to the space that a person knows well with the information he/she wants to remember [50]. The place is a palace created in an image. This method improves an ability of human memorization, for example, a person can remember the figure of π, which is the circle ratio, by using the method. This technique enables effective memorization by combining spatial memory and associative memory. When using MoL, people need to be able to accurately imagine a palace and know the detail of the palace. For example, the palace must be able to accurately represent the type of the layout, room, furniture, and lighting, and how things are placed. It is an unaffordable method because training is necessary to be able to imagine the imaginary space accurately. HoloMoL aims to use MoL without training by applying AR technologies provided by HoloLens to make MoL available to the users in the real world. As a result, many people can remember more information.

HoloMoL realizes MoL in the real space by displaying information that corresponds to a gazed object with AR technologies. Because HoloMoL can be used in the space where the users live on a daily basis, the system satisfies MoL’s “well-known place” condition. By enabling the users to place information that they want to remember, they can remember the location and the information in the same manner as the original MoL. In the prototype, AR markers were installed in the real world, and the information that the users want to remember was displayed when the users saw the markers. This prototype was developed using Unity and Vuforia. Unity is utilized as a platform to render a 3D object in the real space and Vuforia is adopted to recognize fiducial markers and to provide a database function that associates the image of each marker to be recognized with the corresponding information to be displayed in the real space. When the marker is recognized, the database is referenced and the corresponding information is displayed as a sequence of characters written on a plate-shaped object, as shown in Figure 2.

We conducted an experiment to investigate how people would utilize HoloMoL to associate information with places for memorizing the information. We recruited six participants: 20- to 23-year-old Japanese college students (4 males and 2 females with an average age of 21.7). The task of the experiment is to memorize and recall ten African nations, for example, Botswana. We chose nations that are unfamiliar to many young college students. After the memorization phase, participants were asked several questions that required recalling the descriptions. After the task, each participant was interviewed and took a short questionnaire.

We analyzed the observed participants’ behaviors and gathered interview results to understand participants’ fiducial marker placements to accomplish the target task. The results indicated three different methods to place information:

• Association-based method: People who used this method associated an environment’s physical objects with the intended information to memorize. For example, when a participant had to memorize the description about Niger, she imagined water from Niger because the description suggests that Niger had an image of water; thus, she placed a corresponding marker near water bottles. Another participant said that she chose a place related to water because the marker’s texture looks like water.
• Category-based method: The participants categorized the information to be memorized and then placed the similarly categorized information spatially close to each other. For instance, the participants placed information about country names to be memorized in one location and placed population and demographic information in another location and then started memorizing the information.
• Sequence-based method: The participants memorized the information by arranging the information based on personally created rules. For instance, a participant sorted information, via their corresponding fiducial markers, in alphabetical order and then memorized all information.

These results suggested that multiple methods existed to put information in the real spaces. The most interesting insight extracted from the study is that HoloMoL can unintentionally be applied with respect to the designers’ expectations. As shown in [51], the unintentional affordances given by daily products suggest new methods for their use. This “thoughtless act” is important when designing services based on new technologies because recent IT technologies require complex and abstract affordances to people; thus, they sometimes develop the interpretation of the affordances.

3.2.3. Ambient Bot

Currently, our cognitive overload is increased with information technologies. For example, a large amount of information is presented to us in public displays located in trains and stations. Modern social media attempts to capture our available attention by ubiquitous computing technologies [52]. For a more comfortable and peaceful daily life, information should be ambiently delivered to us only when the information is wanted. In addition, the information should be available to us without extra time and effort.

Ambient Bot is a notification service that ambiently provides daily information to users based on the assumption that the users wear a lightweight HMD [47]. The user can see a virtual creature visualized with VR technologies and render the creature in the real world with AR technologies. Figure 3 shows the basic concept of Ambient Bot. The creature only imparts daily information based on the user’s interests when he/she gazes at the creature; however, the creature leaves the user’s scope when he/she does not want to receive any information from the creature. Therefore, the creature can ambiently exist in the user’s view without disturbing his/her current activities when he/she does not focus on the creature. In addition, Ambient Bot provides the information in a social manner because the user develops an intimate relationship with the creature [53]; thus, the interaction with the creature will enable a richer daily life [54].

In the current character design in Ambient Bot, we chose a jellyfish character that floats and moves in a user’s surroundings, because a floating jellyfish is not a strange image in our daily space, especially for young Japanese individuals who consider a jellyfish to be a common agent in popular Japanese animations [55]. On the other hand, if we chose a pet animal as a virtual creature, the user may consider the pet floating in the daily space to be unnatural. In addition, the user may sometimes be too empathetic with the pet animal, which is undesirable for the aim of Ambient Bot because it may increase the user’s cognitive overload. Similarly, a human is not appropriate for Ambient Bot as a virtual agent because a human may not be neutral, which can increase his/her cognitive overload.

As shown in Figure 4, a prototype system is constructed with Unity and shows the virtual character using Oculus Rift. The character conveys the information when the character is at the center of the user’s view and stops speaking when the character is out of the center of the view. In this system, the situation in which the character is at the center of the user’s view is defined as the character making eye contact with the user. Therefore, the character speaks while the user makes eye contact with the character.

The cases in which Ambient Bot was employed were extracted by a preliminary user study using the prototype. In the user study, seven participants are interviewed about the use cases.

Table 1. Use case Analysis [47].

Categories	Use Cases
When users like to use Ambient Bot	Users like to receive news or weather forecasts when they wake up in the morning.
	Users like to receive content when they have free time or take a rest and have nothing to do.
	Users like to receive content when they are walking.
	Users receives potentially necessary information when he/she is not conscious of it.
What type of content users like to receive from Ambient Bot	Users like to receive social media information from the jellyfish.
	Users want to hear content from a lower-pitched human voice.
	Users like to receive news content from a human-like character.
	Users want Ambient Bot to aid in conversation with other people.
How users like to use Ambient Bot	Users want to receive various content without wearing and operating any physical devices.
	Users only like to receive brief message content.
	Users want to see text messages in a clearer manner. In some backgrounds, the clarity of the messages is hindered by their colors and positions.
	Users want to choose the type of content that they receive according to their current situations.
	Users expect to speak with a jellyfish.
	Users expect that a jellyfish behaves with more jellyfish-like motions.

lists the results. The result indicates that the extracted use cases are classified into three categories. The first category is whether the participants like using Ambient Bot. The second category includes the type of content that users like to receive from Ambient Bot. The third category entails how users want to use Ambient Bot.

4. Design Space for Virtuality-Introduced IoT Products

4.1. Preliminary Definition of the Design Space

In this section, we present a design space to introduce the virtuality that is extracted from the survey of IoT products and the case studies. The taxonomy of IoT products was divided into Static IoT and Customizable IoT based on the analysis as described in the previous section.

We define the classification of virtuality according to the physical constraints used in the real world. Virtual Aquarium persuades users to brush their teeth by expressing the virtual world using a physical display. Although it enables the users to change their behavior, there is a physical limitation. For example, virtuality can be seen by using display devices installed in the physical environment. If no physical display exists in the place, people cannot experience the virtual world offered by the virtuality. Since a physical display requires a place to be put and some physical things and might not be typically portable, it can only be used in a fixed place. For example, the user cannot see the Virtual Aquarium while he/she is outside of their home. AR technologies that use projectors encounter the same problem.

In HoloMoL, a user can see the virtual objects anywhere while an HMD is attached. Therefore, HoloMoL does not have the same physical constraint as Virtual Aquarium. However, HoloMoL has another physical constraint, which is derived from the approach that enables it to visually expand physical objects. This case study enables the information that the user wants to memorize to be put into a real space. Extending the physical objects with virtuality shown on the HMD is a fundamental approach of this case study. Therefore, the user needs to access the place that allows the placement of augmented physical objects in the real world. This approach is powerful but is sometimes inadequate for introducing more abstract virtuality, such as a virtual characters.

Ambient Bot does not have these constraints. Since the virtual character is floating without being subject to physical constraints, Ambient Bot is the case study that enables the strongest virtuality in these case studies. The constraints of Ambient Bot are not physical, but are restricted by the reality of the introduced virtuality. If virtuality is not believable from the users, the service may be meaningless for them.

We extracted a design space for developing digital services that introduce virtuality, named Visualizing Level, which identifies how to incorporate virtuality into IoT products. Details of this design space are as follows:

• In this level, an IoT product displays information using a physical device. For example, Virtual Aquarium used a physical display. Since Visualizing Level 0 service employs a fixed display, the display needs to be deployed in places where the context introduced by virtuality is easily understood. For example, Level 0 is good at providing services for spatially limited activities, such as restrooms and bathrooms. In another example, another IoT product named Augmented Go provides functions to support the game of Go with a projector, and it displays the exercises of past games [56]. Some attempts to extend the real world using displays and projectors have also been studied [57].
• In this level, an IoT product visually extends physical objects, and virtual objects can be arranged on the physical objects, which enables the virtual objects to be seamlessly introduced in the real world. We suggest that these services will be seen more often when it becomes commonplace to wear an HMD device, such as HoloLens, on a daily basis. Compared with Visualizing Level 0, if a wearable HMD, such as Google glass, is more likely to be used, it can improve mobility to exploit virtuality. Virtual objects can directly introduce virtuality; thus, the immersiveness can be enhanced.
• In this level, an IoT product displays virtual objects that are free from the physical constraints of the real world. For example, Ambient Bot can display the virtual character without the physical constraints of the real world. Therefore, the real space can be virtually enhanced through virtual objects seamlessly located in the real world. Eliminating the unnaturalness of the virtual objects that appear in the real world is a challenge to be overcome at this level. Ambient Bot approached this problem from the aspect of character design. If creators do not adequately design the reality of the introduced virtuality, users will not be able to believe the introduced virtuality, then the service may become meaningless.

To design the service based on IoT technologies that introduce virtuality, we derived a design space based on the discussion of the taxonomy of IoT products and Visualizing Level. Figure 5 shows the derived design space. The vertical axis represents the taxonomy of IoT products, and the horizontal axis represents Visualizing Level.

4.2. A Preliminary Focus Group-Based Analysis

Based on the derived design space, we conducted a focus group-based analysis to gather the ideas of IoT products that introduced virtuality in August 2017. The participants include 10 Japanese students (8 males and 2 females with an average age of 22.3 years old). Currently, all participants were selected from one country, but it may be possible to extract more complete design space if the focus group includes more diverse participants. In this focus group-based analysis, participants discussed current IoT products that are proposed on Kickstarter described in Section 3. We divided the participants into three focus groups; Group A, Group B and Group C, and the number of participants in each group ranged from 3 to 4.

The procedure of the analysis is described as follows:

• We gave the participants a detailed explanation of the taxonomy of IoT products and the concept of Visualizing Level for approximately 30 min.
• We explained the details of the focus group for approximately 10 min. The participants discussed two IoT products proposed in Kickstarter: Static IoT and Customizable IoT. The goal of this focus group was to propose IoT products by discussing the Visualizing Level of each taxonomy of IoT products. Therefore, the participants were encouraged to propose at least six ideas.
• The participants started their discussion. The participants could ask questions about Visualizing Level and the taxonomy of IoT products.
• When the discussion of three Visualizing Levels of the two types of IoT products is completed, the focus group was terminated.

Table 2. Ideas derived from the focus groups.

		Visualizing Level 0	Visualizing Level 1	Visualizing Level 2
Group A	Static IoT (miaLinkup)	After the user performs eco-driving, he can check from the smartphone app that the island in the virtual world becomes beautiful	By using HUD or HMD, signs and pedestrians hidden in the building are sensed and displayed.	Navigation characters guide the user on HUD.
	Customizable IoT (Brixo)	Providing a game that combines given modules to complete the target circuit and using it to study circuits.	Showing electronic current/ Creating a game stage with the blocks that is inspired by Mario Maker.	Objects created with blocks can be used in the virtual world as avatars.
Group B	Static IoT (miaLinkup)	Using HUD to replace conventional navigation/ Showing how much you are driving on the front passenger side.	Detecting the flow of the surrounding car and telling it to the driver/ Showing the weather of the destination by overlaying his view/ The window of the rear seat displays the beautiful scenery	Displaying movies outside the rear seat window/ Displaying books and movies to non-drivers who do not experience car sickness
	Customizable IoT (Brixo)	Writing a line in the figure displayed on the display and reflecting the result in the physical world/ Providing a puzzle game/ Using block as UI	Displaying a completed circuit as a hint	None. It is hard to tie this product with Level 2 because it is bound to the physical world if it uses physical blocks.
Group C	Static IoT (Raksha SafeDrive)	An agent in the screen calls out when an accident happens and uses a touch screen to confirm the consciousness (to determine whether consciousness is interrupted)/ Displaying the accident map on a HUD.	Displaying vehicles hidden behind the scenes through HUD/ Overlaying the colors so that they can be recognized as young drivers and high accident rate in the car/ Showing road hazards to encourage drivers to slow down.	If the driver speeds, encouraging careful driving by inserting a virtual high-end Mercedes-Benz car in front of him/ Displaying a guide in front/ Displaying a virtual character that will speak from the front passenger’s side to encourage relaxation.
	Customizable IoT (MODI)	In order to prevent losing the moules, showing where the module is in the room in the application.	As people can see where the IoT device was installed, highlighting the location of the IoT device/ Making the electronic currents visible for debugging	A virtual person demonstrates how to use the block for the introduction of MODI/ Simulating by combining virtual modules. If the user can create something good, then he can buy the modules like e-shop.

lists the summarized results. The service can be designed based on this design space via the focus group, where virtuality was introduced to the current IoT product. As a result of considering virtuality from multiple aspects based on Visualizing Level, the participants were able to assign the IoT products to various levels of virtuality.

In this focus group-based analysis, a deeper knowledge about the design of virtuality was obtained. In the design space, virtuality is classified by focusing on the visualization method but we found that virtuality may have another axis from the discussions in the focus group. We noticed that this current design space is insufficient for designing virtuality better; thus, we introduced another axis named Virtuality Level. Two classifications were suggested in the focus group: (1) the visualization of information and (2) the introduction of the virtual context. The visualization of information is an IoT product that visualizes the information that a user wants to know, for example, a user can watch an amount of electronic currents proposed by Group B in the discussion about the combination of Static IoT and Visualizing Level 1. The product of this type enables a user to understand information by visually displaying it. The introduction of the virtual contexts converts the meaning of the original information to a better meaning through the virtual context to make virtual objects believable. In the focus group, the idea can extend an IoT product by introducing the virtual context such as gamification, which introduces game mechanisms into non-game things. For Brixo, both Group A and Group B had the idea that the educational effect of a game can be enhanced by incorporating the virtuality of the video game. Similar to a game, designers can also improve motivation by providing a virtual context that differs from the real world.

4.3. Design Space for Virtuality-Introduced IoT Products

As a result of the discussion described in the previous section, we decided to add another axis named “Virtuality Level” to the design space, as shown in Figure 6. Virtuality Level 0 involves the visualization of information using virtual display technology, and Virtuality Level 1 provides the virtual context that is valuable for users although it is not related to the context of the real world, such as a game and a virtual character. The distinction between the two Virtuality Levels is that an application to use Virtuality Level 0 presents simple information such as texts, and information is visually layered on the real world. On the other hand, an application of Virtuality Level 1 presents information seamlessly fused in the real world, and there are no explicit layers between the information and the real world. For example, at Virtuality Level 0, data obtained from a sensor are directly shown on a display device or projected on a surface in the real world. At Virtuality Level 1, the data are shown as a virtual object seamlessly located in the real world. For developing applications at Virtuality Level 1, AR and VR technologies need to render virtual objects that are believable when located in the real world. VR technologies enable us to create believable virtual objects, and AR technologies seamlessly render the objects in the real world in a believable way.

In this section, we analyzed the case studies and the results of the focus group again to extract the virtual contexts used for designing applications at Virtuality Level 1. Then, we show the results of another focus group to validate the revised design space. In the case study, the examples of Virtuality Level 1 are Virtual Aquarium and Ambient Bot. The virtuality types that these two case studies offer differ. Virtual Aquarium shows a fish tank to a user as a virtual world, and the world is changed according to the user’s current activity. In Virtual Aquarium, the logic is that the aquarium becomes clean when the user brushes are embedded in the virtual world. By showing the worldview of the virtual world to the user, the user’s actions can be prompted. Likewise, Group A proposed an island that becomes cleaner; this type virtually provides the same worldview.

In Ambient Bot, virtuality is given by a virtual character. In addition, the characters proposed in the discussion of the combination of Static IoT and Visualizing Level 3 in Group A, the combination of Static IoT and Visualizing Level 3 in Group C and the combination of Customizable IoT and Visualizing Level 3 in Group C are classified in the same category because they propose to use virtual characters like Ambient Bot.

In addition to these two elements, there is another virtual context type that gives the gaming mechanism as suggested in the discussion of the combination of Customizable IoT and Visualizing Level 1 in Group A. This is the same as what is commonly referred to as gamification.

The following three virtual context types are derived from the above discussions.

(1)

Worldview: in a virtual world, a designer decides what kinds of things have value in the world. In many cases, a deep story is not necessary in the virtual context. For example, in Virtual Aquarium there is a definition of the world that is the aquarium, and there is a worldview, where it is important to keep it beautiful.

(2)

Individuality of a character: it is possible to add virtuality by virtual characters. By displaying the characters a designer can give virtuality, extend an IoT product in various directions, and use it as a rhetoric to change or persuade a user’s behavior.

(3)

Game Mechanism: gamification also gives a type of virtuality. By introducing the gaming context, it becomes possible to use the product more positively and encourage users’ actions.

When designing products based on these dimensions, it is necessary to consider what kind of virtualization methods like AR or VR is good for the products. Therefore, we conducted another focus group to evaluate the revised design space. Four participants who had participated in the previous focus group discussed the issue again. The focus group's procedure was the same as the previous one. We discussed mainly the Virtuality Level in this focus group and they did not discuss Visualizing Level because this focus group was intended to complement the previous focus group analysis.

Table 3. The result of focus group discussing virtuality Levels.

	Virtuality Level 0	Virtuality Level 1
Static IoT (Autonomous Cultivation Controller)	Since the provided functions already achieved visualization of information, virtualization can be advanced by changing the Visualizing Level.	Adapting the idea of Virtual Aquarium to this / Customizing the character of the plant according to the user's preference and changing the state of the character according to the state of the plant/ Adding elements of gamification according to the result of pH level and watering timing.
Customizable IoT (Arduino)	When the user implements the function to visualize the data obtained from the sensor using a module to show a dashboard, he can display information virtually by designing Visualizing Level appropriately.	A fairy is in the module of Arduino, and she has preferences which modules she like. It also contains the virtual character and may include the world view why she exists. The user can know her world using Arduino modules adequately.

shows the result of the focus group. The result suggests that it is possible to discuss the IoT product based on the design space because the participants can propose the products in the focus group-based analysis. At this time, in order to facilitate the discussion, we clarified the concept of the proposed product by discussing it with existing IoT products as the starting point. Therefore, if the concept was decided in designing products, it was possible to consider virtuality based on the design space.

From the discussion, it turns out that it is easier to add virtuality for Static IoT. Introducing virtuality is to make things that do not exist behave as if they do. The product designers need to consider what they can feel through virtuality. In other words, in order to add virtuality to a product, it needs to have a purpose to incorporate it. Since the products and functions of Static IoT are fixed and the purpose of the product is clarified, it was easy for the participants to discuss about adding virtuality. On the other hand, Customizable IoT creates functions that users want to realize by combining modules and programming them. When trying to introduce the virtuality into it, a designer considers about what and how to display it; therefore, there is a possibility that the versatility of the product may be impaired. What is suggested is that if we can provide tools or frameworks for giving virtuality to designers, it will be possible to create Customizable IoT-introduced virtuality more easily without decreasing the customizability.

The individuality of a character that is one of the virtual contexts constituting Virtuality Level 1, which was discussed in the focus group. A participant said that “It is necessary to make a character by a user like a game. For example, it would be better to customize the personality and the appearance easily. However, if the user customizes everything, it will become the character that behaves completely as defined by the user. I think people tend to enjoy when a surprising event happens. Therefore, the interaction with the character whose behavior is also defined by the user will not be enjoyable because it has no unexpectedness. Perhaps customizability may be a trade-off with such interactivity.” In other words, it is suggested that the designers need to be aware of the trade-off between customizability to adapt to a user’s preference and unexpectedness when interacting with virtual characters. If the customizable product may be implemented so that the virtual characters can also be customized, it sometimes may not lead to an improvement in the user experience.

Another participant said that “Giving Virtuality Level may be inappropriate for some critical situations. For example, if an emergent event such as an accident is occurring, Virtuality Level 0, which gives information directly, is more suitable.” Lazlo Ring et al. shows a guideline for virtual agent character design according to the purpose [58]. The creation of guidelines for Virtuality Level according to these situations remains a future issue.

5. Future Direction: Incorporating Virtuality for a Community

The current design space to introduce virtuality only focused on individuals. Although there are examples that extend to the real world by providing virtuality to individuals, as we have introduced heretofore, there are almost no case studies to provide virtuality to a group. Cooperation in a society as a whole is necessary to solve realistic emerging social problems, and it can be predicted that utilizing virtuality is effective. For example, Foldit elucidates the protein structure using gamification involving people. This introduced virtuality to many individuals, but it did not introduce it to a group. The problem with gamification may be that motivation can only be enhanced for specific gamers. In addition, it is known that there are various types of gamers: people can diagnose their personality type with the Bartle test [59]. Since each user has different favorites, it is difficult to create a game that satisfies all users. Therefore, in order to design virtuality for a group, designers need to take into account the personality of each member in the group for making virtuality believable for all members in the group.

We consider that the next step is to provide virtuality on a community basis. Even though it is difficult for many different people to believe a given virtuality, it may be possible to trust virtuality for people in the same community sharing the same space, time and culture. By utilizing the design space at the individual level proposed by us and discussing what type of virtuality would be suitable to provide to a community in the future, the enhancement of the real world by virtuality can be further expanded. Then, introducing virtuality at the social level will be investigated.

In such a society enhanced by virtuality, people will be able to interact with the virtual world related to the real world both implicitly and explicitly, and to share the virtual world with others. The essence of Fusion Reality is that the virtual world is that which human beings created, and the real world is created by the nature. People’s lives will be changed when the two worlds cooperate with each other.

6. Conclusions

We discussed the design space for IoT products introducing virtuality, which means that things that do not physically exist become believable to people. We indicated that the design space has three dimensions: (1) Taxonomy of IoT products: it consists of Static IoT and Customizable IoT. Static IoT provides specific functions, and Customizable IoT enables a user to create his/her own customized IoT devices and functions. (2) Visualizing Level: it consists of three levels. Visualizing Level 0 uses physical displays and projectors, Visualizing Level 1 shows virtual objects on physical objects, and Visualizing Level 2 shows virtual objects independently of physical objects. (3) Virtuality Level: it is classified into two levels. Virtuality Level 0 shows information layered on the real world; for example, the news is shown on the wall like a dashboard. On the other hand, Virtuality Level 1 integrates a virtual context seamlessly embedded in the real world, and we define the virtual context for designing virtuality. The virtual context consists of the worldview, the individuality of a virtual character, and the game mechanism. Designers can investigate future products and services based on the design space we proposed. If virtuality is more widely explored and it is possible to practically provide virtuality to our actual society in the near future, it can be utilized for solving emerging social issues because virtuality offers a possibility to alter human behavior. The next step is to develop a working prototype system to introduce virtuality at the society level, and conduct a user study with a small-scale community. Then, we would like to enhance the proposed design space for incorporating the requirements to design virtuality for a group.