Review on the Application of Artificial Intelligence in Smart Homes

Abstract

Smart home and artificial intelligence technologies are developing rapidly, and various smart home products associated with artificial intelligence (AI) improved the quality of living for occupants. Although some studies discussed the application of artificial intelligence in smart homes, few publications fully considered the integration of literature and products. In this paper, we aim to answer the research questions of “what is the trend of smart home technology and products” and “what is the relationship between literature and products in smart homes with AI”. Literature reviews and product reviews are given to define the functions and roles of artificial intelligence in smart homes. We determined the application status of artificial intelligence in smart home products and how it is utilized in our house so that we could understand how artificial intelligence is used to make smart homes. Furthermore, our results revealed that there is a delay between literature and products, and smart home intelligent interactions will become more and more popular.

1. Introduction

In recent years, the development of smart home technologies contributed to the transition of the home from traditional to a smart internet-connected one. A smart home is a residence equipped with technologies that include sensors, wired and wireless networks, actuators, and intelligent systems [1]. Equipped with highly advanced automatic systems, smart homes can monitor and control home activities for convenience, provide occupants with better comfort, and possibly reduce energy use. Smart home technology collects and analyzes data from the domestic environment. It also relays information to users and enhances the potential of managing different domestic systems [2]. Artificial intelligence (AI) describes any device that perceives its environment and takes actions that maximize its chance of successfully achieving its goals [3]. The ideal state of artificial intelligence is thinking humanly, thinking rationally, acting humanly, and acting rationally. [3]

Several comprehensive review articles were published on applying AI technology to smart homes. Rho et al. selected nine manuscripts related to intelligent surveillance systems in the smart home environment to indicate that many researchers in the image processing and AI community focused on developing image and video analysis and understanding [4]. Researchers like Dermody et al. reviewed philosophical underpinnings and explained how this framework can guide nurse scientists collaborating with engineers to develop intelligent health-assistive smart homes [5]. They also noted that it is critical to integrate clinical nursing knowledge into smart homes and artificial intelligence features. The types of home automation systems and how these systems can utilize AI tools were discussed by Kumar et al. They defined the major applications of these systems as comfort ability, remote control, optimal resource utilization, and security. In these systems, AI plays the role of a knowledge and rule database, decision-maker, action implementor, and appliance controller [6]. There are some publications discussing the application of AI technology in smart homes. Huh et al. attached Raspberry Pi to a shoe cabinet at home to see the list of shoes, to store shoes automatically, and to recommend the right shoes for occasions. The most appropriate shoes would be recommended when information on the type of clothes worn and the destination was put in. The automatic storage of shoes was realized by controlling the input sensor and x–y floater with the Raspberry Pi attached to the shoe cabinet [7]. A house simulator was developed and used as an “expert system shell” to assist with the implementation and verification of the observe, learn, and adapt (OLA) algorithm by Qela et al. for better energy management and conservation in smart homes [8].

Al technology is also used in smart home products. For easy understanding, we defined six core clusters of AI functions in smart homes, i.e., activity recognition, data processing, voice recognition, image recognition, decision-making, and prediction-making. In the aspect of activity recognition, smart home devices can recognize human activity with the help of AI. It analyzes sensor data to detect people’s actions and raises an alarm if there is abnormal activity. Activity recognition is used in Hive Link and Essence Care@Home. In the aspect of data processing, AI is based on data analysis techniques, extracting information from a variety of data sources and identifying intrinsic relationships. It is used in August Smart Lock + Connect and Nest Protect. In the aspect of voice recognition, AI works based on voice-driven technologies, and allows people to interact with it simply by having a conversation, for instance, asking about the weather, ordering products online, or calling a cab. Voice recognition is used in Amazon Alexa, Google Home, Ivee Sleek, Jibo, Athom Homey, Apple HomePod, Josh Micro, etc. In the aspect of image recognition, AI is used to achieve facial recognition, emotion recognition, biometrics, and scene understanding. It can measure and analyze human behavior, as well as physical aspects of the body’s structure and form. It is used in Lighthouse, Nest Cam, Honeywell Smart Home Security System, Tend Secure Lynx Indoor Camera, Canary All-In-One, Netatmo Welcome Indoor Security Camera, etc. In the aspect of decision-making, AI plays the role of the decision-maker. It can decide what action should be taken in response to the input data [6]. For example, in a smart security system, if the camera detects a stranger breaking into the house, it triggers a loud alarm and gives an alert to the user’s smart phone, or it can call 911 automatically. These systems should be fast enough in response and effectiveness. It is used in Arlo Ultra, Ecobee4, VELUX roof windows/blinds, etc. In the aspect of prediction-making, sensors are embedded in the home which generate data while residents perform their daily routines. The sensor data are collected by a computer network and stored in a database to be processed by an intelligent agent generating useful knowledge such as patterns, predictions, and trends. On the basis of this information, a smart home can select and automate actions to achieve the goals of the smart home application [9]. It is used in Nest Thermostat, Olly, Viaroom home, etc.

There are many smart home device and solutions available, but only for lower levels of interaction, which include devices for defining environmental parameters and management of the home. However, there were a few attempts to develop the latest level of a smart home [10]. As we can see from Figure 1, it shows the interest over time based on Google Trends, whereby artificial intelligence is an industry showing rapid growth. It can be combined with smart home technology to become an innovative tool. However, few studies combined AI technology and smart home technology with the space and room types in the house, and the integration of literature and products was not fully considered either. Smart homes also need to be discussed based on the scope of architecture. This research contributed to find out the trends of smart home technology and products and to state the relationship between literature and products in this field. We conducted this review from the perspective of architecture and human concern.

2. Materials and Methods

2.1. Literature Review Method

Publications about applying artificial intelligence to smart homes were identified through three search engines, i.e., Web of Science, Elsevier’s Science Direct, and Scopus, from 2011 (the rise of deep learning, big data, and artificial intelligence) through 2019. The content included magazine articles, scientific papers, and conference papers. With the concept of smart homes, three online databases were used to search by using common search keywords, namely. “smart home”, “smart building”, “smart house”, “intelligent building”. and “home automation”. With the concept of artificial intelligence, we used “artificial intelligence”, “artificial intelligent”. and “AI”. The terms “intelligent house”, “intelligent home”, “machine learning”, “artificial agent”, “artificial neural network”. and “multi-agent system” were also allowed. Specifically, we searched the Scopus database using the search string (TITLE (smart home*) AND TITLE-ABS-KEY (artificial intelligen*)). Here, * represents different possible word endings. For example, “intelligen*” means “intelligence” or “intelligent”. Publications from a wide variety of academic publishers, such as Springer, Institute of Electrical and Electronics Engineers (IEEE), Blackwell, MDPI AG, and Institute of Physics and Elsevier B.V., were identified. However, there was a large overlap between search databases and publications. On one hand, the same publication was identified using same search terms in different databases; on the other hand, different search terms resulted in an overlapping set of publications. To deal with the overlap and select relevant publications in accordance with previously discussed research goals, two rounds of publication selection were conducted. After the first-round selection, 116 publications were chosen for research purposes. The selected data were then analyzed by adopting a qualitative inductive method. As the next step, we conducted a second-round selection to analyze the specific technology of AI and the function of smart homes. After the second-round selection, 20 publications were chosen for research purposes. In addition, we searched based on the international standards of ISO and not yet collected international standards related to applying AI to smart homes.

2.2. Product Review Method

There are three main smart home product databases, namely, Google search engine, iotlist.co, and smarthomedb.com (SmartHomeDB). Although the amount of data in the Google search engine is huge, it is not well organized. The platform, iotlist.co, has a very clean and elegant interface to show the list of Internet of things (IoT) devices on the market, but this platform is not restricted to smart home products and does not have a well-structured category. SmartHomeDB is another online platform that focuses on smart home devices and provides a detailed description of products. For these reasons, we chose SmartHomeDB as our product review data source. We also found some state-of-the-art cases in the Google search engine. For product data from previous years, we used the Wayback Machine website. It is a digital archive of the World Wide Web and other information on the Internet. The selected data were then analyzed by adopting a qualitative inductive method.

2.3. Analysis of the Application of AI in Smart Homes

The qualitative inductive method included several steps. In the aspect of the literature review, we extracted five core functions of smart homes, i.e., device management, energy management, healthcare, intelligent interaction, and security. Tang explained that expert systems, artificial neural networks, and intelligent decision-making systems were applied to intelligent buildings [11]. Based on that, we divided the AI functions in smart homes into six clusters, i.e., activity recognition, data processing, decision-making, image recognition, prediction-making, and voice recognition. In this article, data processing includes data mining, semantic analysis, and rule-based technologies.

In the aspect of the product review, we extracted six functions of products with AI in smart homes, i.e., energy management, entertainment system, healthcare, personal robot, intelligent interaction, and security. Next, we divided them into six clusters, i.e., activity recognition, data processing, decision-making, image recognition, prediction-making, and voice recognition.

Then, we carried out a quantitative analysis of the number of each group under literature and products. Finally, we summarized the role of AI in smart homes with different functions by analyzing the literature from the second-round selection and some specific products.

3. Results and Discussion

3.1. Result of Literature Review

3.1.1. First-Round Selection of Literature

In this section, the publications of five functions and six clusters are discussed in

Table 1. Sample table for results of literature review. AI—artificial intelligence; NLP—neuro-linguistic programming.

Function	Cluster	Title	Year	Discipline
Device Management	Data Processing	Design of TensorFlow-based proactive smart home managers	2018	Engineering
Device Management	Data Processing	Created in close interaction with the industry: the smart appliances reference (SAREF) ontology	2015	Computer Science
Device Management	Data Processing	A semantics-rich information technology architecture for smart buildings	2014	Engineering
Device Management	Decision-Making	Rudas: energy and sensor device management system in home automation	2016	Other
Device Management	Voice Recognition; Decision-Making	A voice-controlled smart home solution with a centralized management framework implemented using AI and NLP	2018	Computer Science
…	…	…	…	…
Energy Management	Activity Recognition	User activity recognition for energy saving in smart home environment	2016	Computer Science
Energy Management	Activity Recognition	Unsupervised detection of unusual behaviors from smart home energy data	2016	Computer Science
Energy Management	Activity Recognition	A user behavior-driven smart-home gateway for energy management	2016	Computer Science
…	…	…	…	…
Healthcare	Activity Recognition	Activity recognition system for dementia in smart homes based on wearable sensor data	2019	Computer Science
Healthcare	Activity Recognition	A novel and distributed approach for activity recognition inside smart homes	2018	Computer Science
Healthcare	Activity Recognition	A novel method for detecting and predicting resident’s behavior in smart homes	2018	Computer Science
…	…	…	…	…
Intelligent Interaction	Data Processing	Design and implementation of a smart home system using multisensor data fusion technology	2017	Engineering
Intelligent Interaction	Data Processing	User needs and wishes in smart homes: What can artificial intelligence contribute	2017	Other
Intelligent Interaction	Data Processing	An interaction-centric dataset for learning automation rules in smart homes	2016	Other
…	…	…	…	…
Security	Data Processing	Distributed and in situ machine learning for smart-homes and buildings: Application to alarm sound detection	2017	Other
Security	Data Processing	Detecting anomalous sensor events in smart home data for enhancing the living experience	2011	Computer Science
Security	Data Processing	Design and implementation of a smart home system using multisensor data fusion technology	2017	Engineering
…	…	…	…	…
Other	Prediction-Making	Hardware simulation of pattern matching and reinforcement learning to predict the user next action of smart home device usage	2013	Computer Science
Other	Voice Recognition	Making context-aware decision from uncertain information in a smart home: A Markov logic network approach	2013	Computer Science
Other	Voice Recognition	Semantic validation of uttered commands in voice-activated home automation	2012	Computer Science

(see Appendix A for full table). Defining and providing extensive discussions on various concepts lies beyond the scope of this paper; instead, the paper aims to reflect a comprehensive application of AI in smart homes.

As shown in Figure 2a, smart home device management is supported by five AI functions, i.e., data processing, decision-making, image recognition, prediction-making, and voice recognition. In the application of AI in smart home device management, the number of publications is not so high, and they are very new. There were some studies in this area since 2016. Smart home energy management is supported by five AI functions, i.e., activity recognition, data processing, decision-making, image recognition, and prediction-making. From 2012 to present, there were many studies in this area. As shown in Figure 2b, AI for data processing and prediction-making is more widely discussed in this area. Smart home healthcare is supported by six AI functions, i.e., activity recognition, data processing, decision-making, image recognition, prediction-making, and voice recognition. From 2011 to present, there were many studies in this area. As shown in Figure 2c, AI for activity recognition is more widely discussed in this area. Smart home intelligent interaction is supported by four AI functions, i.e., data processing, image recognition, prediction-making, and voice recognition. There were many studies in this area since 2012, but most of the research is very new. As shown in Figure 2d, AI for activity recognition is more widely discussed in this area. Smart home security is supported by two AI functions, i.e., data processing and image recognition. As shown in Figure 2e, the number of publications is not so high in this area. The rest of the studies are mainly about basic research on applying AI technology to smart homes. As shown in Figure 2f, data processing and activity recognition are widely used in all smart home applications.

The distribution of smart homes with the AI application field is shown in Figure 3. Taken together, these results show that, as time went on, more and more application fields were discussed, and both diversity and quantity increased over time. Since 2015, the research on healthcare decreased year by year, while the research on intelligent interaction increased year by year. Energy management research is also increasing. It could be perceived that, in the future, smart homes will pay more attention to the interaction between people and the environment, and to making buildings more sustainable and personalized.

3.1.2. Second-Round Selection of Literature

In the second round selection of literature, we chose 20 publications in each application field of smart homes. In this round, we discuss the findings based on five application fields, i.e., device management, energy management, healthcare, intelligent interaction, and security.

Firstly, in terms of smart home device management, with the advancement of technology, the number of electrical appliances in the home is increasing, and operation steps are becoming more and more complicated. It would be convenient if AI could help users automatically manage some devices. Some researchers implemented AI in smart home systems to monitor and manage things in the house by automatically controlling light and temperature conditions [12]. Intelligent control in a smart house can also be realized by analyzing the data from the sensor network, learning the user’s previous behavior [13], or user patterns by applying the logistic classification algorithm based on TensorFlow [14] by AI. Centralized management can make electronic decisions such as monitoring, improving comfort, convenience, controlling surrounding conditions, and delivering required information [15].

Secondly, in terms of smart home energy management, achieving a sustainable society becomes more and more important and urgent. People from all different fields are working hard to reduce energy consumption and improve energy efficiency. Coordinating the energy consumption of smart appliances in smart homes can achieve a higher consumption effciency [16]. Energy consumption patterns and their relationship with environmental factors can be analyzed by AI to predict daily electricity demand [17]. AI can help the smart home gateway in identifying the user’s energy consumption behavior in order to support home automation and reduce energy usage [18]. Activity recognition by AI can also help relate activities and existing home appliances, and then give recommendations to users whenever it detects energy waste [19].

Thirdly, in terms of smart home healthcare, with the gradual increase in life expectancy, home healthcare is becoming more and more important. Using machine learning and artificial intelligence methods from sensor data can track and detect changes in individuals’ behavioral pattern and lifestyle [20]. By adopting an unsupervised clustering algorithm, recurrent output neural network model, and genetic algorithm, AI systems can constantly monitor the elderly in smart homes and send an alert to the caregiver if any abnormal activities occur [21,22]. To achieve the goal of helping adults with cognitive impairments independently accomplish the activities of daily life, intelligent assistant agents need to recognize older adults’ goals and reasons behind the further steps desired [23].

Fourthly, in terms of smart home intelligent interaction, as the number of smart home devices increases, more intelligent interactions can make users feel more comfortable. We no longer need to go near each device to manually operate it. Most researchers utilized artificial neural networks to classify user inputs to create a natural dialogue, giving users the ability to control appliances by voice or text commands [24,25]. Voice recognition based on AI provides audio-based interaction technology that lets the users have full control over their home environment [26]. Image recognition also helps AI understand people’s gestures [27]. Gesture-based human–computer interaction is natural and intuitive. People with speech disorders can communicate with smart home devices through dynamic gestures [28].

Finally, in terms of smart home security, in order to protect property and personal safety, keeping one’s house from unexpected events and accidents is necessary. Artificial intelligence with regard to image recognition can recognize an unusual intruder and warn the house owner [29,30]. Not all danger comes from criminals, but also from CO₂, fire, etc. We can use AI to anaylze sensor data and detect alarm sounds [31].

We can see that AI technology, smart homes, and users have different interaction models. Basically, there are three types of interaction models. The first is shown in Figure 4a, where users directly give commands to each smart home device, and the AI embedded in each device benefits the specific device itself. Smart home energy management, healthcare, and security prefer this pattern. The second is shown in Figure 4b, where users give instructions to the AI, and the AI controls each device. Smart home device management and intelligent interaction work using this pattern.

3.2. Results of Product Review

In this section, the products with six functions in the smart home and six AI function clusters are discussed.

As shown in

Table 2. Sample table for results of product review.

Function	Technology	Product	Year
Energy Management	Decision-making; Prediction-making	Nest Learning Thermostat (3rd Generation)	2015
Energy Management	Decision-making	Ecobee4	2017
Energy Management	Decision-making	VELUX roof windows / blinds	2017
…	…	…	…
Healthcare	Activity Recognition; Decision-Making	Walabot HOME	2018
Healthcare	Activity Recognition	Hive Link	2019
Healthcare	Activity Recognition; Voice Recognition	Essence Care@Home	2016
Healthcare	Voice Recognition; Image Recognition	Pillo Health	2016
Intelligent Interaction	Prediction-making	Viaroom home	2018
Intelligent Interaction	Voice Recognition	Echo Dot (2nd Generation)—Alexa-Enabled	2016
Intelligent Interaction	Voice Recognition	Amazon Echo—Alexa-Enabled	2014
…	…	…	…
Personal Robot	Voice Recognition; Image Recognition	MATRIX	2015
Personal Robot	Voice Recognition; Image Recognition	Jibo	2015
Personal Robot	Voice Recognition; Image Recognition	ElliQ	2019
Personal Robot	Voice Recognition; Image Recognition; Prediction-making	Olly	2017
…	…	…	…
Security	Image Recognition	Honeywell Smart Home Security System	2017
Security	Image Recognition	Tend Secure Lynx Indoor Camera	2017
Security	Image Recognition	Canary All-In-One	2016
Security	Image Recognition	Netatmo Welcome Indoor Security Camera	2015

(see Appendix B for full table), AI for decision-making is more commonly utilized in smart home energy management. As shown in Table 2, the function of smart home entertainment systems is supported by one AI function—voice recognition. The function of smart home healthcare is supported by four AI functions—activity recognition, decision-making, image recognition, and voice recognition. As shown in Table 2, AI for activity recognition is more commonly utilized in smart home healthcare. The function of smart home intelligent interaction is supported by two AI functions—prediction-making and voice recognition. As shown in Table 2, AI for voice recognition is more commonly utilized in smart home intelligent interaction. The function of smart home personal robots is supported by three AI functions, i.e., image recognition, prediction-making, and voice recognition. As shown in Table 2, AI for voice recognition and image recognition is more commonly utilized in smart home personal robots. The function of smart home security is supported by three AI functions, i.e., data processing, decision-making, and image recognition. As shown in Table 2, AI for image recognition is more commonly utilized in smart home security. Figure 5a shows that most smart home products with AI are utilized in intelligent interaction and security. Figure 5b shows that the functions of voice recognition and image recognition are widely used in smart home products, while data processing and activity recognition are seldom utilized.

There are also some companies trying to utilize AI to help control the house. In 2018, Panasonic released a home information system “home X” which can record and analyze the living behaviors of its inhabitants, then automatically calculate and recommend various messages, and automatically close protective doors when a typhoon comes. The system can be iteratively upgraded like mobile phone software. Mark Zuckerberg created an AI assistant Jarvis to control his home, which can manage light, music, thermostat, etc. It uses AI for language processing, speech recognition, and face recognition. Jarvis has a personality and can even interact with users via message.

3.3. Relationship between Literature and Products

At first glance, the disproportionate distribution of functions of AI in smart homes between the literature and products attracts much attention. We compared the distribution of each technology and function in the literature and products. The result is shown in Figure 6. As we can see, there are not many studies on voice recognition and image recognition in publications, while the number of products is large. There are relatively many studies on prediction-making and data processing in publications, while not so many products utilize these technologies. These data are consistent with the notion in practice, whereby AI is more often used in the identification and recognition of the primary stage, while activity recognition, data processing, decision-making, and prediction-making require further development of artificial intelligence technology. From Figure 6, we can see the relationship between literature and products, that is, no one is in an absolute leading position. Literature is leading the way in complex technology of AI in recent years, while products are more subject to the market. Therefore, once a technology is relatively mature, there are more products using this technology.

As shown in Figure 7, in the aspect of the function in our house, energy management and healthcare are discussed in many publications, whereas not so many smart home products associated with AI are applied in this field. This may be explained by it not being necessary to use AI technology to help us in energy and resource management. The products of healthcare may not commonly be used in an ordinary house, or there are potentially some gaps in SmartHomeDB. More products focus on intelligent interaction and entertainment systems.

Generally, there is room for further improvement of AI in smart homes. Currently, smart homes are utilized more in energy management, intelligent interaction, and security with AI functions of voice recognition and image recognition. In the foreseeable future, more and more products will use activity recognition, data processing, and prediction-making.

There may be some possible limitations in this study. Firstly, our subcategories for AI were not be chosen in a very systematic way. Secondly, the smart home product database we chose does not cover the newest products. Thirdly, if the keywords used in this article did not appear in some relevant publications, they were not searched.

4. Conclusions

This study aimed to reveal how AI makes homes smart. To achieve this goal, many studies in the literature and several products were reviewed. We found that AI technology helps smart homes in device management, energy management, healthcare, intelligent interaction, security, entertainment systems, and personal robots by utilizing activity recognition, data processing, decision-making, image recognition, prediction-making, and voice recognition. There is a delay between the literature and products, whereby the products concentrate on relatively simple methods like image recognition and voice recognition. The literature concentrates on relatively complicated methods like activity recognition and prediction-making. AI with voice and image recognition is widely used in smart home products, while the technologies of activity recognition, data processing, and prediction-making still need to be developed.

Furthermore, an interesting finding in this study was that intelligent interaction is becoming more and more important both in the literature and products. In the foreseeable future, smart homes will pay more attention to the interaction between people and the environment to make buildings more sustainable and personalized. One important future direction in applying AI to smart homes is considering both smart home technology and architecture design and developing relevant standards.