Using Conversational Assistants and Connected Devices to Promote a Responsible Energy Consumption at Home ^†

Abstract

Today, households worldwide are being increasingly connected. Mobile devices and embedded systems carry out many tasks supported by applications which are based on artificial intelligence algorithms with the aim of leading homes to be smarter. One of the purposes of these systems is to connect appliances to the power network, as well as to the internet to monitor consumption data among others. In addition, new interaction ways are emerging to manage all these systems. For example, conversational assistants which allow us to interact by voice with devices at home. In this work, we present GreenMoCA, a system to monitor energy consumption data from connected devices at home with the aim of improving sustainability aspects and reducing such energy consumption, supported by a conversational assistant. This system is able to interact with the user in a natural way, providing information of current energy use and feedback based on previous consumption measures in a Smart Home environment. Finally, we assessed GreenMoCA from a usability and user experience approach on a group of users with positive results.

1. Introduction

Integration of new technologies in everyday life, and therefore at home, is a reality today. The arrival of connected devices and embedded systems based on the Internet of Things (IoT) paradigm provides a wide range of ICT solutions according to the context in which these systems are deployed, as well as the user needs.

Development and deployment of IoT solutions at home have led to the building of smarter environments, today known as Smart Home. In these domains the use of applications and connected devices is considerably increasing. In fact, in the year 2017, there were about 209 million Smart Home solutions, and this number is expected to amount to 975 million by 2023 (https://www.statista.com/outlook/279/100/smart-home/worldwide#market-users). These amounts include technological solutions in the areas of: Home Entertainment, Smart Appliances, Energy Management, Control and Connectivity, Comfort and Lighting, and Security. In this regard, Energy Management currently implies 43 million solutions worldwide, and it is expected to increase to 111 million by 2023. The control of energy consumption at home leads to more sustainable households, helping to reduce the current carbon footprint.

Integration of conversational assistants (CAs) at home is another trend to take into account today. Normally, these are used to consume general information from the internet, like weather forecasts, switch on/off lights and appliances connected to compatible devices, or interact with mobile applications, among others. However, CAs in combination with other connected devices can provide use of mechanisms focused on different purposes but based on simple and natural interactions through the use of voice.

In this paper, we present a system to monitor the energy consumption at home using commercial hardware devices, such as CAs and smart plugs, and several software modules developed to gather energy consumption data and to interact with end users by voice. The aim of this work is to empower the user on responsible energy consumption and, at the same time, provide feedback through a natural interaction in a smart home environment.

The paper is divided into the following sections: Section 1 presents the context of this work and its aims. Then, Section 2 shows relevant studies and works related to the one presented. In Section 3, CAs and connected devices used in this work are detailed. Section 4 explains the developed system in detail, taking into account both software and hardware components. Then, in Section 5, a usability assessment and its results are shown and discussed. Finally, Section 6 presents the conclusions of this work.

2. Related Work

In the last decade, the number of computational systems which use natural language (NL) to interact with people has considerably increased. The emergence of new hardware devices, chatbots, and the development of artificial intelligence paradigm involves new ways of communicating between users and computers. In the case of NL, its processing is not new; for example, Weizenbaum [1] developed a computer program in the 1960s, called ELIZA, to simulate the natural language communication between human and machine. Further advances in automatic learning, artificial intelligence, and embedded systems have enabled better speech recognition and understanding by these machines. In recent years, these systems have led to what we know as conversational assistants (CAs).

One of the main concerns about the natural interaction with CAs is the sense of context and their behavior in situations that require greater attention and rapid response, as well as the impact of this on people and consideration of sensitive issues [2,3]. Fundamentals of artificial intelligence, neuronal networks, deep learning, or cognitive computing are the basis of modern conversational agents, in which learning through training of their algorithms implies reasoning for decision-making accompanied by natural interaction [4,5,6,7].

Currently, there are many domains in which CAs, supported by a base of knowledge and the appropriate mechanisms, can be deployed. In the case of Healthcare, authors, such as Chung [8], propose a chatbot embedded into a mobile device for helping patients with basic medical services. For example, it can help to know their health condition or to provide a fast treatment, among others. In more recent studies, Montenegro et al. [9] and Denecke et al. [10] analyzed the application and integration of conversational agents in the field of health, focusing on current flaws and future challenges to deal with that can be anticipated. In terms of Education, CAs are deployed to boost learning in different areas of knowledge. In these cases, the CAs are also integrated into web applications [11] or Smartphones [12]. However, a CA is only a user interface which provides mechanisms to be closer to a human-human interaction [13]. Hence, one of the key points to the success of these systems in any domain lies in the effective application of algorithms and internal information processing, most of which are based on artificial intelligent methods.

Furthermore, the widespread nature of the IoT concept leads to being aware of phenomena, such as climate change, which is a trend today. Thanks to IoT devices and advances in artificial intelligence field, cities, buildings, and households are becoming smarter, and one of the main issues is precisely the struggle associated with the sustainability of these environments in terms of energy. In this regard, energy monitoring systems have been developed in order to control, monitor, and optimize the energy consumption, for example, at home. These kinds of systems are known as Smart Home Energy Management Systems (SHEMS) [14], and they contribute to reducing the energy consumption by taking into account all devices connected to the power line. These connected devices (CDs) are usually appliances which use smart plugs to gather and send information about said appliance to a computer system. In this way, an user can figure out how much energy a CD is consuming (in real time), as well as switching on or off said device remotely, among other features. In addition, feedback from monitored data is essential to promote responsible energy consumption [15]. The usage of CAs as a bridge to control monitoring systems tries to reduce the interaction gap between users and their own systems which occurs mostly in a tangible and visual way and, at the same time, contribute significantly to a more sustainable use of energy at home [16].

Today, there are many options to develop and deploy solutions in this context. In fact, it is not necessary to manufacture dedicated devices, but commercial ones can be used for our purposes. This includes existing conversational devices and smart plugs supported by software systems, as shown in this work.

3. Interaction with Conversational Assistants and Connected Devices at Home

Lately, large companies, like Google or Amazon, are standing up for the integration of CAs at home. Using voice as main mechanism of interaction, users are able to carry out almost any action. Besides, these CAs are a complement to other hardware devices, like smart plugs, which can manage the appliances connected to them. Thus, the switching on and off of these devices, as well as the acquisition of information about energy consumption, are some of their features.

This work uses the following commercial hardware devices:

• Google Home Mini (https://store.google.com/us/product/google_home_mini_specs): Although the system presented in this paper can be deployed by using other CAs, we chose this Google device in order to facilitate the interaction by using voice commands and the communication with the rest of the system components.
• WiFi TP-Link HS110 (https://www.kasasmart.com/us/products/smart-plugs/kasa-smart-plug-energy-monitoring-hs110): This smart plug can be connected to a home appliance, as well as the power line. In this manner, we can switch on/off the connected device remotely or get the energy consumption of said device in real time.
• Raspberry Pi Zero (https://www.raspberrypi.org/products/raspberry-pi-zero/): All software components of our system are deployed in this hardware artifact. Hence, it can establish a wireless communication with the smart plug and with the CA in order to facilitate energy monitoring and the processing of voice commands.

As we mentioned above, this work takes advantage of commercial devices, instead of designing dedicated ones, to facilitate its deployment in any home at a reduced cost.

4. System Overview

The system presented in this paper is divided into the following three main components: (1) GreenMoCA Framework, (2) GreenMoCA Energy API, and (3) GreenMoCA Monitoring Module. Figure 1 shows a general overview of the system, highlighting all software components, as well as the interaction between them, and considering the hardware artifacts and end users.

First, the user interacts with the CA, which receives the voice command and send this to the GreenMoCA Virtual Assistant. This component is in charge of standardizing the data before sending them to GreenMoCA Client. This component communicates with the corresponding services of GreenMoCA Energy API, according to the initial voice command. The result of this request is returned to the GreenMoCA Client, which will build the response to be sent to the GreenMoCA Virtual Assistant. Finally, this component sends the information to the CA, which communicates with the user by voice, providing an appropriate answer to the initial query. The communication between the user and the system is bidirectional, since all information along the elements of the system follows a “request-response” flow.

Moreover, for each home appliance, data of energy consumption can be gathered continuously thanks to the smart plug and the GreenMoCA Monitoring API. These data are saved into a MongoDB (https://www.mongodb.com/) database to be consumed by the GreenMoCA Client by means of the GreenMoCA Energy API.

In next sections, the three main components will be explained in detail.

4.1. GreenMoCA Framework

The first software component of our system is called GreenMoCA framework. This module is divided into two software pieces to allow communication between the CA and the GreenMoCA Energy API. Once the user interacts with the CA by means of voice commands, this information is automatically translated into text to be processed by the GreenMoCA Virtual Assistant. Taking into account that we are using a Google Home Mini as the CA device, we will use Dialogflow, a tool provided by the Google Cloud Platform in order to carry out the User-Assistant interaction through modules to improve the natural language understanding (NLU). Thus, these modules can be included in the GreenMoCA Virtual Assistant, translating all requests from speech to text, and vice versa. Besides, this translation occurs when there is a coincidence between the own voice commands and certain expressions called Intents. Each Intent defines one or more training phrases within a context, so that the more training phrases we create, the more accurate the response of the CA. It is important to mention that a training phrase can contain one or more entities. An Entity is a mechanism to identify and extract valuable data from the natural language inputs, as if they were keywords in a sentence. Figure 2 shows an example of “Energy” entity and some synonyms which could be recognized in a training phrase of an Intent.

Furthermore, this framework is able to manage the following types of requests (as voice commands):

• Device-based requests. This kind of request provides us with information about energy consumption of a specific appliance in real time. Additionally, the user will be able to switch on/off said device.
• Location-based requests. With these requests, the user can find out the real-time energy consumption of a specific location at home, taking into account all connected devices.
• Time-based requests. As historical data, the system can deal with requests based on datetime. In this sense, the framework is able to provide information about energy consumption in different time periods, managing days, weeks, months or seasons, as well as consider the whole environment, a specific location, or even a particular device connected to a smart plug.
• High energy consumption-based requests. These requests are focused on getting to know which devices have the highest energy consumptions at home. Likewise, these also take into account real time data or historical information for specific locations and devices.

As mentioned above, we need to train the GreenMoCA Virtual Assistant with a set of training phrases to provide proper answers to the user, according to all these kinds of requests. Figure 3 shows an example of training phrases which the user can pose to GreenMoCA system. Thus, the system should provide the same answer regardless of how the user asks the question.

In a second stage, the GreenMoCA Virtual Assistant sends the information, conveniently standardized, to the GreenMoCA Client. This component transforms the request into a web service call, which is managed by the GreenMoCA Energy API to return a valid HTTP response. Then, the GreenMoCA Client will send this response to the GreenMoCA Virtual Assistant directly, but it could also enhance the response with a valuable feedback to promote responsible energy consumption. To do this, the GreenMoCA Client will make some extra requests to the GreenMoCA Energy API related to the request initially made by the user. Figure 4a presents an example of response and feedback according to said user requests (the words in quotation marks are replaced by the corresponding value). Thus, the user will be aware of the difference in energy consumption (in this case, associated with current week versus past week). Besides, the system will provide general recommendations in other cases, as shown in Figure 4b. The system behaves in a similar way for other types of requests.

The GreenMoCA Client module implements a multiplatform framework known as Jovo (https://www.jovo.tech/). This element has the necessary features to facilitate the communication between the Dialogflow tool and the GreenMoCA Client’s business logic in an agnostic way. The main advantage of Jovo is the possibility of working with the entities (see an example of this in Figure 2) and carrying out actions based on them. The Jovo framework can also be used by other CAs, such as Amazon Alexa (https://developer.amazon.com/alexa).

4.2. GreenMoCA Energy API

This software module focuses on satisfying all queries which come from the GreenMoCA Client in a low level of abstraction. This means that it can manage the requests by means of web services, in our case developed with the Flask (https://palletsprojects.com/p/flask/) framework, as well as the access to the database. The Green Energy API works with information about entities extracted by the GreenMoCA Client, thanks to the Jovo framework, and uses this to make the corresponding API calls, receiving the appropriate responses which will be returned to the GreenMoCA Client.

Finally, in Figure 5, we can see the sequence diagram with the interaction process between the system components, from the time the user asks a question (by voice) until the answer is received.

4.3. GreenMoCA Monitoring Module

This software module consists of two components (see Figure 1): (1) tplink-smartplug API and (2) GreenMoCA Monitoring API. The first component provides “a bridge” between the smart plug and the GreenMoCA Monitoring API. This second component is a RESTful API (https://restfulapi.net/) developed in Flask which contains all necessary services to switch on/off the connected devices, as well as to save energy consumption data. In this regard, energy measures (in watts) are saved into the database every 30 s. Likewise, a set of triggers have been implemented to save historical energy data from such instant measures, resulting in hourly, daily, and monthly ones. This fact facilitates the processing of time-based requests by the GreenMoCA Energy API.

5. Evaluation

The system was assessed in a lab environment from usability and user experience viewpoints. Ten users participated in the experiment, providing their impressions and feedback, which are useful to improve the GreenMoCA system in future efforts. The protocol followed in the evaluation and the descriptive analysis of the sample of participants are specified as follows. The results in terms of usability and user experience are presented and discussed in the subsequent subsection.

5.1. Description of the Experiment

Each participant received a general description of the GreenMoCA system and then performed five representative tasks, presented in

Table 1. Representative tasks performed by the participants in the usability and user experience evaluation.

#	Task Performed	Pursued Goal
1	Turning an appliance ON and OFF.	The goal is to determine whether the actuation to turn on/off an appliance through speech is the right one for the user.
2	Querying active appliances in a location.	In order to enable control over the appliances via voice, the user is first asked to obtain which ones are connected to the power line in a certain location.
3	Weekly energy consumption.	The goal is to analyze the interaction between the users and the GreenMoCA Virtual Assistant when total energy consumed during the current week is requested. Their opinions are gathered reflecting whether the information provided seems complete or not for them.
4	Querying energy consumption in two different months.	In order to establish comparisons of consumption that help users to have more awareness of energy expenditure, they are asked to formulate a question about total household consumption in two different months.
5	Querying which is the appliance with the highest instant energy consumption.	The users interact with the GreenMoCA Virtual Assistant trying to find out which appliance has the highest instant energy consumption at home.

, with different pursued goals. The entire evaluation was conducted in Spanish language.

The tasks to be performed require interacting by voice with the GreenMoCA system. All of them have to be started with the voice command “Ok Google” or “Hey Google” to establish a conversation thread with the Google Home Mini, and then the user has to pronounce the phrase “Talk to my GreenMoCA assistant” to directly interact with the GreenMoCA Virtual Assistant.

The tasks to be performed by the users include actions related to turn ON/OFF appliances and the monitoring of energy consumption at different time periods and home locations. For instance, task #1 consists of turning a lamp ON and OFF. As already illustrated in Figure 3, different training phases and entities that have been used are associated with every defined intent. For task #1, switching on the lamp, the intent is triggered by different requests (translated from Spanish), such as: (i) Could you turn on the lamp?, (ii) Turn on the lamp, (iii) Can you light the lamp?, etc. The requests are similar for the case of turning off the light. Figure 6 shows the setup used in task #1 during the evaluation experiment, where a desk lamp, the Google Home Mini, and the WiFi TP-Link HS110 smart plug can be observed on the table.

The remaining tasks in Table 1 (#2–#5) guide the users in performing different variations of energy consumption queries, from knowing which appliances are ON (active) in a certain location of the house, like the kitchen (task #2), to consulting which appliance is currently consuming more energy throughout the entire home (task #5). In addition, two more tasks are proposed focused on knowing weekly consumption history (task #3) or getting the accumulated consumption of the home in two different months for comparison (task #4).

The participants in the evaluation experiment could obtain adequate feedback from the GreenMoCA system on the energy consumption queries due to the prior gathering of consumption data of specific household appliances from two locations inside the home (living-room and kitchen). Particularly, two months of energy consumption data were collected from different appliances within these two locations at the time of the evaluation experiment.

Usability and user experience are evaluated by means of a questionnaire with four questions/ statements for each of the five representative tasks.

Table 2. Questions/statements to be answered by the users for each representative task.

#	Task Performed	Questions/Statements
1	Turning an appliance ON and OFF.	1 Has the appliance been turned ON or OFF correctly? 2 Do you think this type of actuation over appliances is appropriate at home? 3 Do you consider the expressions used are appropriate for turning ON/OFF the appliance? 4 Do you think the processing done by GreenMoCA is effective?
2	Querying active appliances in a location.	1 Are the expressions (query) correctly interpreted/understood by GreenMoCA? 2 Does the information returned by GreenMoCA correspond to the query requested? 3 Do you consider the response time (latency) for the return of information to be acceptable? 4 Do you think it is interesting to be able to query the active (powered on) appliances at home?
3	Weekly energy consumption.	1 Is the transmitted order correctly processed by GreenMoCA? 2 Does the information returned by GreenMoCA correspond to the query requested? 3 Do you think that the information returned by this query is useful to promote responsible energy consumption? 4 Does it make it easier to track the consumption of household appliances at the level of location or room of the home?
4	Querying energy consumption in two different months.	1 Are the dates (months) indicated by the user correctly interpreted/understood by GreenMoCA? 2 Does the information returned by GreenMoCA correspond to the query requested? 3 Do you think that with this kind of query and the information retrieved you can control better the activity of the appliances? 4 Do you consider this query to be useful to have more knowledge of the consumption of the appliances?
5	Querying which is the appliance with the highest instant energy consumption.	1 Are the expressions (query) correctly interpreted/understood by GreenMoCA? 2 Do you consider the information returned is complete, clear and concise? 3 Do you consider the response time (latency) for the return of information to be acceptable? 4 Do you think it’s interesting and useful to know the appliance that is consuming the most power at the current moment?

contains all the questions/statements for each task. Users specify their level of agreement or disagreement with each question/statement on a symmetric agree-disagree scale. Particularly, the typical five-level Likert scale is used with ratings ranging from #1 Strongly disagree to #5 Strongly agree, with #2 Disagree, #3 Neither agree nor disagree and #4 Agree as middle categories. A simple numeric scale $[1--5]$ can be created as the average of responses over the set of individual questions/statements enclosed in each representative task.

Participants

Ten users participated in the evaluation experiment. As illustrated in Figure 7, 70% of participants were in the range of 27 to 40 years old and the remaining 30% had ages between 18- and 26-years-old. In relation to the gender, 80% were men and the remaining 20% were women. Finally, to the question as to whether the user has a conversational agent similar to Google Home Mini at home, 70% answered NO, compared with 30% who answered affirmative.

5.2. Results and Discussion

The radial chart in Figure 8 illustrates the final score for each task after averaging the numeric responses (Likert scale) for its set of questions and taking into account all the participants.

Positive results can be observed in the radial chart, with values quite close to the Strongly agree Likert level ($\approx 5$) for each of the tasks. Although differences are small, it can be appreciated that the task of turning ON/OFF an appliance (task #1) is the one that provides a greater degree of satisfaction to the user.

Table 3. Mean and Standard Deviation for the scores in the questions/statements of task #1.

1. Turning an Appliance ON and OFF	(Mean ± StDev)
1. Has the appliance been turned ON or OFF correctly?	5.0 ± 0.0
2. Do you think this type of actuation over appliances is appropriate at home?	4.8 ± 0.4
3. Do you consider the expressions used are appropriate for turning ON/OFF the appliance?	4.9 ± 0.3
4. Do you think the processing done by GreenMoCA is effective?	4.8 ± 0.4

shows the average scores of the questions made for this task. The consensus can be noticed in all the answers with deviations equal to or less than ± 0.4 points and average values that do not fall below 4.8 points, highlighting the correct working of the appliance ON/OFF feature (5.0 ± 0.0) and the positive appreciation of the expressions used in the conversation with the GreenMoCA Virtual Assistant (4.9 ± 0.3). On the other hand, task #4 has the lowest score, although it is still very positive (4.7/5.0). Looking in detail at this particular task in

Table 4. Mean and Standard Deviation for the scores in the questions/statements of task #4.

4. Querying Energy Consumption in Two Different Months	(Mean ± StDev)
1. Are the dates indicated by the user correctly interpreted by GreenMoCA?	4.9 ± 0.3
2. Does the information returned by GreenMoCA correspond to the query requested?	4.7 ± 0.7
3. Do you think that with this kind of query and the information retrieved you can control better the activity of the appliances?	4.7 ± 0.5
4. Do you consider this query to be useful to have more knowledge of the consumption of the appliances?	4.5 ± 0.5

, a slight reduction in the score of the fourth question can be seen, which places it halfway between the categories of Agree and Strongly Agree (4.5 ± 0.5). Thus, not everyone strongly agrees that comparing the consumption of two specific months is useful to have better knowledge of the energy consumption of the appliances.

6. Conclusions and Future Work

The use of IoT devices in combination with CAs implies the development of emerging systems for many purposes, which is a research trend today. In this work, we detailed a system based on commercial smart plugs connected to appliances, as well as CAs, called GreenMoCA, in order to gather and retain data about energy consumption at home with the aim of promoting the responsible energy consumption. The use of CAs as a human-device interface makes it easier to interact with devices, as shown in the results of the usability and user experience evaluation carried out in our work, improving aspects, such as user empowerment and engagement, when we are interacting by voice.

Moreover, we are working on deploying this solution in a larger real environment with more connected appliances, locations, and users. Likewise, the integration of machine learning mechanisms in our system could help to find out wrongful behaviors of people at home. Thus, we can make out usage patterns to study, among other things, mild cognitive problems early.