Activity-Aware Energy-Efficient Automation of Smart Buildings

This paper introduces the idea of activity-aware cyber-physical systems (CPS). Activity-aware systems allow smart city services to adapt to the needs of individual residents by being sensitive to their daily tasks. The paper first defines activity recognition and activity prediction algorithms that form the foundation of activity-aware CPS and implement a prototype activity-aware building automation system, called CASAS activity aware resource learning (CARL). Evaluation of CARL on real sensor data shows not only an accurate ability to sense and predict activities but an effective means of automation buildings that reduces energy consumption while being sensitive to user activities in the building. Our ideas are demonstrated in the context of a smart home but can be utilized in a variety of smart city settings including smart offices, smart hospitals, and smart communities.


Introduction
In recent years, cyber-physical systems (CPS) have been enhanced by the notion of context-aware computing.Sensing the current situation and reasoning about its implication can improve the design of the physical system and enhance its real-time system resiliency and responsiveness.In this paper, CPS are pushed to this level by introducing the notion of activity-aware CPS.Deploying activity-aware CPS requires several computational components to make them aware of user activities.Such CPS need a method of identifying current activities (activity recognition), as well as a method of forecasting when activities are going to begin and end (activity prediction).
Activity-aware systems are valuable when transforming cities to smart cities because the services, such as building automation, transportation routing, and energy provisioning, can now adapt to the needs of individual users.The hypothesis of this paper is that smart buildings, a key component of smart cities, can benefit from being activity aware.This hypothesis is validated in the context of a CPS that automates building control for energy efficiency.Our activity-aware smart automation system, CARL (CASAS activity aware resource learning), is built on the foundation of the CASAS smart environment infrastructure [1].Data are collected from sensors embedded in everyday building settings found in smart cities, such as smart offices, smart hospitals, and smart homes.The collected data are used to identify activities that residents are performing and to determine the devices that are used in the context of those activities.
The goal of CARL is to automate a smart building by turning off devices that are not needed for the current activity and leaving on devices that are required.By recognizing the current activity, a building found in a smart city is sensitive to its residents and does not turn off devices that they need.User adaptation is then further enhanced by predicting when the current activity will end and the next begin.By providing this activity-aware energy-efficient building automation, smart cities can realize energy savings while still meeting the needs of the individuals who live and work there.To validate our notion of an activity-aware energy-efficient building, CARL is evaluated to determine its ability to efficiently automate an actual smart building without disrupting resident activities.

Energy-Efficient Smart Buildings
The impact of lifestyle choices on energy usage and the environment are becoming increasingly noticeable and therefore a focus of resource on building automation and smart cities.As a result, research attention is being directed toward green technology, environmentally-friendly building design, and active demand response within the smart grid.This article examines the behavior side of sustainability and introduces ubiquitous computing technologies that may aid in reducing energy consumption.In particular, an activity-aware intervention is described that promotes energy efficient, sustainable building automation.
In 2015, the United States consumed 97.651 quadrillion BTU of energy, a 300% increase from 1949 [2].The growth of energy usage is not entirely due to manufacturing plants and automobiles: residential and commercial buildings are responsible for 40% of the energy consumption [3].There exists evidence that residential consumer behavior can be influenced to be more sustainable.For example, home residents have reduced consumption by as much as 15% in response to simply viewing raw usage data [4].Changing behavioral patterns in these environments can influence usage by as much as 90% in commercial buildings and 100% in household settings [5].
Until recently, occupant behavior has been difficult to accurately capture.Self-reports of behavior and energy consumption is error prone for some populations [6] and whole-home meter monitoring does not capture the behaviors in the home that influence consumption.Approaches have been utilized to explore the gap between the minimum amount of consumption that is needed for daily activities and the consumption that is actually observed [7].Some early work has focused on linking resident activity with energy consumption.The hypothesis that providing users with knowledge about the relationship between their activities and energy consumption and automation support for energy reduction will result in substantial decreases in overall consumption is supported by an increasing body of work that links awareness of energy consumption and its impact on behavioral routines and behavioral change [8][9][10].Until recently, validating this hypothesis was not possible.However, with the convergence of technologies in ubiquitous computing and machine learning, gathering data on human behavior is now automatable.Data can be collected from sensor-filled smart buildings and smart phones in an unobtrusive manner while individuals perform their normal daily routines.Because these sensor modalities operate in a continuous mode, feedback and interventions repeat ad infinitum, thereby maximizing the persistence effect.
In this paper, information from sensor-filled environments is utilized to intelligently automate a smart building.Automating control of buildings for energy efficiency has been explored by other groups [11].However, this work represents the first known approach in which activity awareness is used to more intelligently automate the environment.

Smart Environments
Computers are commonly embedded in familiar objects such as home appliances and mobile devices, gradually pervading almost every level of society.In the last decade, machine learning and pervasive computing technologies have matured to the point where this power is not only integrated with our lives but it can provide context aware, automated support in our everyday environments.One physical embodiment of such a system is a smart home.In the home or other smart building environment, computer software that plays the role of an intelligent agent perceives the state of the physical environment and residents using sensors, reasons about the state of the environment using artificial intelligence techniques, and then takes actions to achieve specified goals.
Energies 2016, 9, 624 3 of 17 Activity-aware building automation CPS can be accomplished with any sensor-filled physical environment (Figure 1).One physical embodiment of such a system is a smart home.In the home environment, computer software that plays the role of an intelligent agent perceives the state of the physical environment and residents using sensors, reasons about this state using machine learning and data mining, and then takes actions to achieve specified goals.
physical environment and residents using sensors, reasons about this state using machine learning and data mining, and then takes actions to achieve specified goals.
Smart home technology is being increasingly recognized valuable for applications including health monitoring and home automation [12].Smart home projects, including the Aware home [13], the Gator Tech smart home [14], and the MavHome [15], demonstrated the capabilities of using sensors and computers to create a home that reasons about its state and takes actions to make the home more comfortable.Smart homes have recently been a focus for companies including GE, Intel, iControl, Control4, Brillo, and Google, who are creating smart home operating systems, interfaces, developer platforms, and maintenance plans for the consumer.Many of these projects provide a basic infrastructure for collecting sensor data and automating devices.The key to making such environments intelligent is the software that reasons about the home using techniques, such as activity recognition and activity prediction, as is the focus in this paper.In this paper, the system is implemented and evaluated in the context of a CASAS smart home.Due to the difficulty of creating a fully-functional smart environment infrastructure, many of the early smart home projects described in the previous paragraph are tested on simulated or lab-based data [16,17].To support the scaling of smart environment research, a streamlined "smart home in a box" (SHiB) was designed [1], shown in Figure 2. SHiB components communicate via bridges.Bridges are created for Zigbee communication, for archiving sensor messages in a relational database, and for each application.While each site runs independently, the smart building site also securely upload events to be stored in a relational database in the cloud.Smart home technology is being increasingly recognized valuable for applications including health monitoring and home automation [12].Smart home projects, including the Aware home [13], the Gator Tech smart home [14], and the MavHome [15], demonstrated the capabilities of using sensors and computers to create a home that reasons about its state and takes actions to make the home more comfortable.Smart homes have recently been a focus for companies including GE, Intel, iControl, Control4, Brillo, and Google, who are creating smart home operating systems, interfaces, developer platforms, and maintenance plans for the consumer.Many of these projects provide a basic infrastructure for collecting sensor data and automating devices.The key to making such environments intelligent is the software that reasons about the home using techniques, such as activity recognition and activity prediction, as is the focus in this paper.
In this paper, the system is implemented and evaluated in the context of a CASAS smart home.Due to the difficulty of creating a fully-functional smart environment infrastructure, many of the early smart home projects described in the previous paragraph are tested on simulated or lab-based data [16,17].To support the scaling of smart environment research, a streamlined "smart home in a box" (SHiB) was designed [1], shown in Figure 2. SHiB components communicate via bridges.Bridges are created for Zigbee communication, for archiving sensor messages in a relational database, and for each application.While each site runs independently, the smart building site also securely upload events to be stored in a relational database in the cloud.
Data has been collected in 80 smart environment sites to date.This paper highlights one such site, a smart home environment named Navan.Navan is a single-resident apartment with a floor plan, shown in Figure 3. Navan is equipped with 118 sensors.To track the location of smart home residents we place infrared motion sensors on the ceilings with removable adhesive strips.Most of the motion sensors are focused to sense area in a one-meter diameter area immediately below the sensor.However, additional motion sensors are placed in each major room, which have a much broader coverage in order to indicate whether human (or pet) motion is occurring anywhere in the room.The circles in Figure 3 represent the positions of the motion sensors.The square icons in the figure indicate the presents of magnetic door sensors, which register the open/shut status of external doors as well as cabinets in the kitchen and bathrooms.Coupled with these are additional sensors that monitor ambient light and ambient temperature, which are useful for recognizing key activities such as bathing and cooking and for sensing internal (and to an extent, external) weather conditions.Additionally, Navan also includes temperature-only sensors (represented as stars in the figure) that are placed in pairs throughout the apartment at 8 11 from the ceiling and 12 11 from the floor to identify temperature gradients.Electricity usage data are collected in Navan using a Ted5000 power meter that provides instantaneous usage wattages every few seconds.Arduino-based WiFi thermostats (represented by hexagonal icons in the figure) were designed, built, and installed to monitor use of the baseboard heaters in individual rooms and to log temperature setpoints.In this paper, the system is implemented and evaluated in the context of a CASAS smart home.Due to the difficulty of creating a fully-functional smart environment infrastructure, many of the early smart home projects described in the previous paragraph are tested on simulated or lab-based data [16,17].To support the scaling of smart environment research, a streamlined "smart home in a box" (SHiB) was designed [1], shown in Figure 2. SHiB components communicate via bridges.Bridges are created for Zigbee communication, for archiving sensor messages in a relational database, and for each application.While each site runs independently, the smart building site also securely upload events to be stored in a relational database in the cloud.The sensors in the smart home are discrete event sensors.When a state change is sensed (e.g., there is motion in the area, a cessation of motion in the area, a significant temperature change, or a change in door status), the sensor generates a reading that is sent (as a text message) to the smart home middleware.The middleware logs the ID of the sensor generating the reading together with the date and time of the reading and the state of the sensor.Figure 4 shows a sample of the readings that are generated by one such smart home.
To facilitate control of devices inside Navan, ZigBee light switches are installed to control lights and to the bathroom fan.In addition, custom electrical boxes are designed with ZigBee light switches, The sensors in the smart home are discrete event sensors.When a state change is sensed (e.g., there is motion in the area, a cessation of motion in the area, a significant temperature change, or a change in door status), the sensor generates a reading that is sent (as a text message) to the smart home middleware.The middleware logs the ID of the sensor generating the reading together with the date and time of the reading and the state of the sensor.Figure 4 shows a sample of the readings that are generated by one such smart home.

Activity Awareness
Learning and understanding observed activities is at the center of many fields of study and is essential for CPS such as smart buildings that are sensitive to the needs of the humans they serve.An individual's activities affect that individual, those around him/her, society, and the environment.CPS that operate in real-world complex applications such as building automation require the depth of information that is provided by activity learning algorithms because activity labels and models provide a rich vocabulary for expressing behavior within a system.In the past, theories about behavior and activities were formed based on limited observation.More recently, the maturing of technologies, such as the SHiB, has made it possible to automate activity learning.Learning activities in turn enriches smart homes because the home's intelligent agent can reason at a high level about the resident's activities and take appropriate actions.
In our building automation approach, activity learning plays two roles.First, activity recognition is used to identify activities as they are performed in a smart building environment.Second, activity prediction is used to forecast whether a particular activity will occur within the upcoming time window.Together, they provide a basis for building automation that supports current and upcoming tasks the residents will perform in the building.This section provides details for these two critical components of our activity-aware smart building.
The challenge of activity recognition is to map sensor events to a label that indicates the corresponding activity the individual is performing.There are activity recognition challenges that are unique among machine learning problems.The sequential nature of the input data, the ambiguous partitioning of data into activities, and the overlapping of activity classes mean that additional data processing must be performed.As Figure 6 shows, the recognition steps include collecting and preprocessing sensor data, dividing it into subsequences of manageable size, then extracting subsequence features.The final feature vectors are either labeled by an expert to use as training data or are input to an already-trained model to generate the corresponding activity label.To facilitate control of devices inside Navan, ZigBee light switches are installed to control lights and to the bathroom fan.In addition, custom electrical boxes are designed with ZigBee light switches, as shown in Figure 5, to monitor and control additional devices including reading lamps and speakers.Each light switch reports changes in state of the device as well as button taps and tap counts.These taps provide a mechanism for the resident to provide feedback to the home automation system.In Figure 3, the locations of devices that are controlled by the ZigBee light switches are indicated by the name of each device.All of the indicated devices represent lights or lamps except for F001 (the bathroom fan) and LL014 (the television speakers).

Activity Awareness
Learning and understanding observed activities is at the center of many fields of study and is essential for CPS such as smart buildings that are sensitive to the needs of the humans they serve.An individual's activities affect that individual, those around him/her, society, and the environment.CPS that operate in real-world complex applications such as building automation require the depth of information that is provided by activity learning algorithms because activity labels and models provide a rich vocabulary for expressing behavior within a system.In the past, theories about behavior and activities were formed based on limited observation.More recently, the maturing of technologies, such as the SHiB, has made it possible to automate activity learning.Learning activities in turn enriches smart homes because the home's intelligent agent can reason at a high level about the resident's activities and take appropriate actions.
In our building automation approach, activity learning plays two roles.First, activity recognition is used to identify activities as they are performed in a smart building environment.Second, activity prediction is used to forecast whether a particular activity will occur within the upcoming time window.Together, they provide a basis for building automation that supports current and upcoming tasks the residents will perform in the building.This section provides details for these two critical

Activity Awareness
Learning and understanding observed activities is at the center of many fields of study and is essential for CPS such as smart buildings that are sensitive to the needs of the humans they serve.An individual's activities affect that individual, those around him/her, society, and the environment.CPS that operate in real-world complex applications such as building automation require the depth of information that is provided by activity learning algorithms because activity labels and models provide a rich vocabulary for expressing behavior within a system.In the past, theories about behavior and activities were formed based on limited observation.More recently, the maturing of technologies, such as the SHiB, has made it possible to automate activity learning.Learning activities in turn enriches Energies 2016, 9, 624 6 of 17 smart homes because the home's intelligent agent can reason at a high level about the resident's activities and take appropriate actions.
In our building automation approach, activity learning plays two roles.First, activity recognition is used to identify activities as they are performed in a smart building environment.Second, activity prediction is used to forecast whether a particular activity will occur within the upcoming time window.Together, they provide a basis for building automation that supports current and upcoming tasks the residents will perform in the building.This section provides details for these two critical components of our activity-aware smart building.
The challenge of activity recognition is to map sensor events to a label that indicates the corresponding activity the individual is performing.There are activity recognition challenges that are unique among machine learning problems.The sequential nature of the input data, the ambiguous partitioning of data into activities, and the overlapping of activity classes mean that additional data processing must be performed.As Figure 6 shows, the recognition steps include collecting and preprocessing sensor data, dividing it into subsequences of manageable size, then extracting subsequence features.The final feature vectors are either labeled by an expert to use as training data or are input to an already-trained model to generate the corresponding activity label.are extracted from the raw smart home sensor data.Finally, a supervised machine-learning algorithm learns a mapping from the feature vector X to an activity label.Let A " ta 1 , a 2 , .., a T u be the set of all modeled activities, where a i corresponds to the ith activity class.A smart home generates raw sensor data in the form of time-stamped sensor readings or events, Λ " pλ 1 , λ 2 , .., λ N q, where event λ i corresponds to a sensor reading or sensor value generated at time t i .The data are preprocessed to handle missing or noisy data, then, features x P d are extracted from the raw smart home sensor data.Finally, a supervised machine-learning algorithm learns a mapping from the feature vector X to an activity label.
The home automation approach described in this paper employs our activity recognition algorithm called CASAS-AR [18] to label raw data with corresponding activity labels.While many activity recognition algorithms have been proposed, they are typically designed for constrained situations with pre-segmented data, a single user, and no activity interruptions.CASAS-AR extends this to consider generalization of activity models over multiple smart homes.In earlier work, a common vocabulary of sensor locations was defined to facilitate the design of algorithms that recognize activities even in new environments with no training data.Furthermore, CASAS-AR provides real-time activity labeling on streaming data.To do this, CASAS-AR extracts features from a fixed-sized sliding window of sensor events, λ i . . .λ j , and maps the feature vector onto an activity label, indicating the activity that was performed at the time of the last event in the window, or time t j .
To train AR labels are provided for at least one month of sensor data from each smart building location.Human annotators label the sensor data in each dataset with corresponding activities based upon interviews with the residents, photographs of the home, and a floorplan highlighting the locations of sensors in the space.Sensor events are labeled with the activity that was determined to be occurring in the home at that time.In the experimental validation, 15 core activities are modeled that occur daily in a majority of the datasets.These activities are listed in Table 1.
Sensor events that do not fit into one of the core activity classes are labeled as "Other activity" and provide context for AR as well as for the activity forecaster.To maximize consistency of ground truth labels, multiple annotators look at the datasets and disagreements between labels are resolved via discussion.The annotators demonstrate inter-annotator agreement of κ = 0.85 for our selected activities.The approach is tested in (n = 30) smart homes with multi-year data, achieving >95% recognition accuracy based on 3-fold cross validation assessment with each sensor window treated as a separate data point.
Given the foundation of an activity recognition algorithm, such as CASAS-AR, the CPS can then perform activity prediction.In the context of building automation, activity prediction consists of determining which activities will occur within the next 10 min.Smart home-based activity prediction is a new area in the field and has not been used before this for home automation.This is the goal for building automation in this paper because the home can predict activities rather than just react to them, thereby making the home more efficient in its use of resources such as energy.Specifically, this approach allows the CARL building automation system to avoid turning off devices that are currently in use (as determined by activity recognition) or will soon be in use (determined by activity prediction).In contrast with activity recognition, the activity prediction problem is to determine whether a particular activity will occur within the next time window (here, the size of the time window is 10 min).This is viewed as a binary classification problem.As with activity recognition, the input consists of raw sensor events Λ, the duration of the prediction window, w, and a target activity, a.A feature vector X 1 is extracted from the raw sensor data.Features are extracted for activity recognition (feature vector X) and for activity prediction (feature vector X 1 ).A machine learning algorithm is then used to learn the mapping h: < X 1 , w,a > Ñ {0,1}, from the input feature vector, window duration, and activity to a binary label where 0 indicates that activity a will not occur in the next w time units and 1 indicates that a will occur.The activity prediction problem is formulated and solved in the framework of imitation learning.In traditional imitation learning, the goal of the learner is to learn to imitate the behavior of an expert performing a sequential decision making task (such as playing a game) in a way that generalizes to similar tasks or situations.Imitation learning techniques have been applied to a variety of natural language processing and computer vision prediction tasks [35,72,73].In the activity prediction problem, the expert corresponds to a loss function L and the expert behavior corresponds to predicting the best output at each time step.For each time step the activity prediction algorithm computes the feature vector and the correct activity prediction based on activity labels provided by CASAS-AR.If the algorithm can learn a function h that is consistent with these imitation examples then the learned function will generalize and perform well on new instances [74,75].
In principle, any multi-output regression learner can be used for this task.However, inspired by the binary relevance (BR) classifier for multi-label classification [76], we decompose the multi-output regression problem by learning one regression function for each output variable (in this case, each activity's predicted next occurrence) independently.We have a hard learning problem at hand, which means linear functions will not suffice.We experimented with logistic regression, multi-layer perceptrons, and support vector machine regression.We also tested a standard regression tree, which is a decision tree which decomposes the regression space into subspaces based on values for attributes of the data instances.The specific attributes are chosen based on their ability to reduce the entropy of the data, then the output value is stored at the leaf nodes where the space cannot be further decomposed.The regression tree outperformed the other methods on average.However, like the other approaches, the regression tree could not handle the high variance for some of the activity times.Hence, we finally employed a variant of regression trees called model trees [77], where predictions are made by a learned linear function over all of the features at each leaf node of the decision tree.This performed Energies 2016, 9, 624 9 of 17 consistently the best over the alternative regression methods and is used as the backbone of the CARL home automation algorithm.

Activity-Aware Home Automation
CARL automates control of all devices in a smart building space using the smart building infrastructure described in Section 3 and the activity learning elements described in Section 4. The initial strategy of CARL is to turn off all devices that are not needed in support of the current set of activities as well as those that are anticipated to occur within the next 10 min.
As a preliminary step, CARL identifies the set of devices associated with each activity a P A; these should not be turned off if a is a current or upcoming activity.Here, D a represents devices that are associated with activity a, where D is a subset of the total set of devices, D a Ď D.
Assuming that the device sets have been constructed, CARL then performs a check for conditions to perform device automation at every time step t.The goal is to identify each device that is being used by a current or forthcoming activity, denoted as "CurrentDevices", and turn off every device not in the set "CurrentDevices".CARL must therefore identify the current activities being performed at time t (CurrentActivities) and those that will occur between t and t + 10 min (PredictedActivities).The CASAS-AR activity recognizer is used to determine which activities are current at time t, or "CurrentActivities".CARL's GetPredictions function is used to determine which activities will occur within the next 10 min (between time t and time t + 10 min, or "PredictedActivities").
Finally, a command is sent to turn off the device.In some cases, the smart home resident may want the device remain on after CARL decided it is not needed.User buttons are installed around the home.A double tap on the button indicates that the user is overriding CARL to turn a device back on.A delay is imposed on the corresponding device called DelayTime, during which CARL will not turn off the device.For the experiments in this paper, DelayTime is set to twenty minutes.A summary of the CARL operations is given in Figure 7.
Energies 2016, 9, 624 10 of 16 As a preliminary step, CARL identifies the set of devices associated with each activity A a ; these should not be turned off if a is a current or upcoming activity.Here, Da represents devices that are associated with activity a, where D is a subset of the total set of devices, D D a  .
Assuming that the device sets have been constructed, CARL then performs a check for conditions to perform device automation at every time step t.The goal is to identify each device that is being used by a current or forthcoming activity, denoted as "CurrentDevices", and turn off every device not in the set "CurrentDevices".CARL must therefore identify the current activities being performed at time t (CurrentActivities) and those that will occur between t and t + 10 min (PredictedActivities).The CASAS-AR activity recognizer is used to determine which activities are current at time t, or "CurrentActivities".CARL's GetPredictions function is used to determine which activities will occur within the next 10 min (between time t and time t + 10 min, or "PredictedActivities").
Finally, a command is sent to turn off the device.In some cases, the smart home resident may want the device remain on after CARL decided it is not needed.User buttons are installed around the home.A double tap on the button indicates that the user is overriding CARL to turn a device back on.A delay is imposed on the corresponding device called DelayTime, during which CARL will not turn off the device.For the experiments in this paper, DelayTime is set to twenty minutes.A summary of the CARL operations is given in Figure 7.

Experimental Results
CARL's goal is to turn off as many as devices as possible without interfering with resident tasks.Performance can thus be measured in terms of the number of times a device is turned off (or the corresponding reduction in energy consumption) and the number of resident disruptions (or the number of times the user double tapped a switch in order to provide feedback to the system while

Experimental Results
CARL's goal is to turn off as many as devices as possible without interfering with resident tasks.Performance can thus be measured in terms of the number of times a device is turned off (or the corresponding reduction in energy consumption) and the number of resident disruptions (or the number of times the user double tapped a switch in order to provide feedback to the system while turning the device back on. Here, the CARL activity-aware automation architecture is validated using data collected from the Navan smart home described in Section 3.Because much of the system depends on the ability to correctly recognize activities in real time as they occur, first the accuracy of the CASAS-AR activity recognizer is evaluated on the Navan smart home data.Table 2 summarizes the performance of CASAS-AR using 3-fold cross validation.Table 2 also provides a confusion matrix that highlights where the errors lie.The performance evaluation is based on two months of smart home data, collected continuously while the resident performed normal routines.As can be seen in Table 2, the overall accuracy is high but the larger classes, such as "Other activity", create a class imbalance that introduces associated errors.CARL's automation effectiveness builds on this performance because CASAS-AR provides the set of current activities, CurrentActivities, used in Figure 8.The next component is CARL's activity prediction.This is a binary classification problem, indicating for each activity whether it will occur in the next ten minutes (class = Yes) or not (class = No).While activity prediction in this case is expected to outperform activity recognition because there are fewer classes, this is not always the case.Table 3 summarizes the three-fold cross validation results of activity prediction for the Navan smart home.As can be seen in Table 3, activity prediction performance varies greatly between particular activities.Activities that occur often have enough training data to adequately learn the activity times.Activities that are highly predictable, such as sleep, also yield strong predictive accuracy.On the other hands, activities that are less predictable and less frequent have lower accuracy.An additional challenge is the extremely class imbalance in this learning problem.Most activities are not current much more than they are current.For any given activity it is expected that it will be current only 1.0/|A| of the time.Because there are 15 activities, the average time that an activity will occur is 0.067 of the time.Machine learning algorithms attempt to optimize classification accuracy.For imbalanced class distributions this means that most of the predictions will favor the majority class (the activity will not occur within the next 10 min) rather than the minority class (the activity will occur within the next 10 min).These influences are reflected in the results shown in Table 3.The next component is CARL's activity prediction.This is a binary classification problem, indicating for each activity whether it will occur in the next ten minutes (class = Yes) or not (class = No).While activity prediction in this case is expected to outperform activity recognition because there are fewer classes, this is not always the case.Table 3 summarizes the three-fold cross validation results of activity prediction for the Navan smart home.As can be seen in Table 3, activity prediction performance varies greatly between particular activities.Activities that occur often have enough training data to adequately learn the activity times.Activities that are highly predictable, such as sleep, also yield strong predictive accuracy.On the other hands, activities that are less predictable and less frequent have lower accuracy.An additional challenge is the extremely class imbalance in this learning problem.Most activities are not current much more than they are current.For any given activity it is expected that it will be current only 1.0/|A| of the time.Because there are 15 activities, the average time that an activity will occur is 0.067 of the time.Machine learning algorithms attempt to optimize classification accuracy.For imbalanced class distributions this means that most of the predictions will favor the majority class (the activity will not occur within the next 10 min) rather than the minority class (the activity will occur within the next 10 min).These influences are reflected in the results shown in Table 3.Finally, CARL is tested as a fully automated home control system in our Navan smart apartment.To do this, the CASAS-AR activity recognition algorithm and the CARL activity prediction algorithm are trained on three months of data with activity labels provided by a human annotator.The automation results are then collected for one week in the apartment.The training data and testing data were separated in time by several months, during which some routine changes would be expected due to concept draft, seasonal changes, and normal behavior variation.
Anecdotal information from the resident indicated that many of the activities were correctly detected, anticipated, and automated.However, the ones that were incorrect were often detrimental to the resident's comfort.In terms of quantifiable performance evaluation, two measures are used.The first is the number of "double button taps" performed by the resident.These represent false positive cases where CARL turned off a device at a time that was incorrect or inconvenient for the resident and the resident indicate the mistake by tapping the feedback button twice.The resident was at home almost the entire duration of the test week.However, during the times that he was out of the home he provided feedback by looking at the automation and sensor data logs to assess whether each automation step was appropriate or was incorrect.
Table 4 summarizes the performance of CARL in terms of its ability to accurately turn off devices when they are not needed.As the table indicates, not only does performance vary greatly from one device to another but it closely mirrors the activity recognition and activity prediction performance.As an example, Work is an activity with consistent recognition and prediction performance.Similarly, device LL015 is automated with strong true positive rates (TPR) and false negative rates (FNR).This indicates that as the ability of CASAS-AR and CARL's activity predictor improves, so will the ability to accurately automate home control.This can be accomplished through additional training data and greater consistency of human activity label annotations.Finally, Figures 8 and 9 show the minutes saved and energy reduced through CARL automation.Using activity-aware automation reduces device utilization by 56% and reduces energy consumption by 50%.Of course, this savings must be balanced with the 21% average positive rate.In some of these cases, the correct automation step was determined but was not executed at an optimal time.To analyze this type of error more carefully, was also compute the normalized root mean squared error (nRMSE) for each CARL-based device automation.Error is computed as the time between the device automation and when it should have been turned off based on the actual activities that occurred at each time step.Each error value is squared and the set of errors are normalized to fall within the range of 0-1.The nRMSE over the entire dataset is 0.138577.This indicates that CARL is able to automate devices based on its awareness of activities that are occurring in the home.
Using activity-aware automation reduces device utilization by 56% and reduces energy consumption by 50%.Of course, this savings must be balanced with the 21% average positive rate.In some of these cases, the correct automation step was determined but was not executed at an optimal time.To analyze this type of error more carefully, was also compute the normalized root mean squared error (nRMSE) for each CARL-based device automation.Error is computed as the time between the device automation and when it should have been turned off based on the actual activities that occurred at each time step.Each error value is squared and the set of errors are normalized to fall within the range of 0-1.The nRMSE over the entire dataset is 0.138577.This indicates that CARL is able to automate devices based on its awareness of activities that are occurring in the home.

Conclusions
This paper introduces the notion of an activity-aware building automation system.Such a system can be used as a critical component of smart city automation to energy consumption while supporting routine activities.Our activity-aware automation system, CARL, uses activity recognition to identify current activities and activity prediction to anticipate upcoming activities.Both sources of information are utilized to make decisions regarding devices to turn off, thus reducing energy consumption.Experiments run on an actual smart apartment indicate that CARL is aware of resident activities and is able to automate home devices based on this information in a way that reduces resource consumption.
There are many directions to consider for future work.A first step will be to use CARL to, not only turn off devices that are not currently needed in support of current activities, but to also automatically turn on devices that are needed.In addition, CARL's will be enhanced by segmenting and smoothing activities, reducing the amount of jitter in activity labels and improving activity prediction performance.Finally, CARL will be evaluated in a greater number of automated buildings to show the combined energy reduction that can be realized using an activity-aware approach to designing smart buildings and smart cities.

Figure 1 .
Figure 1.The physical system (sensors, home) work together with humans and computational components (activity learning) to provide activity-aware automation.

Figure 2 .
Figure 2. (a) Smart home in a box; (b) smart apartment; and (c) activity graph.

Figure 1 .
Figure 1.The physical system (sensors, home) work together with humans and computational components (activity learning) to provide activity-aware automation.

Figure 1 .
Figure 1.The physical system (sensors, home) work together with humans and computational components (activity learning) to provide activity-aware automation.

Figure 2 .
Figure 2. (a) Smart home in a box; (b) smart apartment; and (c) activity graph.Figure 2. (a) Smart home in a box; (b) smart apartment; and (c) activity graph.

Figure 2 .
Figure 2. (a) Smart home in a box; (b) smart apartment; and (c) activity graph.Figure 2. (a) Smart home in a box; (b) smart apartment; and (c) activity graph.

Figure 4 .
Figure 4. Example text-based sensor data.Sensor IDs starting with M are motion sensors and IDs starting with D are door sensors.Sensor M012 is located near the external door, as is sensor D004.Sensor M013 is located in the kitchen.

Figure 5 .
Figure 5.A ZigBee light switch is used to control devices and provide user feedback.

Figure 4 .
Figure 4. Example text-based sensor data.Sensor IDs starting with M are motion sensors and IDs starting with D are door sensors.Sensor M012 is located near the external door, as is sensor D004.Sensor M013 is located in the kitchen.

Figure 4 .
Figure 4. Example text-based sensor data.Sensor IDs starting with M are motion sensors and IDs starting with D are door sensors.Sensor M012 is located near the external door, as is sensor D004.Sensor M013 is located in the kitchen.

Figure 5 .
Figure 5.A ZigBee light switch is used to control devices and provide user feedback.

Figure 5 .
Figure 5.A ZigBee light switch is used to control devices and provide user feedback.

Figure 6 .
Figure 6.Activity recognition includes stages of raw sensor data collection, data preprocessing and segmentation, feature extraction, and supervised machine learning.

Figure 6 .
Figure 6.Activity recognition includes stages of raw sensor data collection, data preprocessing and segmentation, feature extraction, and supervised machine learning.

Figure 8 .
Figure 8. Minutes each device is on using the baseline method (no automation) and using CARL.

Figure 8 .
Figure 8. Minutes each device is on using the baseline method (no automation) and using CARL.

Figure 9 .
Figure 9. Energy consumed using the baseline method (no automation) and using CARL.Figure 9. Energy consumed using the baseline method (no automation) and using CARL.

Figure 9 .
Figure 9. Energy consumed using the baseline method (no automation) and using CARL.Figure 9. Energy consumed using the baseline method (no automation) and using CARL.

Table 3 .
Activity occurrence prediction performance on Navan smart home data.

Table 3 .
Activity occurrence prediction performance on Navan smart home data.

Table 4 .
Activity occurrence prediction performance on Navan smart home data by device.Device placement is shown in Figure3.TRP: true positive rates; and FNR: false negative rates.