Analysis of Applications to Improve the Energy Savings in Residential Buildings Based on Systemic Quality Model

Abstract

Creating a definition of the features and the architecture of a new Energy Management Software (EMS) is complex because different professionals will be involved in creating that definition and in using the tool. To simplify this definition and aid in the eventual selection of an existing EMS to fit a specific need, a set of metrics that considers the primary issues and drawbacks of the EMS is decisive. This study proposes a set of metrics to evaluate and compare EMS applications. Using these metrics will allow professionals to highlight the tendencies and detect the drawbacks of current EMS applications and to eventually develop new EMS applications based on the results of the analysis. This study presents a list of the applications to be examined and describes the primary issues to be considered in the development of a new application. This study follows the Systemic Quality Model (SQMO), which has been used as a starting point to develop new EMS, but can also be used to select an existing EMS that fits the goals of a company. Using this type of analysis, we were able to detect the primary features desired in an EMS software. These features are numerically scaled, allowing professionals to select the most appropriate EMS that fits for their purposes. This allows the development of EMS utilizing an iterative and user-centric approach. We can apply this methodology to guide the development of future EMS and to define the priorities that are desired in this type of software.

1. Introduction

The topic of saving energy in regards to protecting the environment was first discussed in 1980 when the term “sustainable environment” was officially mentioned by the International Union for Conservation of Nature in their World Conservation Strategy report [1]. Discussions regarding this topic include energy production technologies and regulations regarding the use of energy. The idea of changing the behavior of energy consumers has become more popular in the last 10 years. Moreover, the development of Energy Management Systems (EMS) for domestic users was accelerated by the first trials in this field of well-known international companies that occurred approximately five years ago; incidentally, the trials were unsuccessful, and the projects were terminated [2,3]. Therefore, this market is quite unique and unstructured.

The development of the Eco-Management and Audit Scheme (EMAS) in 1993 by the European Commission and the need for certification compels enterprises and end users to utilize environmental policy instruments such as EMS. However, it is unclear that the use and implementation of EMS implies adaptation of the EMAS or leads to environmental performance improvement. A comprehensive analysis is provided in [4]. A specific analysis of an Italian organization is provided in [5], which highlights critical issues in EMAS implementation.

Certain analyses have focused on the process of selecting enterprise resource planning software [6,7] or selecting electronic medical record [8], but this type of analysis has not yet been conducted specifically for EMS.

In our approach, we are primarily focused on the issues related to the adoption of EMS, as noted in [4], and we are specifically focused on defining the aspects (using our proposed methodology) that allowed us to obtain accurate data from the user. Providing accurate information may help to modify the behavior of the user [9,10,11,12,13]. The proposed methodology has been used on an industrial project that developed the definition of ACE (an intelligent management system for energy efficient buildings user behavior) [14]. A snapshot of this software is provided in Figure 1.

In this Study, we focus on the methodology to determine primary metrics to be considered in the implementation (or selection) of EMS.

2. Information Quality

Information regarding the consumption of energy is required for the development of a precise energy simulation. Therefore, the information we acquire directly relates to the amount of information the simulation produces; greater amounts of information improve the quality of the results, improving the results also improves the quality of the information. However, on the schema is missing one of the most important components of the system, the users. Due to the project nature, the most challenging goal is to determine the balance between game and simulation; a user-friendly interface and representativeness of the application will more likely engage users and provide the needed information.

Users, as represented in Figure 2, are the uniting factor between information and the simulation model to be developed.

Evaluation is divided into four major components. The primary objective is to be able to evaluate the correlation between the information necessary for simulation and the interest of the user (purple arrow) to provide this information. A secondary goal is to find an alignment between the goals of the application and the goals of the users (blue arrows) to evaluate the technical aspects of the simulation (Figure 3).

Persuading homeowners to conserve energy is a difficult goal to achieve. However, energy use can be changed over time by applying governmental support, such as Eco-taxes, or emissions trading [15]. Ecological economics, which is defined by its focus on nature, justice, and time, has the same goal of conserving energy. Issues related to intergenerational equity, irreversibility of environmental changes, uncertainty of long-term outcomes and sustainable development guide both ecological and economic analysis and valuation [16].

For example, in June 2011, the US government launched the Green Button Program [17], which allows all households to download metered data regarding electricity consumption. In addition, the US launched the American Energy Data Challenge [18] to encourage the development of applications that provide energy consumption data in a user-friendly manner.

The challenge is to involve society in the process of conserving energy and to gradually alter individual’s habits using a comprehensive approach. Therefore, an application that provides information to a household regarding where and how they may conserve energy, explains the importance of energy conservation and demonstrates how improvement of domestic electricity devices impacts the environment; such an application could lead to significant changes in energy conservation strategies. Consequently, there is a keen interest in the development of applications like these.

Environmental sustainability for governments is important for both philanthropic reasons and economic stability. The primary goal of environmental companies is to conserve energy resources, decrease carbon emissions and reduce environmental damage. Regarding this issue, building owners assume that reducing energy related costs is a key factor. Finally, application developers view this dilemma as an interesting puzzle to solve, which refers to the Internet of Things (IoT) paradigm. Nevertheless, during the process of application development, every participant seeks information regarding how he/she will profit, what information is necessary, and how to change an individual’s habits; however, occasionally, the needs of the user are not considered. What can persuade individuals to provide information regarding their consumption habits? What information is needed to use the application?

To provide users with necessary information in a convenient manner, we have developed a set of metrics taking into account that the application evaluation should be quick and simple. This gives us an understanding of what type of information facilities will be provided, through use of the application. The process should be precise enough to evaluate the application and compare it with other applications.

The methodology of our evaluation is based on the study, “A discrete-event simulation and continuous software evaluation on a systemic quality model: An oil industry case” [19]. However, this study is focused on metrics used to evaluate the features of an application for the proposed industrial case. To develop a set of metrics, we first examined prevailing tendencies of the market. In the following chapters, we explain why prevailing tendencies include behavioral science and gamification theories and how these improve EMS and provide opportunities for scholars and energy companies to enhance environmental sustainability.

3. Tendencies of the EMS Market

It is important to analyze the tendencies of the market for EMS to avoid reinventing the wheel, thus an effective research can lead to new ideas. The primary aspects considered during EMS development are the following.

Simplicity: There should not be difficult registration forms or fees. Generally, households are not ready to spend their time or money on such applications. The application should come from an energy company as a default, but must include opt-out measures [2]. The application should avoid any association with the energy company because of distrust of energy companies [20], which is a sensitive problem, since in some markets, like in the US market, the energy companies are providing these data. Normative related to Open Data could help in order to democratize this information, maybe making also the data easy to understand and user-friendly.

Interaction: This aspect is primarily defined by competition or cooperation. (i) Competition: the homeowner can compare his/her energy usage to that of friends from social networks or the average user. Certain filters are available that will compare usage by the type of building, the type of apartments, etc.; (ii) Collaboration: users can offer tips to other users to save energy; users can view and save other users’ tips if they choose to do so.

Motivation: The primary motivation is real economic savings. Later, public or private institutions can motivate users by promoting savings and offering compensation based on significant discounts in shops, among other measures. Energy savings will benefit both the client, the company or the building manager, and the user of the platform, usually the resident of the building, because of a reduction in emissions (equivalent Kg CO₂/KWh). The disastrous effects of current emissions from residential housing are incurred by society as a whole [21]. Therefore, in this sense, public institutions are interested in providing incentives so that users receive compensation and thus participate more actively. To improve participation, one proposal explored in the development of ACE is that users can earn “coins” if they propose popular tips. Later, these “coins” may be used as discounts in stores. This type of compensation excites people and gets them involved in competition to conserve energy. However, to apply these characteristics effectively, we sought assistance from the fields of behavioral science and gamification.

3.1. Behavioral Science

Certain global players in the EMS market that seek to reduce the cost of energy production use behavioral science methods to engage customers in saving energy [22]. The systematic analysis and research of human behavior permits them to analyze customers’ behavior in regards to the information that is provided and the feedback received.

For instance, the OPOWER Home Energy Report [23] outlines two key features: (i) “descriptive norm” that indicates where consumers rate in energy consumption relative to the average consumer and their most efficient peers; and (ii) the “injunctive norm” (provided in the “How you’re doing” box, in this case a smiley face) that provides a social appraisal of the household’s relative performance. However, according to the book “Yes! 50 scientifically proven ways to be persuasive” [24], providing feedback to a consumer can result in a “rebound effect” that leads to an increase in energy consumption by low energy consumers due to social norms, or this method may encourage energy-efficient users to continue to outperform average consumers [25].

The most important aspects related to EMS development include the following:

(1)

Demonstrate to the user that they care about energy conservation. This is referred to as a foot-in-the-door technique. Once an individual states that he/she cares about sustainability and wants to help save energy, it will be easier to involve him/her in future participation because of a commitment to energy conservation. People want to show consistency because it is highly valued in our culture [26].

(2)

Involve users in competition. Demonstrate that others use the application and save energy. Demonstrate that someone in a nearby neighborhood saves more energy (and money) and explain how this success was achieved. The competition will last for a finite time to keep the users interests, and propose new challenges to engage new and existing users.

(3)

Teach appreciation. Provide feedback on successes and failures; provide simple tips to improve results.

(4)

Separate information into easy-to-understand and complete segments; do not ask users to provide too much data at one time, but gradually prompt users for information [24].

These are the most important aspects to begin, but not the most unique aspects. Next, we describe how a powerful technique, gamification, may be used to enhance the user experience.

3.2. Gamification

“We always learn better when the experience is FUN” states An Coppens, Chief Game Changer at Gamification Nation Ltd. [27]. A common use of gamification is for serious projects that are simulations of real world events or processes developed to solve a problem [28]. The primary objectives of serious games (not taking entertainment into account) are generally to train or educate users. Gamification may be used to engage users in saving energy without overloading them with large amounts of information. Gamification must not take the user’s time if she or he does not want to be involved in such game, and only want to use the platform to improve the energy savings. When participating in a game, the user could receive useful tips regarding how to conserve energy. The gamified process has two primary objectives: (i) to obtain information about users’ habits and to provide a closer-to-reality simulation for scientific needs; and (ii) to provide information to a user that demonstrates how to use energy efficiently. To engage a person in the interaction with a gamified application, it must include the following four characteristics [29]:

(1)

Challenge: There are two obvious reasons to manage energy, to spend less money on energy bills and, if the user is concerned about the environment, to waste less energy. However, based on the first rule of persuasion, other people’s behavior is the most powerful source of social influence [24]. For instance, competition could be used to make EMS challenging.

(2)

Curiosity: If the information provided is based on reality, is well-explained and allows relationships and connections that are not obvious to an average customer, this information motivates him/her to ask “what else do they have for me?”

(3)

Control: If an individual can make changes, see the difference and understand how decisions changed the situation in the application, he/she may attempt to apply these changes in real life to determine if it results in any actual change in energy bills. Finally, the individual is not only engaged throughout the learning process but also contributes to the acquisition of new knowledge and applies the experience to real-life situations [30].

(4)

Imagination: Some goals can be more difficult; the customer can use his/her imagination to achieve some specific goals. This prevents the users from becoming bored with the application too quickly.

Next, we review the proposed metrics to evaluate this type of software.

4. SQMO

The Systemic Quality Model (SQMO) presented in [19] is used to determine optimal features to be considered when developing EMS. In this section we describe the methodology.

The SQMO was developed in 2001 by the Universidad Simón Bolívar (Venezuela) [19]. Several successful applications of this methodology have demonstrated that it may be successfully used to estimate the quality of software development [31]. This methodology is proposed to evaluate discrete simulation software and, in our case, fits well because EMS are systems that behave similar to discrete simulation systems. This model interacts with users in real time to fulfill a specific goal: in this case, to improve energy savings. SQMO may be used to evaluate fully developed software or to evaluate the development of software. In our case, the focus is on evaluation of fully developed software.

The different levels of the SQMO are presented in the following sections. For a complete review of the methodology [19] can be reviewed.

4.1. Level 0: Dimensions

The dimension level considers not only the product (the software) but also the process utilized to develop and implement this software in an organization. In this study, we do not consider this process, but are more focused on the software and analyzing the primary features that are necessary for EMS. The two dimensions of the product are efficiency and effectiveness, where “Efficiency involves computation of the relation between the quantity obtained and the quantity of resources used. However, the effectiveness measures the ratio between the obtained and the obtainable quantities” [19].

For this analysis, we focus on the effectiveness of the EMS and what we can achieve considering existing features.

4.2. Level 1: Categories

The categories for the sub-model Process are the following [19]:

(1)

Client-Supplier includes processes that have an impact on the client, support the development and transition of the software to the client and provide information regarding the correct operation and use of the software product or service.

(2)

Engineering consists of processes that directly specify, implement or maintain the software product, explain its relationship to the system and provide related documentation.

(3)

Support consists of processes that can be used by any of the processes (including support ones) at several levels of the acquisition life cycle.

(4)

Management consists of processes that contain practices of a generic nature that may be used by managers of the project or process, within a primary life cycle.

(5)

Organizational contains processes that establish the organization’s commercial goals and develops a process, a product and good resources (values) that will help the organization attain the goals of the project.

Because we do not consider the process of implementing the EMS, we will focus on the product dimension. For this dimension, we are focused on the analysis of three primary categories that help to achieve the highest level of effectiveness possible. For the product dimension, these three categories include [19]:

(1)

Functionality analyzes if features implemented in the EMS are sufficient for the desired purpose.

(2)

Usability analyzes if features are sufficiently usable and considers that the target public of an EMS is not generally comprised of experts regarding energy management systems.

(3)

Efficiency analyzes if the time needed to perform the actions are sufficient to maintain the interest of users.

Three additional categories are defined regarding the methodology and include [19]:

(1)

Reliability: The capacity of a software product to maintain a specified level of performance when used under specific conditions.

(2)

Maintainability: The capacity of the software to be modified. Modifications may include corrections, improvements or adaptations to the software and adjust to changes in the environment in terms of functional requirements and specifications.

(3)

Portability: The capacity of the software product to be transferred from one environment to another.

For this analysis, we will not consider these additional categories.

4.3. Level 2: Characteristics

A set of characteristics for each category provides a clear definition of the features we hope to analyze.

Table 1. Main characteristics for the Functionality, Usability and Efficiency categories of the product dimension [19].

Category	Characteristics
Functionality	Fit to Purpose (FPU)
	Interoperability (INT)
	Security (SEC)
Usability	Easy of Understanding and Learning (EUL)
	Graphical Interface (GIN)
	Operability (OPR)
Efficiency	Execution Performance (EPE)
	Resource Utilization (RUT)

displays the main characteristics for the Functionality, Usability and Efficiency categories based on [19].

4.4. Level 3: Metrics

Finally, once the different characteristics are defined, the behavior of the software can be detailed through the use of a set of metrics. These metrics allow the measurement of the effectiveness of the EMS depending on the criteria selected.

Figure 4 represents the overall structure of the SQMO approach.

To apply the proposed methodology, it is necessary to define metrics that will allow us to determine features that a specific EMS should incorporate (or not). Furthermore, it is necessary to define existing EMS available in the market. In future sections, we detail the application of this methodology including a description of the metrics used and the list of EMS analyzed.

4.5. Lists

To determine the software to be considered in this evaluation, the methodology proposed in [32] was used, and we organized the software by collating three lists.

The long list refers to the first list to be used; here, we describe all existing software on the market that fits in the category of the type software that we are analyzing. This list is used as a starting point to evaluate the state-of-the-art subject area. Once this list was detailed, we continued with a medium list that included software from the long list that aligned with the general objectives of our selection criteria. This medium list refined the analysis and included software that may have not been the best options for our goals, but was sufficient to be considered.

Finally, a short list was developed that included only software that met mandatory metrics. Mandatory metrics are metrics that must be accomplished (with a certain level) to assure that the selected software may be used according to our purpose. This short list includes software that must be evaluated and considered as candidates for the final selection process, which is completed through the use of the developed metrics.

Using the metrics, we were able to produce a set of questionnaires that included one set for each of the software applications on the short list and were able to quantitatively evaluate the software.

5. Metrics

To develop the metrics, the criteria of evaluation presented in Figure 5 were taken into consideration.

(1)

In regards to the interface and logic of the application, the metrics must be able to evaluate simulation and educational features.

(2)

The metrics must evaluate technical features.

(3)

The metrics should be divided in two parts for evaluation by software developers and end users because certain metrics may be clear to a technical user but may be unclear for an end user. The developer evaluates both physical and logical features, but the end user only evaluates logical features.

As a basis for logical metrics in regards to the educational component [30] and to avoid overloading the process of evaluation, we selected only the most important features that relate to our criteria.

6. Long List of Energy-Related Applications

The SQMO methodology was used to develop a list of software to be evaluated. This began with a long list that represented all existing software currently available on the market (for this specific purpose).

This long list may be useful to better understand tendencies of this type of software. The medium and short lists may be used to better understand competition if we later desire to develop new EMS, or these lists may be used to select appropriate EMS for specific purposes [19]. Applications were searched on the Internet using English, Spanish and Russian because we are mainly focused on Spanish and Russian markets. The sampling does not demonstrate the worldwide tendency in this field, but is quite representative because English is often used as an international language and allowed us to divide the software by types (see

Table 2. Application types depending on the nature of the information provided and source of information used. Often, these applications are used only to give information to building managers and building residents.

Type	Description
Game	Is not usually connected with the reality, gives the idea how to spend energy, advices in a form of a play.
Energy Management tool with the smart meter	Usage of smart meter allows to obtain quite informative real-time or near real-time data and then watch the changes in the application.
Power Management entering invoice data	The information added by you or provided by Energy Company automatically. You can only see the difference by period.
Energy costs simulator	Simulates your expenses based on comprehensive data you entered about your home.
Emanation of carbon simulation	Simulates the emission of carbon you produce based on energy used by your home and your lifestyle.

).

The types of applications were merged to provide a clear picture of energy consumption and homeowners’ habits. In addition, to encourage the user to share information about his/her habits, it is very important to provide useful information to ensure that the application serves the user’s needs. The long list is presented in

Table 3. Long list of energy-related applications. Not all applications are EMS; those that are EMS are included on the short list. Although certain systems are not currently operative (primarily due to the lack of user feedback), their features are interesting to understand why the user feedback is poor. The description provided is obtained, if possible, from the website, and is intended to understand the aim of the application.

Project	Operative	Company	Description
ACE [33]	Yes	Universitat Politècnica de Catalunya—BarcelonaTech, VIAS, Lavola	The web application is based on the characteristics of the building and the resident use, the system provides a set of recommendations for reducing energy expenditure and consumption associated to CO2.
Boltio [34]	Yes	Kinética Mobile	It is a simple application that shows the price of kilowatt per hour. Thus, we can choose the right time to turn certain appliances and save on the bill.
Carbon Footprint Calculator [35]	Yes	TerraPass	Carbon footprint calculator for individuals and households.
Control4 [36]	Yes	Control4	A home automation system from with the aim to turn your home into a smart home.
Online Conversion [37]	Yes	Robert Fogt.	A-conversion tool for energy units.
DEXCell [38]	Yes	Dexmatech	Software-as-a-Service platform to reduce energy use, through analysis, alarms and recommendations, compatible with most meters, BMS systems and other devices
Drive Mobile App [39]	Yes	Holland Wood, Derek Gabriel, Matthew Ing, and Rod Hinman	The DRIVE™ System is a patent-pending solution that provides incentives to residential customers to reduce power consumption during peak and critical peak times.
E4RSIM [40]	No	E4RSIM	A international project from 2011 to 2013 that develops a tool that allows to analyze the energy consumption of a building, with the aim to improve the rehabilitation.
EcoFactor [41]	Yes	EcoFactor	Ecofactor provides automated energy savings, comfort and control through energy efficiency, demand response and HVAC performance monitoring services.
Energy Cost Calculator [42]	Yes	Vpugazhenthi	Calculates the Operating cost and Energy Usage of Electric Equipment or Machinery.
Energy Tracker [43]	Yes	iOS Apps Austria	Application that allows to track of how much energy are used on average. This allows to estimate the bill.
Etres Consultores [44]	Yes	Ahorra tu energía	Web application that allows to generate an automatic report for your building.
GEMS [45]	Yes	Green Impact Campaign	The GEMS application allows student volunteers to survey building systems using an easy to understand, self-training guide for small businesses in their community.
Greenbutton [46]	Yes	Green Button Alliance, Inc.	Green Button is a secure way to get your energy usage information electronically.
GoodCoins [47]	Yes	Zerofootprint	A social currency that engages communities and drives sustainable change. Essentially, it rewards good behavior with good things.
GreenQuest [48]	Yes	EnergyCap	A personal energy efficiency manager.
Green Outlet [49]	No	iOS iPhon Utilities	Application with the aim to help the user in the reduction of electricity use.
Hog Busters Energy Hogs [50]	Yes	Alliance to Save Energy	The Energy Hog Challenge is a set of classroom activities that guide children through lessons about different sources of energy, how we use energy at home, and how to bust energy hogs to save energy.
Hohm [51]	No	Microsoft	Microsoft Hohm was an online web application by Microsoft that enables consumers to analyze their energy usage and provides energy saving recommendations.
iControl [52]	Yes	iControl Networks	Smart Home devices and solutions.
JoulBug [53]	Yes	Cleanbit Systems	JouleBug is the easy way to make your everyday habits more sustainable, at home, work, and play. Discover how you and your friends can use resources—without using them up.
Kids Energy Zone [54]	Yes	Touchstone Energy	A fully integrated energy education campaign that includes web-based lessons, web activities and games, and printed materials. The program teaches children about energy, electrical safety, energy efficiency and renewable energy.
Kill-Ur-Watts [55]	Yes	KeyLogic Systems	Kill-Ur-Watts is an iPhone/iPad/iPod Touch application that uses your Green Button data provided by your electric utility provider. The application will calculate your annual, monthly, daily or hourly usage and display graphical representation of each.
Leafully [56]	Yes	Trick Shot Studios	Leafully recognizes that energy usage is more than just electricity usage and thus tries to give the user a total tree footprint—the amount of trees needed to offset the pollution created by one’s energy consumption.
Luz + Precio [57]	Yes	the3devs	Gives recommendations of when are the best times to consume electricity in Spain.
Melon [58]	Yes	Wegowise	Benchmarking tool for the buildings energy and water usage.
Meter Readings [59]	Yes	Graham Haley	Application to read meters (electricity, gas, water, …) on the phone.
My CO2 Calculator [60]	Yes	Zero Above Ltd.	Allows to quantify your effect on the environment whilst at work, home and while travelling.
NECADA [61,62]	Yes	Universitat Politècnica de Catalunya—BarcelonaTech	Urban area and building co-simulator capable to find the optimal parameters to improve sustainability.
Nest [63]	Yes	Nest Labs	A home automation company that builds programmable, self-learning thermostats as well as builds smoke detectors, security cameras and other systems.
Ollie’s club Energy Saving [64]	No	Sustain Ability International	A game with the goal to reduce energy in a house and make the energy usage meter go down.
OPOWER [65]	Yes	ORACLE	A set of solutions to achieve an efficiency use of the energy.
PowerMeter [66]	No	Google	Google PowerMeter was a software project of Google’s philanthropic arm, Google.org, to help consumers track their home electricity usage.
Precio de la Luz [67]	Yes	Neapp Soft	A simple application to know in time real the cost of the energy electric that is consumed in your home or place of work in Spain.
Precio Luz [68]	Yes	redpolas	With this application you can access to the ranking of the energy auction that will be applied to the current and the next day in Spain.
EnerByte [69]	Yes	Enerbyte	Virtual Energy Advisor guides you to be more efficient, according to your consumption profile, your behavior, and your motivation.
Standby Energy Cost Calculator [70]	Yes	EmpMobile	Calculate how much money you are spending just by leaving common electronics plugged in.
Tarifazo [71]	Yes	AppEventos Mobile S.L.	Save energy and reduce the bill applying the new Spain electric rates.
Tendril [72]	Yes	Tendril	A software platform using behavioral science and energy information to improve how products are marketed and consumed.
The Green Deal App [73]	No	Green Deal Group	The Green Deal UK program helps you make energy-saving improvements to your home and find the best way to pay for them.
Wiser EMS [74]	Yes	Schneider Electric’s	The Energy Monitor Pro system, a comprehensive demand management solution for utilities and consumers, allows homeowners to reduce or shift energy use during peak times and helps electricity providers improve grid efficiency and network reliability.
Wotz [75]	Yes	Students work	A game using GreenButton data to represent the amount of energy used on a building.

.

In addition, it is necessary to propose a set of metrics that can analyze the software.

Table 4. Logical metrics to evaluate application features. In parenthesis is represented the characteristic of Functionality, “Fit to Purpose” (FPU), or the Usability characteristic, “Easy of Understanding and Learning” (EUL), where this metric can be placed depending on the final purpose of our EMS.

Criterion	Description and Importance to the Project	Scale
1. Content (FPU)
1.1. Presentation	This is one of the most important blocks because we must balance the information we give to user that should not be obvious, avoiding overwhelming the user with tons of information.	1: Information is poorly structured
		3: Some blocks of information are presented better than others
		5: Well-structured information
1.2. Accuracy		1: The information is not reliable
		3: The information is not totally reliable
		5: The information is reliable
1.3. Relevance		1: High
		3: Medium
		5: Low
1.4. Connection to the learning objectives		1: High
		3: Medium
		5: Low
1.5. Adequacy for the consumer		1: Does not meet customer’s expectations
		3: Moderately satisfied with the application
		5: Totally satisfied
2. Control (FPU)	The ability of user to personalize settings by his needs. The possibility of adjusting content and settings to meet specific needs of the customer.	1: Low ability of control
		3: Medium
		5: High
3. Meaningful Feedback (FPU)	Important criterion to provide customers understanding. Good feedbacks can change customer’s habits.	1: Low useful feedback
		3: Medium
		5: High
4. High-order thinking skills (FPU)	Implies customer’s curiosity and imagination. Shows the development of knowledge.	1: The tasks are too easy/not interesting
		2: The tasks are too difficult/the user loses interest
		3: The tasks have medium difficulty but the user is not captivated
		4: The tasks are quite interesting but not difficult enough
		5: Tasks are difficult and very interesting
5. Usability and technical performance (EUL)
5.1. Learnability	How easy is it for users to accomplish basic tasks the first time they encounter the design?	1: Difficult
		3: Medium
		5: Easy
5.2. Efficiency	Once users have learned the design, how quickly can they perform tasks?	1: It takes a lot of time
		3: It does not take a lot of time
		5: It is quick
5.3. Memorability	When users return to the design after a period of not using it, how easily can they re-establish proficiency?	1: It takes a lot of time
		3: It takes some time
		5: It is easy
5.4. Errors	How many errors do users make, how severe are these errors, and how easily can they recover from the errors?	1: Many severe errors, difficult to recover from them
		2: Low amount of severe errors, difficult to recover from them
		3: A lot of insignificant errors
		4: Low amount of insignificant errors, difficult to recover
		5: Low amount of insignificant errors, easy to recover
5.5. Satisfaction	How pleasant is it to use the design?	1: There is no comparison
		3: Comparison with ideal situation or anonymous users only
		5: Specific Users (friends in social networks)
6. Interactivity and engagement (EUL)	Does the application show the information or require the user to participate in order to see this information?	1: Low level of interactivity and engagement
		2: Low level of interactivity and medium level of engagement
		3: Medium level of interactivity and low level of engagement
		4: More than a medium level of interactivity and engagement
		5: High level of interactivity and engagement
7. Type of comparison (EUL)	If there is any way comparison of to others.	1: There is no comparison
		3: Comparison with ideal situation or anonymous users only
		5: Specific Users (friends in social networks)
8. Need to solve missions (EUL)	Using the imagination in solving user tasks playing educates and can use new knowledge into reality.	1: There are missions
		5: There are no missions
9. Advice (EUL)	Does the application give tips on how to spend less energy?	1: The application does not give advice
		3: The application gives advice
		5: The users and the application give advice

lists the metrics we proposed to evaluate this type of software. These proposed metrics are useful not only to understand the features of a specific software package but also to guide the development of new EMS.

Using these metrics and the lists (long, medium and short lists), we can quantitatively evaluate existing software by determining the effectiveness of the solution prior to implementation (utilizing the SQMO process [19]). If a solution does not exist to achieve the desired effectiveness, the implementation of new EMS may be guided by the suggested metrics. The tables provided in this study may provide detailed insight regarding the requirements that SMS should meet to be effective.

7. Discussion

Generally, none of the software packages presented on the long list will achieve 100% effectiveness, implying that the criteria will not be fully met by existing software. This is an expected result because the ideal software does not exist. This analysis, however, helps us to understand which software is most effective. In addition, we may be in a situation where the greatest effectiveness we can achieve is very low, implying that it may not be an option to use any of the existing EMS available on the market, and it may become necessary to develop new software. The SQMO methodology and the proposed metrics improve the analysis of software and define the requirements needed for new EMS.

To determine the effectiveness of the analyzed software, we utilized the SQMO methodology to develop proposed metrics. Specifically, the calculation used to analyze the software is the Weighed Global Quality Rate Strategy. As proposed in [19], this calculation quantifies the influence of the “weight” assigned to each category. The WGQR is defined as:

$$\mathrm{WGQR}={\displaystyle \sum }_i(QR{|}_i\times Weight{|}_i)$$

where QR|_i is the quality rate in the functionality, usability or efficiency categories; and Weight|_i is the “weight” of the functionality, usability or efficiency categories. The “weights” of all categories must equal 100%. A value near 100% represents software with a perfect fit for our purposes.

This calculation was necessary to analyze the software on the short list; the answers can represent the different profiles of users of the software. In addition, this is a key element to determine the Weight for each category and is dependent on the overall goals of our work.

Due to confidentiality reasons, we are not going to show here the short list, but on Figure 6 we show the comparison of the representation of the compared software on the short list for the Usability and Technical Performance metrics.

WGQR obtained for each software package in the short list can be used to understand if the software is enough for our purposes through its effectiveness, and to select the best alternative (highest value) form the existing ones. The calculus of effectiveness can be done performing a ratio of the WGQR value of a specific software package on the short list, with the scores of the best theoretical software (the proportion between the values obtained and the maximum possible values one can obtain).

Compiling the lists was useful to analyze the target market, review the existing applications and using behavioral science and gamification techniques, engage users to use the applications. The use of behavioral techniques is necessary to obtain information from the user and provide useful feedback. Common criteria must be established to properly evaluate software and provide insight regarding the primary desired features of new EMS. The proposed metrics shown in Table 4 represent a method to systematically measure the primary features of EMS. Because these metrics may be used to quantitatively analyze existing applications, these features must be incorporated into new EMS.

8. Conclusions

Applying the SQMO methodology and proposed metrics, we were able to classify software proposed on the long list and to assign a numerical score using an objective approach. Next, using the medium and short lists, we analyzed the effectiveness and determined if the proposed EMS fit specific criteria.

The proposed methodology and proposed metrics can be useful for developing new EMS or selecting existing EMS that fit the requirements of the users and the project needs.

This study applied the SQMO methodology to a different area than it is commonly used for and demonstrated that it is a useful methodology to address problems related to EMS.

This study helps in the development of ACE gamification application [14] and may be useful in the development of new EMS. This analysis includes state-of-the-art processes and metrics for EMS and may be the starting point to develop a software capable of generating interest in public and private institutions (such as residential complexes, hotels, sports centers, leisure centers, and others); this analysis may also improve application use and parameterization. The active participation of users may help both institutions and clients benefit from use of the software application.

Analysis of the proposed metrics provides insight regarding what is needed to successfully implement EMS (or to use an existing one). The primary categories are Content, Control, Meaningful Feedback, High-order thinking skills, Usability and technical performance, Interactivity and engagement, Type of comparison, Need to solve missions and Advice. Of these categories, Meaningful Feedback, implies the prior understanding of user of the EMS; hence, it is the most complex category. This can be easily reviewed in Table 4 because the metrics that we propose to evaluate this criterion are the most subjective. Additional research must be conducted to establish additional metrics and allow for a detailed evaluation of Meaningful Feedback. The EMS should be continuously adapted to increase the knowledge of the user and to retain interest in the system and enhance Interactivity and engagement. The experience must be personalized for each user and must modify assumptions regarding user knowledge.

This study compiled a long list of EMS software for the Spanish, Russian and US markets.