Measuring the Use of the Active and Assisted Living Prototype CARIMO for Home Care Service Users: Evaluation Framework and Results

Abstract

To address the challenges of aging societies, various information and communication technology (ICT)-based systems for older people have been developed in recent years. Currently, the evaluation of these so-called active and assisted living (AAL) systems usually focuses on the analyses of usability and acceptance, while some also assess their impact. Little is known about the actual take-up of these assistive technologies. This paper presents a framework for measuring the take-up by analyzing the actual usage of AAL systems. This evaluation framework covers detailed information regarding the entire process including usage data logging, data preparation, and usage data analysis. We applied the framework on the AAL prototype CARIMO for measuring its take-up during an eight-month field trial in Austria and Italy. The framework was designed to guide systematic, comparable, and reproducible usage data evaluation in the AAL field; however, the general applicability of the framework has yet to be validated.

1. Introduction

In response to the demographic change in Europe, various technologies for assisting older people have been developed in active and assisted living (AAL) projects [1,2]. These AAL technologies mainly aim to support aging in place. Due to public co-funding of national and European collaborative AAL-projects [3], the variety of AAL systems and service prototypes is continuously increasing.

The evaluation of AAL projects usually focusses on analyses of usability, acceptance, and more recently, on the effects of such technologies [4]. However, little is known about the actual take-up of the AAL prototypes. The actual take-up or usage of an information and communication technology (ICT) system relates to the degree to which people make use of new technologies, and in the end, have adopted them. Information about the usage of an ICT system allows us to learn more about what users like and which preferences they have [5]. Usage data analysis is thus able to single out relevant features of a multi-feature system and facilitate the ability to assess the importance of newly developed features. Additionally, these data enable us to identify target groups interested in the system and provide an evidence base for the estimation of market potential.

The take-up or usage of ICT can be measured either explicitly, by collecting subjective data in surveys or focus groups [5,6,7], or implicitly, by collecting objective data, such as user interaction records with the system [5,8,9]. Survey data reflect self-reports from users but may lack accuracy and details due to a range of possible response biases (i.e., systematic errors in answering survey questions). Response biases, such as limitations in the ability to recall past system interactions [10] or responses reflecting overly positive self-descriptions [11], may lead to over- or under-reporting of system usage.

On the other hand, automatic recording of usage is less invasive. Automatically recorded usage data measure the use of a system without users always being aware of them being tracked, which is less prone to response bias than self-reporting. Usage data collection, however, also requires careful preparation to gain meaningful data, to avoid gathering a useless mass of data, and finally, to enable researchers to answer their research questions. Thus, careful planning and preparation of log data collection and analysis is essential to achieving good data quality and reliable results [5,12].

Previous research on automatic usage recording has either focused on technical frameworks for usage measurement [13,14,15] or the user grouping/classification of usage patterns [16,17,18]. Topics less developed in previous work on usage measurement include the selection of the logging component(s) and data source(s), the determination of logging capacity, and ethical and data privacy considerations.

Only a few AAL projects report the practice of automatic usage recording. In MobileSage, a technical framework for usage measurement was developed and used to adapt user profiles [8,19]. In Zentr AAL, the actual utilization of system components and features by different user groups was analyzed [20]. To increase the use of automatic usage measurement in AAL projects, a comprehensive usage data evaluation framework is required to guide systematic, comparable, and reproducible usage data logging, preparation, and analysis that covers technical, ethical, and data protection requirements.

This paper aims to contribute to the evaluation of AAL systems by proposing a standard for usage data evaluation and applying this for the first time to an AAL prototype, called CARIMO. The suggested usage data evaluation framework comprises all steps relevant for a comprehensive usage data collection and offers options for data collection and analysis. The framework and experiences in applying the framework for assessing the take-up of the AAL-prototype CARIMO aim to better guide future usage data collection and analysis in the research field of AAL. Researchers may use this framework for evaluating the take-up of AAL systems in a systematic, comparable, and reproducible manner.

The remainder of this paper is structured as follows: In Section 2, the AAL prototype CARIMO is presented. Section 3 considers the usage data evaluation and defines the framework for usage data evaluation in AAL projects. In Section 4, the application of the framework is presented. The paper closes in Section 5 with a discussion.

2. The AAL Prototype CARIMO for Home Care Service Users

According to the online catalogue for AAL products, the majority of AAL systems address “health & care,” “living & buildings,” or “safety & security.” Only a few relate to supporting and strengthening “vitality & abilities,” particularly the “physical abilities” of older people [21]. Maintaining the abilities of older people by fostering physical activities, such as walking or exercising, promotes healthy aging, delays the onset of disabilities, and postpones frailty [22,23]. Being physically active can maintain mobility and thus postpone the demand for long-term care. Hence, physical activity is an important factor for active aging, allowing older adults (even with dependencies) to age in place and participate in social life [22,24,25].

On that account, the “Active Assisted Living Programme” project “CareInMovement—CiM” developed CARIMO, an ICT-based system to promote physical activities for home care service users in Austria and Italy. Home care service users are older, care-dependent people living at home and receiving professional support to better cope with (instrumental) activities of daily living, such as getting dressed, bathing, cooking, or shopping. The CARIMO system was tested and evaluated in an eight-month field trial in Austria and Italy with test and control groups [26].

Contrary to most AAL systems that address care, safety, or mobility issues [21], the AAL prototype CARIMO was primarily designed to maintain the physical abilities of home care service users by fostering physical activities, such as walking or exercising. It complements the few apps available for seniors, such as “Fitivity Senior Fitness” or “Daily Senior Fitness Exercise,” as CARIMO was especially tailored to older, care-dependent people living at home. To assign suitable fitness exercises to this target group, sport scientists developed a test [27] to assess the physical abilities of the users. Amongst others, the data collection involved testing muscle strength (e.g., grip strength test) and endurance (e.g., chair rise test), as well as physical limitations. Based on the assessment conducted by care workers, sport scientists assigned the participants to one out of two training levels (physically frail, physically dependent) [27]. The customized training plans for each participant were derived from their training level, and if available, data on physical limitations. Unlike existing fitness apps, CARIMO additionally provided an information and entertainment section especially prepared for older, care-dependent people to provide stimuli for the body and the mind [26].

CARIMO differed from other AAL physical training prototypes for older people in three areas: intention for use, devices used, and the size and duration of the trial. First, a low-threshold exercise program developed for care-dependent people living at home offered a multi-component training program to improve coordination, strength, balance, and endurance [27] rather than training selected parts of the body, such as training using a treadmill [28] or ergometer [29,30]. Second, it applied two commercially available, comparatively low-cost components (with prices expected to fall over time) and thus did not require expensive fitness equipment, sensors, or computing hardware [28,29,30,31]. Finally, compared to most of the previous work reporting on short-term tests and small sample sizes [29,30,31], CARIMO was tested in an eight-month field trial with about 100 participants.

To incorporate the user requirements of home care service users, lead users of the target group were involved in all development phases according to the human-centered design (HCD) approach [32]. To ensure acceptable usability and user experience of this prototype, we especially paid attention to appropriate text and button sizes, labeling of icons, high contrast, text and audio output (for fitness exercises), and a simplified user interface (UI) [33].

2.1. Components of CARIMO

CARIMO consisted of two commercially available hardware components: (1) a tablet (Galaxy Tab A 2016, Samsung Electronics Co., Ltd., 129, Samsung-ro, Yeongtong-gu, Suwon-si, Gyeonggi-do, Korea) and (2) a bracelet with a fitness/activity tracker (Gear Fit2, Samsung Electronics Co., Ltd., 129, Samsung-ro, Yeongtong-gu, Suwon-si, Gyeonggi-do, Korea) (see Figure 1). Both components were selected according to predefined criteria. For the tablet these were: (i) display size of at least nine inches (aspect ratio of 16:10 or 16:9), (ii) Android 5 or higher, (iii) 3G or 4G internet connection, and (iv) a maximum price of EUR 320. The main criteria for the activity tracker were: (i) battery life of three to five days without using GPS, (ii) standalone operating mode (no permanent contact with a paired smartphone or tablet needed), (iii) waterproof or at least splash proof, (iv) adaptable UI and data transfer via Bluetooth or Adaptive Network Topology (ANT), and (v) maximum price of EUR 200. More details on the selection process for the fitness/activity tracker can be found in Willner et al. [34].

CARIMO consisted of a web app (for configuration purposes), an Android native app for the tablet, and a new watch face for the activity tracker. Existing systems (e.g., SimpliFlow—SimpliFlow International GmbH, Krone Platz 1, Klagenfurt, Austria—as a sports/training platform and SITOS—bit media e-solutions GmbH, Kärntner Straße 337, Graz, Austria—as an eLearning tool for online courses) were enhanced and integrated with new components developed within the project to provide a new solution to meet the aims of the AAL Programme. On the tablet, the CARIMO app was set as the home screen displaying the menu, such that the users always came back to the menu when pressing the home button. For the fitness bracelet, a digital CARIMO watch face was implemented, indicating the time and step count, as well as a shortcut for tracking activities, such as walking, via GPS. The activity tracker was used to measure steps and physical activities lasting more than ten minutes. These parameters were considered for rating the daily amount of physical activity of the users.

2.2. Fitness Program: Body-Related Features of CARIMO

The “body-related features” of CARIMO included four functions to improve or at least maintain physical abilities: (i) fitness exercises, (ii) activity overview, (iii) tip of the week, and (iv) recording of activities [26].

“Fitness exercises” provided health-enhancing physical exercises focusing on mobilization, stabilization, muscle strengthening, and endurance [27]. The 10-min training sessions changed daily, were based on the fitness level of the home care service users, and consisted of twelve exercises each. Each session was intended to be completed by the users on their own or together with a caregiver (care worker, family carer, or volunteer). For enhanced training planning functionalities (e.g., management of training levels and training plans), the SimpliFlow training management platform was integrated. More than 250 exercise videos were created by a sports scientist, with people aged 50–60 demonstrating how to perform the exercises. All exercise instructions were available as both text and audio output.

The “activity overview” enabled home care service users to keep track of their activities on a daily, weekly, and monthly basis. According to their physical achievements, users were awarded a bronze, silver, or golden trophy cup on a daily or weekly basis. Data from the fitness bracelet were automatically recorded and included in this section.

A “tip of the week” was meant to promote the use of the CARIMO tablet and to stimulate the users to keep physically active, either by suggesting simple changes for boosting activities outdoors (e.g., walking around a bench twice before sitting down) or by exercising at home.

The activities part of the “activities and notes” features enabled CARIMO users to manually enter activities into their “list of activities” if not already recorded by the activity tracker.

2.3. Information and Entertainment: Mind-Related Features of CARIMO

The “mind-related features” of CARIMO comprised five functions providing information and entertainment: (i) notes, (ii) newspapers, (iii) internet and games, (iv) a system manual, and (v) a function to arrange appointments [26].

CARIMO offered access to six regional newspapers and magazines in each country. Based on feedback from design workshops with lead users from the target group, we implemented an RSS (really simple syndication) reader with a large font size and high contrast to make information more legible than in traditional paper newspapers.

The feature “internet and games” provided an internet browser and up to six games (Math Quiz, Solitaire, 4-Pictures-1-Word, Crossword Puzzles, Connect Four, and Word Search) to provide an intellectual challenge and to train the handling of a tablet in general.

In addition to a series of visits of care workers to introduce CARIMO to the home care service users at the beginning of the trial, CARIMO also offered an interactive system tutorial to enable users to train themselves in the use of CARIMO. For this feature, the e-learning system SITOS was integrated.

Finally, the care network (consisting of care workers, family members, and volunteers) could arrange meetings via the CARIMO appointment feature. This allowed the home care service user to check the date and time of the next visit of a carer. The notes part of the “activities and notes” feature enabled the home care service users to leave a note for their caregiver.

3. Methods

3.1. Usage Measurement and Its Challenges

System usage data reflect the actual use of a system by measuring how often, when, and by whom an ICT-system was used [14,20]. Automatic recorded usage data are quantitative data that are automatically collected when a system is used [5,14,35]. Automatic usage data can be measured on two levels of the ICT system: on the system/application level via application usage measuring [36] and on the feature level via in-app logging of events [37]. Measuring usage data on the application level enables the comparison of different applications by assessing which applications are used, how often, and when they are used. Thus, usage prediction and recommendations for new applications can be derived [38]. For logging the usage data of mobile applications, a range of tools exists, e.g., AppTracker [36] or ProfileDroid [39]. The usage data at the application level, however, do not tell anything about which features or subfeatures of a system/application are used. On the feature level, insights into the take-up of application features can be gained, user profiles can be created or adapted, and information for further feature development and redesign can be derived [9,19,37]. Usage data on the feature level are measured by instrumenting source code to automatically log user interactions [14]. This can either be done using existing usage mining tools [40] or applying one’s own implementation [14].

No matter whether usage data are measured at the application or feature level, large amounts of data will be generated. If usage data evaluation is not planned carefully, a mass of data is produced, which challenges both the data processing and the generation of reliable results. Previous work addressed parts of the usage data evaluation process. They focused on tools and software for usage measurement [13,14,15,36]. Other work in the field of software development point to the need for estimating the expected amount of data and data transmissions beforehand, and to calculate the capacities required [41]. In addition to the amount of data to be processed, protocols also need to be defined on how to merge different data sources, e.g., to evaluate the usage of multicomponent systems.

For the purposeful analysis of usage data, the measures for “use” have to be appropriately defined, which may turn out not to be trivial. While in some cases, it is sufficient to know that a user just “clicked” a button, in other cases, it might also be of interest how long a certain application or feature has been used. This implies that not every user interaction may have a meaningful interpretation. First, raw usage records can contain corrupted data (errors in logging) or unintended duplicates (e.g., data like walking are logged automatically with an activity tracker and can also be entered by a user manually using an app interface), which require careful data preparation [42]. Second, measures are needed that provide meaningful information for answering the research questions. Examples for such measures are groupings/classifications of usage patterns that either result from data-driven or concept-driven approaches. In order to measure the actual take-up, an application or feature should be used repeatedly, which also needs to be reflected in an appropriate measure. Additionally, data sources may be needed for creating user profiles (e.g., sociodemographic characteristics of frequent users).

Usage data collection has been rarely discussed in the context of data protection and ethical considerations. Most of the previous literature does not consider the ethical appropriateness of the usage measurement approaches chosen [43]. Such considerations are, however, relevant as usage data may reveal personal details of user behavior while its collection is not obvious and visible for users. Data collection has to abide by the principle of data minimization according to the General Data Protection Regulation (GDPR) of the European Union. Taken together, as information on usage data collection, processing, and analysis is spread across different pieces of work, a framework providing a comprehensive approach comprising all relevant steps for usage data measurement would support future evaluation of the adoption of new AAL prototypes and systems.

3.2. The Usage Data Measurement Framework for AAL Systems

A general methodology, for instance, for web search transaction log analysis [42], is not available for usage data evaluation. Thus, based on the methodology proposed for web search transaction log analysis [42], the proposed usage data evaluation framework aims to ensure a systematic, comparable, and reproducible approach by addressing: (i) usage data logging, (ii) usage data preparation, and (iii) methods for usage data analysis of applications in the field of AAL (Figure 2). Iterations between the successive steps allow for going back to the previous step for revising previous decisions and refining the results. The framework also considers that usage can be measured and analyzed in various ways by offering different options for measurement and analysis.

3.2.1. Step 1: Usage Data Logging

The usage data logging step of the framework comprises: (i) the assurance of ethical appropriateness, particularly for ICT-supported solutions for older or vulnerable people; (ii) selection of evaluation objectives and decision on the logging levels; (iii) determination of the logging capacity and bandwidth; (iv) selection of the logging component(s)/data source(s); and (v) implementation of the data logging procedures.

Whenever (personal) data of users are logged, legal and ethical assessments are required. National and European regulations aim to improve data protection and user rights [44]. In addition, guidelines for ethical assessments of AAL projects (e.g., References [45,46,47]) seek to raise awareness of ethical issues. Ethical considerations usually relate to the type, duration, and frequency of data logging, data storage, and data protection to users’ rights to access, rectify, erase, withdraw, and transfer the data. In addition, it is important that researchers only collect as much data as is necessary to answer (pre-defined) research questions. To meet European legal requirements, users must give informed consent. To ensure the ethical appropriateness of AAL research projects, it is recommended to involve an ethics board or committee to assess the study design, including the aims, materials, and approaches for data collection and methods for the analyses.

Evaluation objectives determine the level of usage data collection. If the research focus is on the general use of a system/application, then logging can be restricted to the application level. However, if information about the use of features is needed, then logging at the feature level is required.

Considerations of the capacity have to address both the logging duration and the maximum number of users to be logged by the system. For feature-level logging, the logging granularity, i.e., the maximum number of sub-levels per feature, has to be determined. Based on the logging level, the logging capacity and bandwidth can be determined such that the logging component can be selected, configured, and if necessary, integrated into the system of which the usage data should be analyzed. To enable data logging, the logging component has to be configured, and if necessary, integrated into the system of which the usage data will be analyzed. For different logging components, see References [14,36,39,40].

For a multi-component system, one logging component or data source may not be sufficient. Particularly, AAL systems aim to combine different systems and technologies. Depending on the technologies used and the type of the feature, logging a feature’s usage may differ on different devices. For example, feature usage on a tablet is not recorded in the same way as, for example, the steps on an activity tracker. While logging of feature usage requires user interaction and user interaction logging, steps are recorded automatically using sensor data without user interaction on the device. Thus, for multi-component systems, where different data sources are needed, a user identifier, such as a universally unique identifier (UUID) [48] connecting the data to the single user, has to be implemented into each of the data sources.

The definition of use may depend on the features of a system an on the research questions. For some features, clicking on a button (e.g., to view the weather forecast) may count as “use.” Other features may require sophisticated approaches to assess the take-up. This is the case for features that aim to incentivize a user’s activities, such as doing exercises. A simple count of clicking on the program will thus not generate meaningful data. In such cases, additional information from time stamps or other components could be useful.

In addition, if user profiles are of interest, further information about the characteristics of the users of a system will be needed for the analysis [49], e.g., demographic data, such as gender or age. In both cases, the storage and data transmission of these data need to be planned and implemented beforehand [41]. Finally, if all needed data sources are identified, the logging procedure using one or more logging components must be implemented.

3.2.2. Step 2: Usage Data Preparation

The focus of this step is in usage data extraction, enrichment, and pre-processing. Subsequent analyses depend on the quality of this step. Depending on the logging component, data extraction can either be done through existing application programming interfaces (APIs) of the logging component, or if accessible, through the component’s database. If additional data sources are needed for the analysis (e.g., data generated by an activity tracker), these data must also be extracted. The extracted raw logging data require appropriate data pre-processing procedures to access the information represented by the data. In the first pre-processing step, in line with the research interest, the different sources are merged together and brought into a common structure using the UUID. Subsequently, pre-processing procedures for data cleaning help to identify corrupted data and unintended duplicates [42,50].

3.2.3. Step 3: Methods for Usage Data Analysis

Automatically generated usage data can be analyzed in a variety of ways, for example, descriptive statistics [51], analytical models [9,18], or using expert-driven conceptual approaches [17]. Descriptive statistics provide information on the frequency of use and measures of central tendency, e.g., median or mean [51] for a certain point in time or over a period. For the analysis of usage behavior, user workflow or process mining techniques [9,52,53] or time series analysis [54] can be applied. Another method for analyzing usage patterns and usage behavior is user grouping. User groupings can be defined either in a data-driven manner, e.g., by using data mining techniques, such as Jenks’s natural breaks [55], or in a conceptually-driven manner, e.g., based on expert knowledge [17] or previous work. Common groupings address the frequency of use [56], duration of use [16], or usage phases [17]. A comparatively simple grouping is a dichotomous “use or non-use” classification. The grouping granularity is dependent on the duration of the usage data logging. If only a week is recorded, the granularity might be hourly or daily, while if several months or years are recorded, the granularity might also be weekly, monthly, or yearly.

For assessing the repeated use of an application feature, the retention rate indicator is a useful measure. The retention rate measures the percentage of users who continue using an application over a certain time period [57,58].

4. Results: The Usage of CARIMO

The proposed framework was applied to evaluate the usage of the AAL prototype CARIMO in an 8-month field trial with test and control groups in Austria and Italy; the results are presented in this section.

4.1. The CARIMO Sample

Table 1. Sample description of the trial participants (test group).

Test Month	1	2	3	4	5	6	7	8
Number of participants	104	97	94	89	89	85	85	85
Gender
Female	77	71	69	66	66	63	63	63
Male	27	26	25	23	23	22	22	22
Age group
<60 years	2	2	2	2	2	2	2	2
60–69 years	29	28	27	27	27	26	26	26
70–79 years	51	47	45	41	41	39	39	39
>79 years	22	20	20	19	19	18	18	18
Country
Austria	64	60	57	55	55	54	54	54
Italy	40	37	37	34	34	31	31	31
Dependency level
Can do some (I) ADLs with help only (help needed)	28	25	24	23	23	21	21	21
Can do some (I) ADLs with difficulty but manage on their own (difficulty)	49	45	44	40	40	39	39	39
Can do all (I) ADLs without help (independent)	24	24	23	23	23	23	23	23
Missing	3	3	3	3	3	2	2	2

Notes: Source: SRFG (Salzburg Research), CiM usage data 2017/2018, and WU (Vienna University of Economics and Business), CiM effectiveness survey 2017/2018.

provides the sample description. The field trial started with 104 participants (64 in Austria and 40 in Italy) and ended after eight months with 85 participants (54 in Austria and 31 in Italy). After eight months, 82% of home care service users were still participating. The decrease in trial participants was mainly due to hospitalization, admission to a care home, or lack of interest [26]. About three quarters of participants were women. The largest age group was participants between 70 and 79 years. Twenty percent were 80 years and older. About 60% of the trial participants were Austrians and 40% Italians. With respect to self-assessed care dependency, about 70% needed personal help or had difficulties coping with (instrumental) activities of daily living ((I) ADL) [59].

4.2. Usage Data Logging for CARIMO

4.2.1. Approval of the Ethical Appropriateness of the Usage Data Collection

During the field trial, CARIMO usage data were collected as pseudonymized personal data as every user was assigned an identifier that linked the data of the logging component to the other system components and personal characteristics of the participants. The CARIMO system stored usernames that were fed into selected features, e.g., for greeting the user when starting the fitness app (“Hello, Peter!”).

Independent experts in ethics and privacy policy were consulted to assess the data processing procedures. The CiM project and its evaluation design (including the informed consent) were reviewed and approved by the independent ethics committee at the University of Salzburg (EK-GZ 30/2016). Additionally, the proposed data collection project was submitted to the Austrian data protection authority and registered to the Austrian data processing register (DVR: 4008479/006).

4.2.2. Evaluation Objectives and the Selection of Logging Levels

The evaluation of the take-up of CARIMO mainly aimed to measure the use of both the CARIMO app and its body- and mind-related features in two European countries, Austria and Italy. Accordingly, we decided to log at the CARIMO feature level, including the first subfeature level (see Section 3.1). For example, we selected the logging level to get access to data on, for example, the use of the “internet & games” feature and the subfeature “games.” However, further levels of this feature, such as specific games, were not of interest and hence not selected. Thus, the logging granularity was set on the first subfeature level.

4.2.3. Determination of Logging Capacity and Bandwidth

The logging capacity calculations considered two dimensions: the number of users (more than 100), and the duration of the trial (eight months + extension options). Thus, in addition to a monthly bandwidth of approximately 70 to 80 MB per user, for the CARIMO app, we calculated 5 MB for the logging data per user and month.

4.2.4. Selection of the Logging Component(s) and Data Source(s)

Since there is a variety of logging tools available (see Section 3.2.1), we decided not to go for a time-consuming proprietary development. We selected a logging component that met the following system and privacy requirements: (i) supporting mobile application logging, enabling (ii) the research team to get access to the raw usage data for an independent usage data analysis, and (iii) self-hosting the data since third-party hosting did not fulfill data privacy requirements.

According to Kumar and Thakur [40] and our own investigations, only two tools (Piwik and Google Analytics 360) were able to support mobile application usage logging at the time of decision in 2016. Piwik (today known as Matomo, InnoCraft, 150 Willis St, Wellington, New Zealand) was preferred to Google Analytics 360 as its use was free of charge and allowed self-hosting with access to raw usage data. For the server, we used Piwik version 3.0.1. On the client level (CARIMO tablet), we started data logging with Piwik SDK for Android 0.0.3. In month four, a new version of CARIMO including additional games and bug fixes was released. This release was also used for a client update of Piwik to version 2.0.0.

For the fitness program of CARIMO, Piwik usage measurement alone was not sufficient for two reasons. The operating system of the activity tracker (Tizen) was not supported by Piwik. Thus, the use of the activity tracker (e.g., recording of steps and activities) could not be tracked by Piwik. We accessed this information indirectly via the CARIMO system database where these data were stored. Second, the duration and the number of completed exercises on the tablet were difficult to measure and extract. As the determination of a completed exercise involved considering many exceptions (e.g., skipping or pausing of exercises), the recorded time per exercise stored in the CARIMO system database, excluding, for example, skipped exercises and pauses, was used to measure the number of exercises performed.

As we were interested in comparing the take-up of CARIMO and its features between two countries (Austria and Italy), we imported this information from the CARIMO system database, where the place of residence of each user was stored for support reasons. As different data sources (Piwik and the system’s database) were used, we implemented a UUID in each data source, which allowed for merging the data into one data set.

4.2.5. Implementation of the Data Logging Procedures

Integrating Piwik for usage data logging into CARIMO enabled the logging of: (i) the starting and stopping of the CARIMO activity, (ii) changes in the content (e.g., for Android displaying and removing fragments), and (iii) every interaction with user interface components, such as the use of buttons, input fields, or check boxes.

In alignment with our research interests, usage measurement for CARIMO had to provide information on the characteristics of the users and the frequency of their use of the CARIMO app and its features. Thus, we implemented the logging of: (i) a unique user identification number (UUID) (who); (ii) type of interaction, such as click, touch, or scroll (what); (iii) UI component that was interacted with (where); and (iv) date and time of the interaction (when). On this basis, further information, such as how often the user accessed the feature, could be derived. The interaction was identified using a unique page/view identifier (e.g., “/main” for the home screen) combined with a unique user interface component identifier (e.g., “/main/btnAdvice” for the tip of the week button in the home screen). Thus, clicking somewhere on the screen resulted into no further action and was not recorded.

4.3. Usage Data Preparation

4.3.1. Data Extraction and Enrichment

We extracted the app usage data from Piwik. The data for calculating the activity tracker usage were obtained from the system’s database. Additional data for determining the exercise duration and the number of completed exercises, as well as the country affiliation of each user, were extracted from the system’s database. Both data from Piwik and the system’s database were extracted, and a pseudonym was applied using the UUID.

4.3.2. Data Pre-Processing

The extracted raw data from the system’s database and Piwik were merged together into a common structure using the UUIDs. Next, corrupted data were removed by sorting each field in sequence. In the final pre-processing step, unintended activity duplicates were removed whereby the automatically recorded data were preferred to the manually recorded data.

4.4. Methods for Usage Data Analysis

4.4.1. Definition of Usage Measures

For the tablet, the usage frequency was measured using the Piwik metrics “visits” and “unique page views” [60]. A “visit” starts with the first use of CARIMO on a new day or after a pause of at least 30 min. A “unique page view” represents the number of “visits” that included a certain CARIMO feature or subfeature. If a feature or subfeature was used multiple times during a “visit,” it was only counted once. As not every interaction with the app was expected to be a meaningful measure of use, we set certain time thresholds. For exercises, “time spent on training” was only counted if the screen time per exercise was 30 s or longer. This timespan was set based on the time required to perform ten iterations of a simple exercise, such as shoulder shrugs. Originally, we planned to link the pulse measurement of the activity tracker to the exercises. This did not work satisfactorily in the pre-trial phase, as many women had such slender wrists that the necessary skin contact for a reliable pulse measurement was not attainable. As other work also reported inaccuracies [61], we decided to not use the pulse measurement of the activity tracker. The “number of performed exercises” captured the total number of exercises lasting 30 s or longer. As the activity tracker generated data automatically without user interaction, we decided to calculate “usage days” instead of “visits.” The measure “recorded activities per day” was introduced for activities logged by the fitness tracker for physical activities lasting more than ten minutes or activities that were entered manually. A monthly analysis for the 8-month field trial seemed fine-grained enough to describe usage behavior appropriately.

4.4.2. Definition of CARIMO User Groups

User groups for CARIMO were calculated for the test group in total, as well as on a country basis. According to previous usage measurement for AAL technologies [20], the framework used four user groups for assessing the intensity of using a system.

(i)

frequent users: participants who used a CARIMO feature more often than a regular user.

(ii)

regular users: participants who used a CARIMO feature on a regular basis.

(iii)

infrequent users: participants who used a CARIMO feature less often than a regular user.

(iv)

non-users: participants who did not use a CARIMO feature at all.

Particularly, the characteristics “frequent,” “regular,” and “infrequent” needed thresholds to assign users to these groups. For the thresholds below, we applied an expert-knowledge-driven approach.

Definition of Usage Groups for Body-Related Features of CARIMO

Body-related features of CARIMO comprised two functions “fitness exercises” (training program personalized by sport scientists based on assessments of physical abilities by care workers; see Section 2) and “activities” (manually entered into the system by the users or automatically recorded by the activity tracker). We specified thresholds that informed about the frequency of use based on the recommendations for health-enhancing physical activity [62]. In general, for healthy adults aged 65 and above, it is recommended that they perform physical activities of at least 150 min at a moderate intensity or 75 min of vigorous intensity per week [62]. Moderate-intensity corresponds to tasks requiring effort of three to six “metabolic equivalents” (METs), e.g., cycling for pleasure or transport (≤10 mph), and vigorous intensity corresponds to more than six METs, e.g., cycling fast (>10 mph) [63].

Table 2. Usage groups for body-related features.

User Group	Fitness Exercises (Per Month)	Activities (Per Month)	Body-Related Functions (Per Month)
Frequent user	8 times or more	25 times or more	Fitness exercises = frequent AND activities = frequent Fitness exercises = frequent AND activities = regular fitness exercises = regular AND activities = frequent
Regular user	Between 4 and 7 times	Between 9 and 24 times	All other possibilities
Infrequent user	Between 1 and 3 times	Between 1 and 8 times	Fitness exercises = infrequent AND activities = infrequent Fitness exercises = infrequent AND activities = non-user Fitness exercises = non AND activities = infrequent
Non-user	Never	Never	Fitness exercises = non AND activities = non-user

shows the user group definition for the body-related features (fitness exercises, activities, and total). Based on the WHO recommendations and considering the health conditions of our target group, we defined a regular user as a user who does “fitness exercises” between 4 and 7 times a month (60 and 105 min) and performs between 9 and 24 “activities” a month (225 and 600 min). One run of “fitness exercises” was estimated with 15 min and one activity with 25 min. Consequently, a frequent user was defined per the above guidelines and an infrequent user was below the values of a regular user.

Definition of Usage Groups for Mind-Related Features of CARIMO

Mind-related features of the CARIMO app covered “newspapers” and “internet and games,” where “newspapers” was a single feature and “internet and games” consisted of two subfeatures. The categories were defined specifically for this study using the expertise of end-user experts, i.e., home care providers. They assumed that people interested in “newspapers” (regular users) will use this feature at least 10 times a month. They also stated that highly interested people (frequent users) will use this feature at least 20 times a month. Accordingly, an infrequent user was defined as being below these values (

Table 3. Usage groups for mind-related features.

Usage Groups	Newspapers (Per Month)	Internet (Per Month)	Games (Per Month)	Internet & Games (I & G) (Per Month)	Mind-Related Functions (Newspapers, Internet & Games) (Per Month)
Frequent user	20 times or more	16 times or more	15 times or more	Games = frequent AND Internet = frequent Games = frequent AND Internet = regular Games = regular AND Internet = frequent	Newspapers = frequent AND I & G = frequent Newspapers = frequent AND I & G = regular Newspapers = regular AND I & G = frequent
Regular user	Between 10 and 19 times	Between 7 and 15 times	Between 8 and 14 times	All other possibilities	All other possibilities
Infrequent user	Between 1 and 9 times	Between 1 and 6 times	Between 1 and 7 times	Games = infrequent AND Internet = infrequent Games = infrequent AND Internet = non-user Games = non-user AND Internet = infrequent	Newspapers = infrequent AND I & G = infrequent Newspapers = infrequent AND I & G = non Newspapers = non-user AND I & G = infrequent
Non-user	Never	Never	Never	Games = non-user AND Internet = non-user	Newspapers = non-user AND I & G = non-user

—Newspapers).

For many CARIMO users, it was the first time in their lives that they had internet access. About two thirds of the users did not use the internet before [64]. Thus, the end user experts expected that a regular user would use the internet between 7 and 15 times a month (Table 3—Internet). Participants who used the internet at least 16 times were defined as frequent users. Usage between 1 and 9 times was classified as infrequent.

With respect to “games,” a regular user was defined as a user who used games between 8 and 14 times a month. Here too, frequent users were defined above and an infrequent below these values (Table 3—Games). Table 3 shows the usage groups for “newspapers” and “internet & games (I&G)” and the aggregation of both mind-related features.

4.4.3. Definition of the Retention Rate

The average retention rate for mobile apps after three months is 29% [65]. The adherence of older people to home-based fitness exercise programs is between 27% and 42.6% [24,66,67]. Thus, a monthly retention rate higher than 45% for body- and mind-related features was defined as a target for using CARIMO.

We calculated two retention rate indicators. The first indicator ($R{R}_{v1}$) included all retaining users (frequent, regular, and infrequent users) (Equation (1)). For the second indicator ($R{R}_{v2}$), a more conservative approach was applied. Only participants who used a system frequently or regularly were included (Equation (2)).

$$R{R}_{v1}= \frac{\mathrm{retained}\mathrm{users}\left(\mathrm{frequent}+\mathrm{regular}+\mathrm{infrequent}\right)}{\mathrm{total}\mathrm{users}}\times 100$$

(1)

$$R{R}_{v2}= \frac{\mathrm{retained}\mathrm{users}\left(\mathrm{frequent}+\mathrm{regular}\right)}{\mathrm{total}\mathrm{users}}\times 100$$

(2)

The retention rates of CARIMO (

Table 4. General use of CARIMO per test month.

Test Month	1	2	3	4	5	6	7	8
Total users	104	97	94	89	89	85	85	85
Users per country
Austria	64	60	57	55	55	54	54	54
Italy	40	37	37	34	34	31	31	31
Median usage of CARIMO (per user/month)
CARIMO app: median visits	32	36	27	30	29	27	33	29
Activity tacker; median days	14	18	15	20	17	15	3	0
Median usage of CARIMO in Austria (per user/month)
CARIMO app median visits	44	45	38	38	39	30	33	35
Activity tracker median days	17	20	20	28	22	20	16.5	0
Median usage of CARIMO in Italy (per user/month)
CARIMO app median visits	18	18	18	23	26	32	17	32
Activity tracker median days	6	10	5	11	9	3	0	0.5

Notes: Source: SRFG, CiM usage data 2017/2018.

—$R{R}_{v1}$ and $R{R}_{v2}$) were calculated for the test group in total and on a country basis.

4.5. Results of the CARIMO Usage Data Analysis

4.5.1. Usage Frequencies of CARIMO

Table 4 shows the participant flow and the median use of CARIMO (total and per country). The take-up of the CARIMO app remained quite stable over time while the median use per user of the activity tracker declined in the last two months.

Country-specific analysis revealed different use patterns over time in Austria and in Italy. In the first four months, the median use of the tablet in Austria was twice as high as in Italy. In month six, the use of the CARIMO app in Italy even slightly exceeded the Austrian level. The activity tracker use was higher in Austria than in Italy, while in the last two months, it decreased significantly in both countries (Table 4).

4.5.2. User Groups of CARIMO: Intensity of Use

Table 5. Usage groups and retention rates for body-related and mind-related features of the CARIMO app per test month.

Body-Related Features (Aggregation of ‘Fitness Exercises’ and ‘Activities’)								CARIMO Features	Mind-Realted Features (Aggregation of ‘Newspapers’ and ‘Internet and Games’)
1	2	3	4	5	6	7	8	Test month	1	2	3	4	5	6	7	8
								Total
44.2% (46)	44.3% (43)	45.7% (43)	51.7% (46)	41.6% (37)	37.6% (32)	40.0% (34)	32.9% (28)	Frequent users	18.3% (19)	16.5% (16)	11.7% (11)	18.0% (16)	14.6% (13)	16.5% (14)	18.8% (16)	15.3% (13)
22.1% (23)	25.8% (25)	24.5% (23)	23.6% (21)	24.7% (22)	29.4% (25)	30.6% (26)	29.4% (25)	Regular users	36.5% (38)	39.2% (38)	40.4% (38)	36.0% (32)	40.4% (36)	49.4% (42)	44.7% (38)	37.6% (32)
22.1% (23)	8.2% (8)	17.0% (16)	10.1% (9)	18.0% (16)	15.3% (13)	11.8% (10)	10.6% (9)	Infrequent users	37.5% (39)	32.0% (31)	36.2% (34)	32.6% (29)	30.3% (27)	24.7% (21)	29.4% (25)	29.4% (25)
11.5% (12)	21.6% (21)	12.8% (12)	14.6% (13)	15.7% (14)	17.6% (15)	17.6% (15)	27.1% (23)	Non-users	7.7% (8)	12.4% (12)	11.7% (11)	13.5% (12)	14.6% (13)	9.4% (8)	7.1 (6)	17.6% (15)
88.5%	78.4%	87.2%	85.4%	84.3%	82.4%	82.4%	72.9%	RRv1	92.3%	87.6%	88.3%	86.5%	85.4%	90.6%	92.9%	82.4%
66.3%	70.1%	70.2%	75.3%	66.3%	67.1%	70.6%	62.4%	RRv2	54.8%	55.7%	52.1%	53.9%	55.1%	65.9%	63.5%	52.9%
								Austria
54.7% (35)	55.0% (33)	64.9% (37)	65.5% (36)	52.7% (29)	50.0% (27)	48.1% (26)	40.7% (22)	Frequent users	25.0% (16)	21.7% (13)	12.3% (7)	20.0% (11)	14.5% (8)	18.5% (10)	16.7% (9)	14.8% (8)
26.6% (17)	28.3% (17)	17.5% (10)	20.0% (11)	21.8% (12)	25.9% (14)	37.0% (20)	33.3% (18)	Regular users	40.6% (26)	45.0% (27)	49.1% (28)	36.4% (20)	40.0% (22)	48.1% (26)	46.3% (25)	37.0% (20)
14.1% (9)	5.0% (3)	7.0% (4)	5.5% (3)	14.5% (8)	11.1% (6)	5.6% (3)	5.6% (3)	Infrequent users	28.1% (18)	25.0% (15)	26.3% (15)	27.3% (15)	25.5% (14)	24.1% (13)	31.5% (17)	29.6% (16)
4.7% (3)	11.7% (7)	10.5% (6)	9.1% (5)	10.9% (6)	13.0% (7)	9.3% (5)	20.4% (11)	Non-users	6.3% (4)	8.3% (5)	12.3% (7)	16.4% (9)	20.0% (11)	9.3% (5)	5.6% (3)	18.5% (10)
95.3%	88.3%	89.5%	90.9%	89.1%	87.0%	90.7%	79.6%	RRv1	93.8%	91.7%	87.7%	83.6%	80.0%	90.7%	94.4%	81.5%
81.3%	83.3%	82.5%	85.5%	74.5%	75.9%	85.2%	74.1%	RRv2	65.6%	66.7%	61.4%	56.4%	54.5%	66.7%	63.0%	51.9%
								Italy
27.5% (11)	27.0% (10)	16.2% (6)	29.4% (10)	23.5% (8)	16.1% (5)	25.8% (8)	19.4% (6)	Frequent users	7.5% (3)	8.1% (3)	10.8% (4)	14.7% (5)	14.7% (5)	12.9% (4)	22.6% (7)	16.1% (5)
15.0% (6)	21.6% (8)	35.1% (13)	29.4% (10)	29.4% (10)	35.5% (11)	19.4% (6)	22.6% (7)	Regular users	30.0% (12)	29.7% (11)	27.0% (10)	35.3% (12)	41.2% (14)	51.6% (16)	41.9% (13)	38.7% (12)
35.0% (14)	13.5% (5)	32.4% (12)	17.6% (6)	23.5% (8)	22.6% (7)	22.6% (7)	19.4% (6)	Infrequent users	52.5% (21)	43.2% (16)	51.4% (19)	41.2% (14)	38.2% (13)	25.8% (8)	25.8% (8)	29.0% (9)
22.5% (9)	37.8% (14)	16.2% (6)	23.5% (8)	23.5% (8)	25.8% (8)	32.3% (10)	38.7% (12)	Non-users	10.0% (4)	18.9% (7)	10.8% (4)	8.8% (3)	5.9% (2)	9.7% (3)	9.7% (3)	16.1% (5)
77.5%	62.2%	83.8%	76.5%	76.5%	74.2%	67.7%	61.3%	RRv1	90.0%	81.1%	89.2%	91.2%	94.1%	90.3%	90.3%	83.9%
42.5%	48.6%	51.4%	58.8%	52.9%	51.6%	45.2%	41.9%	RRv2	37.5%	37.8%	37.8%	50.0%	55.9%	64.5%	64.5%	54.8%

Notes: Source: SRFG, CiM usage data 2017/2018.

shows that, on average, about two-thirds of the trial participants used the body-related features of CARIMO on a frequent or regular basis. Toward the end of the field trial (month 8), the use decreased and the share of non-users increased. A comparison between Austria and Italy revealed that, on average, the usage of CARIMO in Austria was about 30% higher than in Italy. Furthermore, the proportion of frequent users over time was higher in Austria than the proportion of regular users. In Italy, this was only the case in months one, two, and seven. In general, there were more non-users for body-related features in Italy than in Austria.

With respect to mind-related features, Table 5 indicates that between 52% and 66% of the users used these features on a frequent or regular basis. In months six and seven, usage increased, which reflects the rollout of three new games in month six. In addition, for mind-related features, the share of non-users increased toward the end of the field trial (month 8). At the beginning of the trial, Italians used these features less often than Austrians. Starting with month four, the use in Italy approached the Austrian level. For the “information and entertainment” function in both countries, the proportion of regular users over time was higher than the proportion of frequent users.

The body- and mind-related features of CARIMO were perceived equally well by its users. However, body-related features were more frequently used (between 62% and 75% frequent and regular users) than mind-related features (between 52% and 66% frequent and regular users).

4.5.3. Retention Rates of CARIMO

Table 5 also shows the retention rates. On average, more than 80% of the users were retained ($R{R}_{v1}$), while in Austria, the share was about 90%, and in Italy it was about 70%. Seventy percent of all users used the fitness program to the expected extent of 45% ($R{R}_{v2}$). Country-specific patterns remained for the use of the body-related features of CARIMO. Retention rates in Austria (≈80%) were higher than in Italy (≈50%). For mind-related features, on average, 88% of the users were retained ($R{R}_{v1}$) with no differences between Austria and Italy. Overall, on average, 57% used CARIMO’s mind-related features to the expected extent of 45 % ($R{R}_{v2}$). However, in contrast to $R{R}_{v1}$, for $R{R}_{v2}$, there was a difference between Austria (≈61%) and Italy (≈50%), which can be explained by the higher proportion of infrequent users in $R{R}_{v1}$ in Italy.

5. Discussion and Conclusions

AAL projects aim to make a positive change in older people’s lives by facilitating access to new technologies. To assess whether the target groups effectively adopt the developed prototypes, automatically recorded usage data are a valuable source of information. However, publications on the actual use of AAL-technologies are rare, as are publications that apply usage data for other types of analysis, such as usability assessments or effectiveness analysis. We thus aimed to support the usage data assessment for AAL solutions by both providing a comprehensive approach for the evaluation and by reporting on the results of applying this approach on a complete and fully operational AAL prototype, called CARIMO.

We presented a usage data evaluation framework that offers a systematic, comparable, and reproducible process to improve usage data logging, preparation, and analysis in (longer) field trials of AAL prototypes. In comparison to previous work [8,14,17,18,36,37], it covers all steps relevant for a comprehensive usage data assessment and, by offering options for usage data evaluation, allows adjustments according to the research objectives or technical requirements of the AAL solution. Data collection and analysis is guided by a step-by-step process to generate awareness and to prepare for important decisions and point to tasks that may be easily overlooked (e.g., calculating logging capacity and bandwidth, and preparation for ethical clearance). The presented framework thus seeks to support preparing and conducting a systematic usage data measurement for AAL projects.

Usage data can be recorded without the knowledge of the users and may thus serve as important information on the actual use of an AAL system. On the other hand, usage data in general have become a “currency” in many areas of the ICT economy. Access to many apps and ICT services seem to be for free, but user behavior is being tracked and data are stored or used and sold for purposes that are often not transparent to users. To balance both interests, i.e., generating knowledge and assuring privacy, ethical approval of usage data collection and analysis is important for responsible research in the field of AAL. For this reason, ethical considerations have been included into the usage data evaluation framework.

In a first application, the framework’s systematics was used to measure the take-up of the AAL prototype CARIMO, an ICT-supported fitness and entertainment program for older, care-dependent people, in an eight months’ field trial. In general, logging generates a mass of data, which need to be prepared for a meaningful analysis. “One-size-fits-all” definitions do not always work. Thus, we suggested a context-dependent classification scheme. The framework offers options to decide on tools, measures, and groupings depending on the research aims and technologies to be assessed. For example, it proposes both data-driven and concept-driven approaches for usage groupings, with the latter based on expert knowledge and/or previous work. In the case of CARIMO, a concept-driven approach was taken to define user groups for the body-related features of CARIMO as well-accepted globally recommended levels of physical activities were available for the group of older adults.

Since there are differences in how people use an app, it is important to define what usage means in a certain context e.g., a person just having a look at fitness exercises versus performing those exercises for several minutes. Such definitions set beforehand may avoid misinterpretations of the data. A particular challenge was to assess the use of the “fitness exercise” feature. As there were no technologies (such as 3D cameras) that could ensure that the exercises had actually been performed, we had to rely on a proxy measure. In order to avoid overestimating the use of the “fitness exercise” feature by, for example, simply counting the clicks on the training program, we only counted an exercise if a predefined minimal screen time had been exceeded. For future studies on the use of “fitness exercise” features that also aim to measure the quality of the interaction, a 3D camera system for gesture recognition, such as the Kinect or Orbbec sensor [68], may turn out to be useful. Such changes in technologies could easily be addressed by the usage measurement framework as it allows for the adoption of additional data sources and usage measures with respect to the data of the new sensor.

In terms of insights gained by evaluating an AAL system’s use, we observed that a significant share of the trial participants had used CARIMO for eight months on a regular basis. The goal of using CARIMO to keep care-dependent people physically and mentally active was thus achieved. This seems remarkable given the vulnerable target group and their restricted experience with ICT-based solutions. In addition, we learned that incentives, such as new games (introduced in month 6), increased the usage during the field trial. At the end of the field trial, however, for both body- and mind-related features, the usage decreased in Austria and Italy, which may have been due to trial participants preparing themselves for the end of the field trial or due to users who just lost interest in the system over time.

The application of $R{R}_{v2}$ (only including frequent and regular users) on mind-related features showed that only considering $R{R}_{v1}$ could be misleading due to the high proportion of infrequent users in Italy. If the proportion of infrequent users is high, the calculation of $R{R}_{v2}$ allows for better insight into the system’s use.

Compared to existing work, the presented framework was applied to an AAL project that involved many users over a longer period of time [14]. As AAL systems normally consist of more than one device, the framework foresees the possibility to integrate more than one device [17]. In its first application on CARIMO, it was applied to two devices: a tablet and a fitness tracker.

This study has some limitations. First, as detailed above, the definition of an appropriate measure for the usage of an app or feature that neither overestimates nor underestimates the use is not always trivial. In particular, features that require users to engage in activities that are encouraged by an app but cannot be further tracked, e.g., fitness exercises on a smartphone or tablet app, rely on proxy measures, which may only approximate the actual usage as intended by the developers of an app. If actual tracking is the main aim, then other technologies, such as 3D cameras or reliable body sensors, may be useful for future work given that users’ privacy can still be guaranteed. Second, by using the proposed usage measurement framework, we were able to assess which features were used, how often, when, and by whom. We were not able to make statements on how or why the system was used. To learn more about the characteristics of users in AAL projects, future work could aim to combine logging data with survey data. Finally, while the usage data evaluation framework was designed to guide systematic, comparable, and reproducible usage data evaluation in the AAL field, the general applicability of the framework has yet to be validated.