Next Article in Journal
On the Active Adsorption of Chromium(III) from Alkaline Solutions Using Multiwalled Carbon Nanotubes
Previous Article in Journal
Hybrid Pulse Width Modulation Strategy of a High-Frequency Link Three-Phase Four-Leg Matrix Converter Based on Compensation Theory
Previous Article in Special Issue
Helping the Visually Impaired See via Image Multi-labeling Based on SqueezeNet CNN
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:

Measuring the Use of the Active and Assisted Living Prototype CARIMO for Home Care Service Users: Evaluation Framework and Results

Cornelia Schneider
Birgit Trukeschitz
2 and
Harald Rieser
Institute of Computer Sciences, University of Applied Sciences Wiener Neustadt, 2700 Wiener Neustadt, Austria
Research Institute for Economics of Aging, Vienna University of Economics and Business, 1020 Vienna, Austria
Mobile and Web-based Information Systems, Salzburg Research Forschungsgesellschaft m.b.H., 5020 Salzburg, Austria
Author to whom correspondence should be addressed.
Appl. Sci. 2020, 10(1), 38;
Submission received: 12 November 2019 / Revised: 12 December 2019 / Accepted: 17 December 2019 / Published: 19 December 2019
(This article belongs to the Special Issue Ambient Assisted Living and Cognitive Assistants)


To address the challenges of aging societies, various information and communication technology (ICT)-based systems for older people have been developed in recent years. Currently, the evaluation of these so-called active and assisted living (AAL) systems usually focuses on the analyses of usability and acceptance, while some also assess their impact. Little is known about the actual take-up of these assistive technologies. This paper presents a framework for measuring the take-up by analyzing the actual usage of AAL systems. This evaluation framework covers detailed information regarding the entire process including usage data logging, data preparation, and usage data analysis. We applied the framework on the AAL prototype CARIMO for measuring its take-up during an eight-month field trial in Austria and Italy. The framework was designed to guide systematic, comparable, and reproducible usage data evaluation in the AAL field; however, the general applicability of the framework has yet to be validated.

1. Introduction

In response to the demographic change in Europe, various technologies for assisting older people have been developed in active and assisted living (AAL) projects [1,2]. These AAL technologies mainly aim to support aging in place. Due to public co-funding of national and European collaborative AAL-projects [3], the variety of AAL systems and service prototypes is continuously increasing.
The evaluation of AAL projects usually focusses on analyses of usability, acceptance, and more recently, on the effects of such technologies [4]. However, little is known about the actual take-up of the AAL prototypes. The actual take-up or usage of an information and communication technology (ICT) system relates to the degree to which people make use of new technologies, and in the end, have adopted them. Information about the usage of an ICT system allows us to learn more about what users like and which preferences they have [5]. Usage data analysis is thus able to single out relevant features of a multi-feature system and facilitate the ability to assess the importance of newly developed features. Additionally, these data enable us to identify target groups interested in the system and provide an evidence base for the estimation of market potential.
The take-up or usage of ICT can be measured either explicitly, by collecting subjective data in surveys or focus groups [5,6,7], or implicitly, by collecting objective data, such as user interaction records with the system [5,8,9]. Survey data reflect self-reports from users but may lack accuracy and details due to a range of possible response biases (i.e., systematic errors in answering survey questions). Response biases, such as limitations in the ability to recall past system interactions [10] or responses reflecting overly positive self-descriptions [11], may lead to over- or under-reporting of system usage.
On the other hand, automatic recording of usage is less invasive. Automatically recorded usage data measure the use of a system without users always being aware of them being tracked, which is less prone to response bias than self-reporting. Usage data collection, however, also requires careful preparation to gain meaningful data, to avoid gathering a useless mass of data, and finally, to enable researchers to answer their research questions. Thus, careful planning and preparation of log data collection and analysis is essential to achieving good data quality and reliable results [5,12].
Previous research on automatic usage recording has either focused on technical frameworks for usage measurement [13,14,15] or the user grouping/classification of usage patterns [16,17,18]. Topics less developed in previous work on usage measurement include the selection of the logging component(s) and data source(s), the determination of logging capacity, and ethical and data privacy considerations.
Only a few AAL projects report the practice of automatic usage recording. In MobileSage, a technical framework for usage measurement was developed and used to adapt user profiles [8,19]. In Zentr AAL, the actual utilization of system components and features by different user groups was analyzed [20]. To increase the use of automatic usage measurement in AAL projects, a comprehensive usage data evaluation framework is required to guide systematic, comparable, and reproducible usage data logging, preparation, and analysis that covers technical, ethical, and data protection requirements.
This paper aims to contribute to the evaluation of AAL systems by proposing a standard for usage data evaluation and applying this for the first time to an AAL prototype, called CARIMO. The suggested usage data evaluation framework comprises all steps relevant for a comprehensive usage data collection and offers options for data collection and analysis. The framework and experiences in applying the framework for assessing the take-up of the AAL-prototype CARIMO aim to better guide future usage data collection and analysis in the research field of AAL. Researchers may use this framework for evaluating the take-up of AAL systems in a systematic, comparable, and reproducible manner.
The remainder of this paper is structured as follows: In Section 2, the AAL prototype CARIMO is presented. Section 3 considers the usage data evaluation and defines the framework for usage data evaluation in AAL projects. In Section 4, the application of the framework is presented. The paper closes in Section 5 with a discussion.

2. The AAL Prototype CARIMO for Home Care Service Users

According to the online catalogue for AAL products, the majority of AAL systems address “health & care,” “living & buildings,” or “safety & security.” Only a few relate to supporting and strengthening “vitality & abilities,” particularly the “physical abilities” of older people [21]. Maintaining the abilities of older people by fostering physical activities, such as walking or exercising, promotes healthy aging, delays the onset of disabilities, and postpones frailty [22,23]. Being physically active can maintain mobility and thus postpone the demand for long-term care. Hence, physical activity is an important factor for active aging, allowing older adults (even with dependencies) to age in place and participate in social life [22,24,25].
On that account, the “Active Assisted Living Programme” project “CareInMovement—CiM” developed CARIMO, an ICT-based system to promote physical activities for home care service users in Austria and Italy. Home care service users are older, care-dependent people living at home and receiving professional support to better cope with (instrumental) activities of daily living, such as getting dressed, bathing, cooking, or shopping. The CARIMO system was tested and evaluated in an eight-month field trial in Austria and Italy with test and control groups [26].
Contrary to most AAL systems that address care, safety, or mobility issues [21], the AAL prototype CARIMO was primarily designed to maintain the physical abilities of home care service users by fostering physical activities, such as walking or exercising. It complements the few apps available for seniors, such as “Fitivity Senior Fitness” or “Daily Senior Fitness Exercise,” as CARIMO was especially tailored to older, care-dependent people living at home. To assign suitable fitness exercises to this target group, sport scientists developed a test [27] to assess the physical abilities of the users. Amongst others, the data collection involved testing muscle strength (e.g., grip strength test) and endurance (e.g., chair rise test), as well as physical limitations. Based on the assessment conducted by care workers, sport scientists assigned the participants to one out of two training levels (physically frail, physically dependent) [27]. The customized training plans for each participant were derived from their training level, and if available, data on physical limitations. Unlike existing fitness apps, CARIMO additionally provided an information and entertainment section especially prepared for older, care-dependent people to provide stimuli for the body and the mind [26].
CARIMO differed from other AAL physical training prototypes for older people in three areas: intention for use, devices used, and the size and duration of the trial. First, a low-threshold exercise program developed for care-dependent people living at home offered a multi-component training program to improve coordination, strength, balance, and endurance [27] rather than training selected parts of the body, such as training using a treadmill [28] or ergometer [29,30]. Second, it applied two commercially available, comparatively low-cost components (with prices expected to fall over time) and thus did not require expensive fitness equipment, sensors, or computing hardware [28,29,30,31]. Finally, compared to most of the previous work reporting on short-term tests and small sample sizes [29,30,31], CARIMO was tested in an eight-month field trial with about 100 participants.
To incorporate the user requirements of home care service users, lead users of the target group were involved in all development phases according to the human-centered design (HCD) approach [32]. To ensure acceptable usability and user experience of this prototype, we especially paid attention to appropriate text and button sizes, labeling of icons, high contrast, text and audio output (for fitness exercises), and a simplified user interface (UI) [33].

2.1. Components of CARIMO

CARIMO consisted of two commercially available hardware components: (1) a tablet (Galaxy Tab A 2016, Samsung Electronics Co., Ltd., 129, Samsung-ro, Yeongtong-gu, Suwon-si, Gyeonggi-do, Korea) and (2) a bracelet with a fitness/activity tracker (Gear Fit2, Samsung Electronics Co., Ltd., 129, Samsung-ro, Yeongtong-gu, Suwon-si, Gyeonggi-do, Korea) (see Figure 1). Both components were selected according to predefined criteria. For the tablet these were: (i) display size of at least nine inches (aspect ratio of 16:10 or 16:9), (ii) Android 5 or higher, (iii) 3G or 4G internet connection, and (iv) a maximum price of EUR 320. The main criteria for the activity tracker were: (i) battery life of three to five days without using GPS, (ii) standalone operating mode (no permanent contact with a paired smartphone or tablet needed), (iii) waterproof or at least splash proof, (iv) adaptable UI and data transfer via Bluetooth or Adaptive Network Topology (ANT), and (v) maximum price of EUR 200. More details on the selection process for the fitness/activity tracker can be found in Willner et al. [34].
CARIMO consisted of a web app (for configuration purposes), an Android native app for the tablet, and a new watch face for the activity tracker. Existing systems (e.g., SimpliFlow—SimpliFlow International GmbH, Krone Platz 1, Klagenfurt, Austria—as a sports/training platform and SITOS—bit media e-solutions GmbH, Kärntner Straße 337, Graz, Austria—as an eLearning tool for online courses) were enhanced and integrated with new components developed within the project to provide a new solution to meet the aims of the AAL Programme. On the tablet, the CARIMO app was set as the home screen displaying the menu, such that the users always came back to the menu when pressing the home button. For the fitness bracelet, a digital CARIMO watch face was implemented, indicating the time and step count, as well as a shortcut for tracking activities, such as walking, via GPS. The activity tracker was used to measure steps and physical activities lasting more than ten minutes. These parameters were considered for rating the daily amount of physical activity of the users.

2.2. Fitness Program: Body-Related Features of CARIMO

The “body-related features” of CARIMO included four functions to improve or at least maintain physical abilities: (i) fitness exercises, (ii) activity overview, (iii) tip of the week, and (iv) recording of activities [26].
“Fitness exercises” provided health-enhancing physical exercises focusing on mobilization, stabilization, muscle strengthening, and endurance [27]. The 10-min training sessions changed daily, were based on the fitness level of the home care service users, and consisted of twelve exercises each. Each session was intended to be completed by the users on their own or together with a caregiver (care worker, family carer, or volunteer). For enhanced training planning functionalities (e.g., management of training levels and training plans), the SimpliFlow training management platform was integrated. More than 250 exercise videos were created by a sports scientist, with people aged 50–60 demonstrating how to perform the exercises. All exercise instructions were available as both text and audio output.
The “activity overview” enabled home care service users to keep track of their activities on a daily, weekly, and monthly basis. According to their physical achievements, users were awarded a bronze, silver, or golden trophy cup on a daily or weekly basis. Data from the fitness bracelet were automatically recorded and included in this section.
A “tip of the week” was meant to promote the use of the CARIMO tablet and to stimulate the users to keep physically active, either by suggesting simple changes for boosting activities outdoors (e.g., walking around a bench twice before sitting down) or by exercising at home.
The activities part of the “activities and notes” features enabled CARIMO users to manually enter activities into their “list of activities” if not already recorded by the activity tracker.

2.3. Information and Entertainment: Mind-Related Features of CARIMO

The “mind-related features” of CARIMO comprised five functions providing information and entertainment: (i) notes, (ii) newspapers, (iii) internet and games, (iv) a system manual, and (v) a function to arrange appointments [26].
CARIMO offered access to six regional newspapers and magazines in each country. Based on feedback from design workshops with lead users from the target group, we implemented an RSS (really simple syndication) reader with a large font size and high contrast to make information more legible than in traditional paper newspapers.
The feature “internet and games” provided an internet browser and up to six games (Math Quiz, Solitaire, 4-Pictures-1-Word, Crossword Puzzles, Connect Four, and Word Search) to provide an intellectual challenge and to train the handling of a tablet in general.
In addition to a series of visits of care workers to introduce CARIMO to the home care service users at the beginning of the trial, CARIMO also offered an interactive system tutorial to enable users to train themselves in the use of CARIMO. For this feature, the e-learning system SITOS was integrated.
Finally, the care network (consisting of care workers, family members, and volunteers) could arrange meetings via the CARIMO appointment feature. This allowed the home care service user to check the date and time of the next visit of a carer. The notes part of the “activities and notes” feature enabled the home care service users to leave a note for their caregiver.

3. Methods

3.1. Usage Measurement and Its Challenges

System usage data reflect the actual use of a system by measuring how often, when, and by whom an ICT-system was used [14,20]. Automatic recorded usage data are quantitative data that are automatically collected when a system is used [5,14,35]. Automatic usage data can be measured on two levels of the ICT system: on the system/application level via application usage measuring [36] and on the feature level via in-app logging of events [37]. Measuring usage data on the application level enables the comparison of different applications by assessing which applications are used, how often, and when they are used. Thus, usage prediction and recommendations for new applications can be derived [38]. For logging the usage data of mobile applications, a range of tools exists, e.g., AppTracker [36] or ProfileDroid [39]. The usage data at the application level, however, do not tell anything about which features or subfeatures of a system/application are used. On the feature level, insights into the take-up of application features can be gained, user profiles can be created or adapted, and information for further feature development and redesign can be derived [9,19,37]. Usage data on the feature level are measured by instrumenting source code to automatically log user interactions [14]. This can either be done using existing usage mining tools [40] or applying one’s own implementation [14].
No matter whether usage data are measured at the application or feature level, large amounts of data will be generated. If usage data evaluation is not planned carefully, a mass of data is produced, which challenges both the data processing and the generation of reliable results. Previous work addressed parts of the usage data evaluation process. They focused on tools and software for usage measurement [13,14,15,36]. Other work in the field of software development point to the need for estimating the expected amount of data and data transmissions beforehand, and to calculate the capacities required [41]. In addition to the amount of data to be processed, protocols also need to be defined on how to merge different data sources, e.g., to evaluate the usage of multicomponent systems.
For the purposeful analysis of usage data, the measures for “use” have to be appropriately defined, which may turn out not to be trivial. While in some cases, it is sufficient to know that a user just “clicked” a button, in other cases, it might also be of interest how long a certain application or feature has been used. This implies that not every user interaction may have a meaningful interpretation. First, raw usage records can contain corrupted data (errors in logging) or unintended duplicates (e.g., data like walking are logged automatically with an activity tracker and can also be entered by a user manually using an app interface), which require careful data preparation [42]. Second, measures are needed that provide meaningful information for answering the research questions. Examples for such measures are groupings/classifications of usage patterns that either result from data-driven or concept-driven approaches. In order to measure the actual take-up, an application or feature should be used repeatedly, which also needs to be reflected in an appropriate measure. Additionally, data sources may be needed for creating user profiles (e.g., sociodemographic characteristics of frequent users).
Usage data collection has been rarely discussed in the context of data protection and ethical considerations. Most of the previous literature does not consider the ethical appropriateness of the usage measurement approaches chosen [43]. Such considerations are, however, relevant as usage data may reveal personal details of user behavior while its collection is not obvious and visible for users. Data collection has to abide by the principle of data minimization according to the General Data Protection Regulation (GDPR) of the European Union. Taken together, as information on usage data collection, processing, and analysis is spread across different pieces of work, a framework providing a comprehensive approach comprising all relevant steps for usage data measurement would support future evaluation of the adoption of new AAL prototypes and systems.

3.2. The Usage Data Measurement Framework for AAL Systems

A general methodology, for instance, for web search transaction log analysis [42], is not available for usage data evaluation. Thus, based on the methodology proposed for web search transaction log analysis [42], the proposed usage data evaluation framework aims to ensure a systematic, comparable, and reproducible approach by addressing: (i) usage data logging, (ii) usage data preparation, and (iii) methods for usage data analysis of applications in the field of AAL (Figure 2). Iterations between the successive steps allow for going back to the previous step for revising previous decisions and refining the results. The framework also considers that usage can be measured and analyzed in various ways by offering different options for measurement and analysis.

3.2.1. Step 1: Usage Data Logging

The usage data logging step of the framework comprises: (i) the assurance of ethical appropriateness, particularly for ICT-supported solutions for older or vulnerable people; (ii) selection of evaluation objectives and decision on the logging levels; (iii) determination of the logging capacity and bandwidth; (iv) selection of the logging component(s)/data source(s); and (v) implementation of the data logging procedures.
Whenever (personal) data of users are logged, legal and ethical assessments are required. National and European regulations aim to improve data protection and user rights [44]. In addition, guidelines for ethical assessments of AAL projects (e.g., References [45,46,47]) seek to raise awareness of ethical issues. Ethical considerations usually relate to the type, duration, and frequency of data logging, data storage, and data protection to users’ rights to access, rectify, erase, withdraw, and transfer the data. In addition, it is important that researchers only collect as much data as is necessary to answer (pre-defined) research questions. To meet European legal requirements, users must give informed consent. To ensure the ethical appropriateness of AAL research projects, it is recommended to involve an ethics board or committee to assess the study design, including the aims, materials, and approaches for data collection and methods for the analyses.
Evaluation objectives determine the level of usage data collection. If the research focus is on the general use of a system/application, then logging can be restricted to the application level. However, if information about the use of features is needed, then logging at the feature level is required.
Considerations of the capacity have to address both the logging duration and the maximum number of users to be logged by the system. For feature-level logging, the logging granularity, i.e., the maximum number of sub-levels per feature, has to be determined. Based on the logging level, the logging capacity and bandwidth can be determined such that the logging component can be selected, configured, and if necessary, integrated into the system of which the usage data should be analyzed. To enable data logging, the logging component has to be configured, and if necessary, integrated into the system of which the usage data will be analyzed. For different logging components, see References [14,36,39,40].
For a multi-component system, one logging component or data source may not be sufficient. Particularly, AAL systems aim to combine different systems and technologies. Depending on the technologies used and the type of the feature, logging a feature’s usage may differ on different devices. For example, feature usage on a tablet is not recorded in the same way as, for example, the steps on an activity tracker. While logging of feature usage requires user interaction and user interaction logging, steps are recorded automatically using sensor data without user interaction on the device. Thus, for multi-component systems, where different data sources are needed, a user identifier, such as a universally unique identifier (UUID) [48] connecting the data to the single user, has to be implemented into each of the data sources.
The definition of use may depend on the features of a system an on the research questions. For some features, clicking on a button (e.g., to view the weather forecast) may count as “use.” Other features may require sophisticated approaches to assess the take-up. This is the case for features that aim to incentivize a user’s activities, such as doing exercises. A simple count of clicking on the program will thus not generate meaningful data. In such cases, additional information from time stamps or other components could be useful.
In addition, if user profiles are of interest, further information about the characteristics of the users of a system will be needed for the analysis [49], e.g., demographic data, such as gender or age. In both cases, the storage and data transmission of these data need to be planned and implemented beforehand [41]. Finally, if all needed data sources are identified, the logging procedure using one or more logging components must be implemented.

3.2.2. Step 2: Usage Data Preparation

The focus of this step is in usage data extraction, enrichment, and pre-processing. Subsequent analyses depend on the quality of this step. Depending on the logging component, data extraction can either be done through existing application programming interfaces (APIs) of the logging component, or if accessible, through the component’s database. If additional data sources are needed for the analysis (e.g., data generated by an activity tracker), these data must also be extracted. The extracted raw logging data require appropriate data pre-processing procedures to access the information represented by the data. In the first pre-processing step, in line with the research interest, the different sources are merged together and brought into a common structure using the UUID. Subsequently, pre-processing procedures for data cleaning help to identify corrupted data and unintended duplicates [42,50].

3.2.3. Step 3: Methods for Usage Data Analysis

Automatically generated usage data can be analyzed in a variety of ways, for example, descriptive statistics [51], analytical models [9,18], or using expert-driven conceptual approaches [17]. Descriptive statistics provide information on the frequency of use and measures of central tendency, e.g., median or mean [51] for a certain point in time or over a period. For the analysis of usage behavior, user workflow or process mining techniques [9,52,53] or time series analysis [54] can be applied. Another method for analyzing usage patterns and usage behavior is user grouping. User groupings can be defined either in a data-driven manner, e.g., by using data mining techniques, such as Jenks’s natural breaks [55], or in a conceptually-driven manner, e.g., based on expert knowledge [17] or previous work. Common groupings address the frequency of use [56], duration of use [16], or usage phases [17]. A comparatively simple grouping is a dichotomous “use or non-use” classification. The grouping granularity is dependent on the duration of the usage data logging. If only a week is recorded, the granularity might be hourly or daily, while if several months or years are recorded, the granularity might also be weekly, monthly, or yearly.
For assessing the repeated use of an application feature, the retention rate indicator is a useful measure. The retention rate measures the percentage of users who continue using an application over a certain time period [57,58].

4. Results: The Usage of CARIMO

The proposed framework was applied to evaluate the usage of the AAL prototype CARIMO in an 8-month field trial with test and control groups in Austria and Italy; the results are presented in this section.

4.1. The CARIMO Sample

Table 1 provides the sample description. The field trial started with 104 participants (64 in Austria and 40 in Italy) and ended after eight months with 85 participants (54 in Austria and 31 in Italy). After eight months, 82% of home care service users were still participating. The decrease in trial participants was mainly due to hospitalization, admission to a care home, or lack of interest [26]. About three quarters of participants were women. The largest age group was participants between 70 and 79 years. Twenty percent were 80 years and older. About 60% of the trial participants were Austrians and 40% Italians. With respect to self-assessed care dependency, about 70% needed personal help or had difficulties coping with (instrumental) activities of daily living ((I) ADL) [59].

4.2. Usage Data Logging for CARIMO

4.2.1. Approval of the Ethical Appropriateness of the Usage Data Collection

During the field trial, CARIMO usage data were collected as pseudonymized personal data as every user was assigned an identifier that linked the data of the logging component to the other system components and personal characteristics of the participants. The CARIMO system stored usernames that were fed into selected features, e.g., for greeting the user when starting the fitness app (“Hello, Peter!”).
Independent experts in ethics and privacy policy were consulted to assess the data processing procedures. The CiM project and its evaluation design (including the informed consent) were reviewed and approved by the independent ethics committee at the University of Salzburg (EK-GZ 30/2016). Additionally, the proposed data collection project was submitted to the Austrian data protection authority and registered to the Austrian data processing register (DVR: 4008479/006).

4.2.2. Evaluation Objectives and the Selection of Logging Levels

The evaluation of the take-up of CARIMO mainly aimed to measure the use of both the CARIMO app and its body- and mind-related features in two European countries, Austria and Italy. Accordingly, we decided to log at the CARIMO feature level, including the first subfeature level (see Section 3.1). For example, we selected the logging level to get access to data on, for example, the use of the “internet & games” feature and the subfeature “games.” However, further levels of this feature, such as specific games, were not of interest and hence not selected. Thus, the logging granularity was set on the first subfeature level.

4.2.3. Determination of Logging Capacity and Bandwidth

The logging capacity calculations considered two dimensions: the number of users (more than 100), and the duration of the trial (eight months + extension options). Thus, in addition to a monthly bandwidth of approximately 70 to 80 MB per user, for the CARIMO app, we calculated 5 MB for the logging data per user and month.

4.2.4. Selection of the Logging Component(s) and Data Source(s)

Since there is a variety of logging tools available (see Section 3.2.1), we decided not to go for a time-consuming proprietary development. We selected a logging component that met the following system and privacy requirements: (i) supporting mobile application logging, enabling (ii) the research team to get access to the raw usage data for an independent usage data analysis, and (iii) self-hosting the data since third-party hosting did not fulfill data privacy requirements.
According to Kumar and Thakur [40] and our own investigations, only two tools (Piwik and Google Analytics 360) were able to support mobile application usage logging at the time of decision in 2016. Piwik (today known as Matomo, InnoCraft, 150 Willis St, Wellington, New Zealand) was preferred to Google Analytics 360 as its use was free of charge and allowed self-hosting with access to raw usage data. For the server, we used Piwik version 3.0.1. On the client level (CARIMO tablet), we started data logging with Piwik SDK for Android 0.0.3. In month four, a new version of CARIMO including additional games and bug fixes was released. This release was also used for a client update of Piwik to version 2.0.0.
For the fitness program of CARIMO, Piwik usage measurement alone was not sufficient for two reasons. The operating system of the activity tracker (Tizen) was not supported by Piwik. Thus, the use of the activity tracker (e.g., recording of steps and activities) could not be tracked by Piwik. We accessed this information indirectly via the CARIMO system database where these data were stored. Second, the duration and the number of completed exercises on the tablet were difficult to measure and extract. As the determination of a completed exercise involved considering many exceptions (e.g., skipping or pausing of exercises), the recorded time per exercise stored in the CARIMO system database, excluding, for example, skipped exercises and pauses, was used to measure the number of exercises performed.
As we were interested in comparing the take-up of CARIMO and its features between two countries (Austria and Italy), we imported this information from the CARIMO system database, where the place of residence of each user was stored for support reasons. As different data sources (Piwik and the system’s database) were used, we implemented a UUID in each data source, which allowed for merging the data into one data set.

4.2.5. Implementation of the Data Logging Procedures

Integrating Piwik for usage data logging into CARIMO enabled the logging of: (i) the starting and stopping of the CARIMO activity, (ii) changes in the content (e.g., for Android displaying and removing fragments), and (iii) every interaction with user interface components, such as the use of buttons, input fields, or check boxes.
In alignment with our research interests, usage measurement for CARIMO had to provide information on the characteristics of the users and the frequency of their use of the CARIMO app and its features. Thus, we implemented the logging of: (i) a unique user identification number (UUID) (who); (ii) type of interaction, such as click, touch, or scroll (what); (iii) UI component that was interacted with (where); and (iv) date and time of the interaction (when). On this basis, further information, such as how often the user accessed the feature, could be derived. The interaction was identified using a unique page/view identifier (e.g., “/main” for the home screen) combined with a unique user interface component identifier (e.g., “/main/btnAdvice” for the tip of the week button in the home screen). Thus, clicking somewhere on the screen resulted into no further action and was not recorded.

4.3. Usage Data Preparation

4.3.1. Data Extraction and Enrichment

We extracted the app usage data from Piwik. The data for calculating the activity tracker usage were obtained from the system’s database. Additional data for determining the exercise duration and the number of completed exercises, as well as the country affiliation of each user, were extracted from the system’s database. Both data from Piwik and the system’s database were extracted, and a pseudonym was applied using the UUID.

4.3.2. Data Pre-Processing

The extracted raw data from the system’s database and Piwik were merged together into a common structure using the UUIDs. Next, corrupted data were removed by sorting each field in sequence. In the final pre-processing step, unintended activity duplicates were removed whereby the automatically recorded data were preferred to the manually recorded data.

4.4. Methods for Usage Data Analysis

4.4.1. Definition of Usage Measures

For the tablet, the usage frequency was measured using the Piwik metrics “visits” and “unique page views” [60]. A “visit” starts with the first use of CARIMO on a new day or after a pause of at least 30 min. A “unique page view” represents the number of “visits” that included a certain CARIMO feature or subfeature. If a feature or subfeature was used multiple times during a “visit,” it was only counted once. As not every interaction with the app was expected to be a meaningful measure of use, we set certain time thresholds. For exercises, “time spent on training” was only counted if the screen time per exercise was 30 s or longer. This timespan was set based on the time required to perform ten iterations of a simple exercise, such as shoulder shrugs. Originally, we planned to link the pulse measurement of the activity tracker to the exercises. This did not work satisfactorily in the pre-trial phase, as many women had such slender wrists that the necessary skin contact for a reliable pulse measurement was not attainable. As other work also reported inaccuracies [61], we decided to not use the pulse measurement of the activity tracker. The “number of performed exercises” captured the total number of exercises lasting 30 s or longer. As the activity tracker generated data automatically without user interaction, we decided to calculate “usage days” instead of “visits.” The measure “recorded activities per day” was introduced for activities logged by the fitness tracker for physical activities lasting more than ten minutes or activities that were entered manually. A monthly analysis for the 8-month field trial seemed fine-grained enough to describe usage behavior appropriately.

4.4.2. Definition of CARIMO User Groups

User groups for CARIMO were calculated for the test group in total, as well as on a country basis. According to previous usage measurement for AAL technologies [20], the framework used four user groups for assessing the intensity of using a system.
frequent users: participants who used a CARIMO feature more often than a regular user.
regular users: participants who used a CARIMO feature on a regular basis.
infrequent users: participants who used a CARIMO feature less often than a regular user.
non-users: participants who did not use a CARIMO feature at all.
Particularly, the characteristics “frequent,” “regular,” and “infrequent” needed thresholds to assign users to these groups. For the thresholds below, we applied an expert-knowledge-driven approach.

Definition of Usage Groups for Body-Related Features of CARIMO

Body-related features of CARIMO comprised two functions “fitness exercises” (training program personalized by sport scientists based on assessments of physical abilities by care workers; see Section 2) and “activities” (manually entered into the system by the users or automatically recorded by the activity tracker). We specified thresholds that informed about the frequency of use based on the recommendations for health-enhancing physical activity [62]. In general, for healthy adults aged 65 and above, it is recommended that they perform physical activities of at least 150 min at a moderate intensity or 75 min of vigorous intensity per week [62]. Moderate-intensity corresponds to tasks requiring effort of three to six “metabolic equivalents” (METs), e.g., cycling for pleasure or transport (≤10 mph), and vigorous intensity corresponds to more than six METs, e.g., cycling fast (>10 mph) [63].
Table 2 shows the user group definition for the body-related features (fitness exercises, activities, and total). Based on the WHO recommendations and considering the health conditions of our target group, we defined a regular user as a user who does “fitness exercises” between 4 and 7 times a month (60 and 105 min) and performs between 9 and 24 “activities” a month (225 and 600 min). One run of “fitness exercises” was estimated with 15 min and one activity with 25 min. Consequently, a frequent user was defined per the above guidelines and an infrequent user was below the values of a regular user.

Definition of Usage Groups for Mind-Related Features of CARIMO

Mind-related features of the CARIMO app covered “newspapers” and “internet and games,” where “newspapers” was a single feature and “internet and games” consisted of two subfeatures. The categories were defined specifically for this study using the expertise of end-user experts, i.e., home care providers. They assumed that people interested in “newspapers” (regular users) will use this feature at least 10 times a month. They also stated that highly interested people (frequent users) will use this feature at least 20 times a month. Accordingly, an infrequent user was defined as being below these values (Table 3—Newspapers).
For many CARIMO users, it was the first time in their lives that they had internet access. About two thirds of the users did not use the internet before [64]. Thus, the end user experts expected that a regular user would use the internet between 7 and 15 times a month (Table 3—Internet). Participants who used the internet at least 16 times were defined as frequent users. Usage between 1 and 9 times was classified as infrequent.
With respect to “games,” a regular user was defined as a user who used games between 8 and 14 times a month. Here too, frequent users were defined above and an infrequent below these values (Table 3—Games). Table 3 shows the usage groups for “newspapers” and “internet & games (I&G)” and the aggregation of both mind-related features.

4.4.3. Definition of the Retention Rate

The average retention rate for mobile apps after three months is 29% [65]. The adherence of older people to home-based fitness exercise programs is between 27% and 42.6% [24,66,67]. Thus, a monthly retention rate higher than 45% for body- and mind-related features was defined as a target for using CARIMO.
We calculated two retention rate indicators. The first indicator ( R R v 1 ) included all retaining users (frequent, regular, and infrequent users) (Equation (1)). For the second indicator ( R R v 2 ), a more conservative approach was applied. Only participants who used a system frequently or regularly were included (Equation (2)).
  R R v 1 =   retained   users   frequent + regular + infrequent total   users × 100
  R R v 2 =   retained   users   frequent + regular   total   users × 100
The retention rates of CARIMO (Table 4 R R v 1 and R R v 2 ) were calculated for the test group in total and on a country basis.

4.5. Results of the CARIMO Usage Data Analysis

4.5.1. Usage Frequencies of CARIMO

Table 4 shows the participant flow and the median use of CARIMO (total and per country). The take-up of the CARIMO app remained quite stable over time while the median use per user of the activity tracker declined in the last two months.
Country-specific analysis revealed different use patterns over time in Austria and in Italy. In the first four months, the median use of the tablet in Austria was twice as high as in Italy. In month six, the use of the CARIMO app in Italy even slightly exceeded the Austrian level. The activity tracker use was higher in Austria than in Italy, while in the last two months, it decreased significantly in both countries (Table 4).

4.5.2. User Groups of CARIMO: Intensity of Use

Table 5 shows that, on average, about two-thirds of the trial participants used the body-related features of CARIMO on a frequent or regular basis. Toward the end of the field trial (month 8), the use decreased and the share of non-users increased. A comparison between Austria and Italy revealed that, on average, the usage of CARIMO in Austria was about 30% higher than in Italy. Furthermore, the proportion of frequent users over time was higher in Austria than the proportion of regular users. In Italy, this was only the case in months one, two, and seven. In general, there were more non-users for body-related features in Italy than in Austria.
With respect to mind-related features, Table 5 indicates that between 52% and 66% of the users used these features on a frequent or regular basis. In months six and seven, usage increased, which reflects the rollout of three new games in month six. In addition, for mind-related features, the share of non-users increased toward the end of the field trial (month 8). At the beginning of the trial, Italians used these features less often than Austrians. Starting with month four, the use in Italy approached the Austrian level. For the “information and entertainment” function in both countries, the proportion of regular users over time was higher than the proportion of frequent users.
The body- and mind-related features of CARIMO were perceived equally well by its users. However, body-related features were more frequently used (between 62% and 75% frequent and regular users) than mind-related features (between 52% and 66% frequent and regular users).

4.5.3. Retention Rates of CARIMO

Table 5 also shows the retention rates. On average, more than 80% of the users were retained ( R R v 1 ), while in Austria, the share was about 90%, and in Italy it was about 70%. Seventy percent of all users used the fitness program to the expected extent of 45% ( R R v 2 ). Country-specific patterns remained for the use of the body-related features of CARIMO. Retention rates in Austria (≈80%) were higher than in Italy (≈50%). For mind-related features, on average, 88% of the users were retained ( R R v 1 ) with no differences between Austria and Italy. Overall, on average, 57% used CARIMO’s mind-related features to the expected extent of 45 % ( R R v 2 ). However, in contrast to R R v 1 , for R R v 2 , there was a difference between Austria (≈61%) and Italy (≈50%), which can be explained by the higher proportion of infrequent users in R R v 1 in Italy.

5. Discussion and Conclusions

AAL projects aim to make a positive change in older people’s lives by facilitating access to new technologies. To assess whether the target groups effectively adopt the developed prototypes, automatically recorded usage data are a valuable source of information. However, publications on the actual use of AAL-technologies are rare, as are publications that apply usage data for other types of analysis, such as usability assessments or effectiveness analysis. We thus aimed to support the usage data assessment for AAL solutions by both providing a comprehensive approach for the evaluation and by reporting on the results of applying this approach on a complete and fully operational AAL prototype, called CARIMO.
We presented a usage data evaluation framework that offers a systematic, comparable, and reproducible process to improve usage data logging, preparation, and analysis in (longer) field trials of AAL prototypes. In comparison to previous work [8,14,17,18,36,37], it covers all steps relevant for a comprehensive usage data assessment and, by offering options for usage data evaluation, allows adjustments according to the research objectives or technical requirements of the AAL solution. Data collection and analysis is guided by a step-by-step process to generate awareness and to prepare for important decisions and point to tasks that may be easily overlooked (e.g., calculating logging capacity and bandwidth, and preparation for ethical clearance). The presented framework thus seeks to support preparing and conducting a systematic usage data measurement for AAL projects.
Usage data can be recorded without the knowledge of the users and may thus serve as important information on the actual use of an AAL system. On the other hand, usage data in general have become a “currency” in many areas of the ICT economy. Access to many apps and ICT services seem to be for free, but user behavior is being tracked and data are stored or used and sold for purposes that are often not transparent to users. To balance both interests, i.e., generating knowledge and assuring privacy, ethical approval of usage data collection and analysis is important for responsible research in the field of AAL. For this reason, ethical considerations have been included into the usage data evaluation framework.
In a first application, the framework’s systematics was used to measure the take-up of the AAL prototype CARIMO, an ICT-supported fitness and entertainment program for older, care-dependent people, in an eight months’ field trial. In general, logging generates a mass of data, which need to be prepared for a meaningful analysis. “One-size-fits-all” definitions do not always work. Thus, we suggested a context-dependent classification scheme. The framework offers options to decide on tools, measures, and groupings depending on the research aims and technologies to be assessed. For example, it proposes both data-driven and concept-driven approaches for usage groupings, with the latter based on expert knowledge and/or previous work. In the case of CARIMO, a concept-driven approach was taken to define user groups for the body-related features of CARIMO as well-accepted globally recommended levels of physical activities were available for the group of older adults.
Since there are differences in how people use an app, it is important to define what usage means in a certain context e.g., a person just having a look at fitness exercises versus performing those exercises for several minutes. Such definitions set beforehand may avoid misinterpretations of the data. A particular challenge was to assess the use of the “fitness exercise” feature. As there were no technologies (such as 3D cameras) that could ensure that the exercises had actually been performed, we had to rely on a proxy measure. In order to avoid overestimating the use of the “fitness exercise” feature by, for example, simply counting the clicks on the training program, we only counted an exercise if a predefined minimal screen time had been exceeded. For future studies on the use of “fitness exercise” features that also aim to measure the quality of the interaction, a 3D camera system for gesture recognition, such as the Kinect or Orbbec sensor [68], may turn out to be useful. Such changes in technologies could easily be addressed by the usage measurement framework as it allows for the adoption of additional data sources and usage measures with respect to the data of the new sensor.
In terms of insights gained by evaluating an AAL system’s use, we observed that a significant share of the trial participants had used CARIMO for eight months on a regular basis. The goal of using CARIMO to keep care-dependent people physically and mentally active was thus achieved. This seems remarkable given the vulnerable target group and their restricted experience with ICT-based solutions. In addition, we learned that incentives, such as new games (introduced in month 6), increased the usage during the field trial. At the end of the field trial, however, for both body- and mind-related features, the usage decreased in Austria and Italy, which may have been due to trial participants preparing themselves for the end of the field trial or due to users who just lost interest in the system over time.
The application of R R v 2 (only including frequent and regular users) on mind-related features showed that only considering R R v 1 could be misleading due to the high proportion of infrequent users in Italy. If the proportion of infrequent users is high, the calculation of R R v 2 allows for better insight into the system’s use.
Compared to existing work, the presented framework was applied to an AAL project that involved many users over a longer period of time [14]. As AAL systems normally consist of more than one device, the framework foresees the possibility to integrate more than one device [17]. In its first application on CARIMO, it was applied to two devices: a tablet and a fitness tracker.
This study has some limitations. First, as detailed above, the definition of an appropriate measure for the usage of an app or feature that neither overestimates nor underestimates the use is not always trivial. In particular, features that require users to engage in activities that are encouraged by an app but cannot be further tracked, e.g., fitness exercises on a smartphone or tablet app, rely on proxy measures, which may only approximate the actual usage as intended by the developers of an app. If actual tracking is the main aim, then other technologies, such as 3D cameras or reliable body sensors, may be useful for future work given that users’ privacy can still be guaranteed. Second, by using the proposed usage measurement framework, we were able to assess which features were used, how often, when, and by whom. We were not able to make statements on how or why the system was used. To learn more about the characteristics of users in AAL projects, future work could aim to combine logging data with survey data. Finally, while the usage data evaluation framework was designed to guide systematic, comparable, and reproducible usage data evaluation in the AAL field, the general applicability of the framework has yet to be validated.

Author Contributions

Conceptualization, methodology, validation, formal analysis: C.S. and B.T.; software and data curation: H.R.; writing—original draft preparation: C.S. supported by B.T.; writing—review and editing: B.T. and H.R.; project administration: C.S.; funding acquisition: C.S. and B.T. All authors have read and agreed to the published version of the manuscript.


This work was supported by the Active Assisted Living Programme grant number AAL 2014-1-021. It received funding from the Active Assisted Living Programme, cofounded by the European Commission, National Funding Authorities of Austria, Italy, and Switzerland, and the individual project partners. Collaborating partners in the CiM project were Salzburg Research Forschungsgesellschaft m.b.H. (lead partner), Vienna University of Economics and Business, Paris Lodron University of Salzburg, ilogs mobile software GmbH, bit media e-solutions GmbH, ALDIA Cooperativa Sociale, Hilfswerk Salzburg, and Eichenberger-Szenografie. The CiM project ran from October 2015 until September 2018. The Research Institute for Economics of Aging at the Vienna University of Economics and Business received additional funding from the Social Fund Vienna (FSW). Open Access Funding by Fachhochschule Wiener Neustadt GmbH.


Special thanks go to Melanie Griesser (Salzburg Research Forschungsgesellschaft mbH) who supported the usage data preparation, Marlene Blüher (Research Institute for Economics of Aging, WU Vienna University of Economics and Business) for data collection and supporting the sample description, Sonja Jungreitmayr and Susanne Ring-Dimitriou (Interfaculty Department of Sport & Exercise Science, University of Salzburg) for feedback on the thresholds for the user groups of the fitness program, and Taylor Rosemeyer (Research Institute for Economics of Aging, WU Vienna University of Economics and Business) for proofreading the text.

Conflicts of Interest

The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, nor in the decision to publish the results.


  1. AAL Joint Programme Brussels; The Central Managment Unit (CMU) AAL Joint Programme: Brussels, Belgium, 2013; p. 172.
  2. AAL Joint Programme Brussels; AAL Programme: Brussels, Belgium, 2016; p. 40.
  3. AAL Association AAL Programme. Available online: (accessed on 20 March 2019).
  4. Van Grootven, B.; Van Achterberg, T. The European Union’s Ambient and Assisted Living Joint Programme: An evaluation of its impact on population health and well-being. Health Inform. J. 2016, 25, 27–40. [Google Scholar] [CrossRef] [Green Version]
  5. Agosti, M.; Crivellari, F.; Di Nunzio, G.M. Web log analysis: A review of a decade of studies about information acquisition, inspection and interpretation of user interaction. Data Min. Knowl. Discov. 2012, 24, 663–696. [Google Scholar] [CrossRef]
  6. Ray, P.P. IoT: An Architectural Framework for Monitoring Health of Elderly People. In Proceedings of the 2014 International Conference on Science Engineering and Management Research (ICSEMR), IEEE, Chennai, India, 27–29 November 2014; pp. 3–5. [Google Scholar]
  7. Mitzner, T.L.; Boron, J.B.; Fausset, C.B.; Adams, A.E.; Charness, N.; Czaja, S.J.; Dijkstra, K.; Fisk, A.D.; Rogers, W.A.; Sharit, J. Older adults talk technology: Technology usage and attitudes. Comput. Hum. Behav. 2010, 26, 1710–1721. [Google Scholar] [CrossRef] [Green Version]
  8. Burns, W.; Chen, L.; Nugent, C.; Donnelly, M.; Skillen, K.L.; Solheim, I. Mining usage data for adaptive personalisation of smartphone based help-on-demand services. In Proceedings of the 6th International Conference on PErvasive Technologies Related to Assistive Environments—PETRA ’13, Rhodes, Greece, 29–31 May 2013; pp. 1–7. [Google Scholar]
  9. Rubin, V.A.; Mitsyuk, A.A.; Lomazova, I.A.; van der Aalst, W.M.P. Process mining can be applied to software too! In Proceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement—ESEM ’14, Torino, Italy, 18–19 September 2014; pp. 1–8. [Google Scholar]
  10. Clarke, P.M.; Fiebig, D.G.; Gerdtham, U.-G. Optimal recall length in survey design. J. Health Econ. 2008, 27, 1275–1284. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  11. Paulhus, D.L. Socially desirable responding: The evolution of a construct. In The Role of Constructs in Psychological and Educational Measurement; Braun, H.I., Jackson, D.N., Wiley, D.E., Eds.; Lawrence Erlbaum Associates: Mahwah, NJ, USA; London, UK, 2002; pp. 49–69. [Google Scholar]
  12. Agosti, M.; Di Nunzio, G.M. Gathering and Mining Information from Web Log Files. In Digital Libraries: Research and Development. DELOS 2007; Thanos, C., Borri, F., Candela, L., Eds.; (Lecture Notes in Computer Science); Springer: Berlin/Heidelberg, Germany, 2007; Volume 4877, pp. 104–113. [Google Scholar]
  13. Lettner, F.; Holzmann, C. Automated and unsupervised user interaction logging as basis for usability evaluation of mobile applications. In Proceedings of the 10th International Conference on Advances in Mobile Computing & Multimedia—MoMM ’12, Bali, Indonesia, 3–5 December 2012; ACM Press: New York, NY, USA, 2012; p. 118. [Google Scholar]
  14. Jensen, K.L.; Larsen, L.B. Evaluating the usefulness of mobile services based on captured usage data from longitudinal field trials. In Proceedings of the 4th International Conference on Mobile Technology Applications, and Systems and the 1st International Symposium on Computer Human Interaction in Mobile Technology—Mobility ’07, Singapore, 10–12 September 2007; ACM Press: New York, NY, USA, 2007; p. 675. [Google Scholar]
  15. Noldus, L.; Loke, B.; Kelia, M.; Spink, A. Automated Mobile User Experience Measurement: Combining Movement Tracking with App Usage Logging. In Proceedings of the Creating the Difference: Proceedings of the Chi Sparks 2014 Conference, The Hague, The Netherlands, 3 April 2014; pp. 31–34. [Google Scholar]
  16. Epstein, D.A.; Kang, J.H.; Pina, L.R.; Fogarty, J.; Munson, S.A. Reconsidering the device in the drawer. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing—UbiComp ’16, Heidelberg, Germany, 12–16 September 2016; pp. 829–840. [Google Scholar]
  17. Meyer, J.; Wasmann, M.; Heuten, W.; El Ali, A.; Boll, S.C.J. Identification and Classification of Usage Patterns in Long-Term Activity Tracking. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems—CHI ’17, Denver, CO, USA, 6–11 May 2017; pp. 667–678. [Google Scholar]
  18. Zhao, S.; Ramos, J.; Tao, J.; Jiang, Z.; Li, S.; Wu, Z.; Pan, G.; Dey, A.K. Discovering different kinds of smartphone users through their application usage behaviors. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing—UbiComp ’16, Heidelberg, Germany, 12–16 September 2016; pp. 498–509. [Google Scholar]
  19. Skillen, K.-L.; Nugent, C.D.; Donnelly, M.P.; Chen, L.L.; Burns, W.; Solheim, I. A Novel Approach for the Population and Adaptation of Ontology-Based User Profiles. In Ubiquitous Computing and Ambient Intelligence. Personalisation and User Adapted Services. 8th International Conference, UCAmI 2014, Belfast, UK, 2–5 December 2014; Springer: Cham, Switzerland, 2014; pp. 280–287. ISBN 978-3-319-13101-6. [Google Scholar]
  20. Schneider, C.; Maringer, V.; Rieser, H.; Venek, V.; Krainer, D. Nutzungshäufigkeit von „meinZentrAAL. In Smartes Betreutes Wohnen Nutzung, Systemakzeptanz und Wirkungen von „meinZentrAAL“; Trukeschitz, B., Schneider, C., Ring-Dimitriou, S., Eds.; Books on Demand: Norderstedt, Deutschland, 2018; pp. 123–177. ISBN 978-3-744-88233-0. [Google Scholar]
  21. University of Innsbruck AAL Products—The Online Catalogue for Assistive and Smart Technologies. Available online: (accessed on 20 March 2019).
  22. Mollenkopf, H. The significance of out-of-home mobility in modern society. In Enhancing Mobility in Late Life; Mollenkopf, H., Fiorella, M., Ruoppila, I., Széman, Z., Tacken, M., Eds.; ISO Press: Amsterdam, The Netherlands, 2005; pp. 1–9. ISBN 978-1-58603-564-8. [Google Scholar]
  23. Paganini-Hill, A.; Greenia, D.E.; Perry, S.; Sajjadi, S.A.; Kawas, C.H.; Corrada, M.M. Lower likelihood of falling at age 90+ is associated with daily exercise a quarter of a century earlier: The 90+ Study. Age Ageing 2017, 46, 951–957. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  24. Geraedts, H.A.; Zijlstra, W.; Zhang, W.; Bulstra, S.; Stevens, M. Adherence to and effectiveness of an individually tailored home-based exercise program for frail older adults, driven by mobility monitoring: Design of a prospective cohort study. BMC Public Health 2014, 14, 570. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  25. Webber, S.C.; Porter, M.M.; Menec, V.H. Mobility in older adults: A comprehensive framework. Gerontologist 2010, 50, 443–450. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  26. Trukeschitz, B.; Blüher, M. Measuring the Effectiveness of ‘CARIMO’, an ICT-Supported Fitness and Entertainment App for Home Care Recipients: Study Protocol and Survey Data Collection; Discussion Paper of the AAL-project CareInMovement (CiM) No. 2/2018 and Discussion Paper No. 2/2018 of the Research Institute for Economics of Aging; Vienna University of Economics and Business (WU): Vienna, Austria, 2018. [Google Scholar]
  27. Jungreitmayr, S.; Ring-Dimitriou, S. Training Concepts Report; Deliverable of the AAL-Project CareInMovement (CiM) No. 5: Salzburg, Austria, 2016. [Google Scholar]
  28. Doniger, G.M.; Beeri, M.S.; Bahar-Fuchs, A.; Gottlieb, A.; Tkachov, A.; Kenan, H.; Livny, A.; Bahat, Y.; Sharon, H.; Ben-Gal, O.; et al. Virtual reality-based cognitive-motor training for middle-aged adults at high Alzheimer’s disease risk: A randomized controlled trial. Alzheimer’s Dement. Transl. Res. Clin. Interv. 2018, 4, 118–129. [Google Scholar] [CrossRef]
  29. Arlati, S.; Colombo, V.; Spoladore, D.; Greci, L.; Pedroli, E.; Serino, S.; Cipresso, P.; Goulene, K.; Stramba-Badiale, M.; Riva, G.; et al. A Social Virtual Reality-Based Application for the Physical and Cognitive Training of the Elderly at Home. Sensors 2019, 19, 261. [Google Scholar] [CrossRef] [Green Version]
  30. Baldassini, D.; Colombo, V.; Spoladore, D.; Sacco, M.; Arlati, S. Customization of domestic environment and physical training supported by virtual reality and semantic technologies: A use-case. In Proceedings of the 2017 IEEE 3rd International Forum on Research and Technologies for Society and Industry (RTSI), Modena, Italy, 11–13 September 2017; pp. 1–6. [Google Scholar]
  31. Konstantinidis, E.I.; Billis, A.; Hlauschek, W.; Panek, P.; Bamidis, P.D. Integration of cognitive and physical training in a smart home environment for the elderly people. Stud. Health Technol. Inform. 2010, 160, 58–62. [Google Scholar] [PubMed]
  32. ISO. ISO 9241-210:2010 Ergonomics of Human-System Interaction—Part 210: Human-Centred Design for Interactive Systems; ISO: Geneva, Switzerland, 2010; p. 32. [Google Scholar]
  33. Arch, A. Web Accessibility for Older Users: A Literature Review. Available online: (accessed on 30 November 2019).
  34. Willner, V.; Rieser, H.; Venek, V.; Schneider, C. Selection and assessment of activity trackers for enthusiastic seniors. In Proceedings of the ICT4AWE 2017—Proceedings of the 3rd International Conference on Information and Communication Technologies for Ageing Well and e-Health; SCITEPRESS, Setúbal, Portugal; 2017; pp. 25–35. [Google Scholar]
  35. Schneider, C.; Maringer, V.; Rieser, H.; Henneberger, S. Methode für die Evaluierung des Nutzungsverhaltens von „meinZentrAAL“. In Smartes Betreutes Wohnen Nutzung, Systemakzeptanz und Wirkungen von „meinZentrAAL“; Trukeschitz, B., Schneider, C., Ring-Dimitriou, S., Eds.; Books on Demand: Norderstedt, Deutschland, 2018; pp. 59–62. ISBN 978-3-744-88233-0. [Google Scholar]
  36. Bell, M.; Chalmers, M.; Fontaine, L.; Higgs, M.; Morrison, A.; Rooksby, J.; Rost, M.; Sherwood, S. Experiences in Logging Everyday App Use. Digit. Econ. 2013, 13, 4–6. [Google Scholar]
  37. Andrei, O.; Calder, M.; Chalmers, M.; Morrison, A.; Rost, M. Probabilistic formal analysis of app usage to inform redesign. In Lecture Notes in Computer Scicience; (Including Subseries Lecture Notes in Artificial Intelligence Lecture Notes in Bioinformatics); Springer: Cham, Switzerland; Volume 9681, pp. 115–129.
  38. Cao, H.; Lin, M. Mining smartphone data for app usage prediction and recommendations: A survey. Pervasive Mob. Comput. 2017, 37, 1–22. [Google Scholar] [CrossRef]
  39. Wei, X.; Gurkok, C. ProfileDroid: Multi-layer Profiling of Android Applications Categories and Subject Descriptors. In Proceedings of the MobiCom’12 18th Annual International Conference on Mobile Computing and Networking, Istanbul, Turkey, 22–26 August 2012; pp. 137–148. [Google Scholar]
  40. Kumar, V.; Thakur, R.S. A Brief Investigation on Web Usage Mining Tools ( WUM ). Saudi J. Eng. Technol. 2017, 2, 1–11. [Google Scholar]
  41. Elzer, P.F. Der „Lebensdauerzyklus“ der Softwareentwicklung. In Management von Softwareprojekten; Vieweg + Teubner: Wiesbaden, Germany, 1994; pp. 33–82. [Google Scholar]
  42. Jansen, B.J. Search log analysis: What it is, what’s been done, how to do it. Libr. Inf. Sci. Res. 2006, 28, 407–432. [Google Scholar] [CrossRef]
  43. Zwijsen, S.A.; Niemeijer, A.R.; Hertogh, C.M.P.M. Ethics of using assistive technology in the care for community-dwelling elderly people: An overview of the literature. Aging Ment. Health 2011, 15, 419–427. [Google Scholar] [CrossRef]
  44. European Union Data Protection in the EU. Available online: (accessed on 23 March 2019).
  45. Felnhofer, A.; Kothgassner, O.D.; Hauk, N.; Kastenhofer, E.; Kryspin-Exner, I. Ethik-Checkliste; University of Vienna: Wien, Austria, 2013. [Google Scholar]
  46. Weber, K. MEESTAR: Ein Modell zur ethischen Evaluierung sozio-technischer Arrangements in der Pflege- und Gesundheitsversorgung. In Technisierung des Alltags—Beitrag für ein Gutes Leben; Weber, K., Frommeld, D., Manzeschke, A., Fangerau, H., Eds.; Franz Steiner: Stuttgart, Germany, 2015; pp. 247–262. ISBN 978-3-515-11004-4. [Google Scholar]
  47. Manzeschke, A.; Weber, K.; Rother, E.; Fangerau, H. Ethical Questions in the Area of Age Appropriate Assisting Systems; VDI/VDE: Munich, Germany, 2015; ISBN 9783897501690. [Google Scholar]
  48. ISO/IEC. ISO/IEC 9834-8:2005 Information Technology—Open Systems Interconnection—Procedures for the Operation of OSI Registration Authorities: Generation and Registration of Universally Unique Identifiers (UUIDs) and Their Use as ASN.1 Object Identifier Compon; ISO: Geneva, Switzerland, 2005; p. 24. [Google Scholar]
  49. Craddock, G. The AT continuum in education: Novice to power user. Disabil. Rehabil. Assist. Technol. 2006, 1, 17–27. [Google Scholar] [CrossRef]
  50. Castellano, G.; Fanelli, A.M.; Torsello, M.A. Log Data Preparation for Mining Web Usage Patterns. In Proceedings of the IADIS International Conference Applied Computing, Salamanca, Spain, 18–20 February 2007; pp. 371–378. [Google Scholar]
  51. Jain, M.; Kumar, M.; Aggarwal, N. Web Usage Mining: An Analysis. J. Emerg. Technol. Web Intell. 2013, 5, 240–246. [Google Scholar] [CrossRef]
  52. Mans, R.; Reijers, H.; Wismeijer, D.; Van Genuchten, M. A process-oriented methodology for evaluating the impact of IT: A proposal and an application in healthcare. Inf. Syst. 2013, 38, 1097–1115. [Google Scholar] [CrossRef] [Green Version]
  53. Garcia, C.; Dos, S.; Meincheim, A.; Faria Junior, E.R.; Dallagassa, M.R.; Sato, D.M.V.; Carvalho, D.R.; Santos, E.A.P.; Scalabrin, E.E. Process mining techniques and applications—A systematic mapping study. Expert Syst. Appl. 2019, 133, 260–295. [Google Scholar] [CrossRef]
  54. Adhikari, R.; Agrawal, R.K. An Introductory Study on Time Series Modeling and Forecasting; LAP Lambert Academic Publishing: Saarbrücken, Germany, 2013; ISBN 978-3-659-33508-2. [Google Scholar]
  55. Jenks, G.F. The Data Model Concept in Statistical Mapping. Int. Yearb. Cartogr. 1967, 7, 186–190. [Google Scholar]
  56. Komsky, S.H. A Profile of Users of Electronic Mail in a University. Manag. Commun. Q. 1991, 4, 310–340. [Google Scholar] [CrossRef]
  57. Egan, W. What Is a Good Retention Rate for Online Software. Available online: (accessed on 23 March 2019).
  58. Tong, H.L.; Laranjo, L. The use of social features in mobile health interventions to promote physical activity: A systematic review. NPJ Digit. Med. 2018, 1, 43. [Google Scholar] [CrossRef]
  59. Lawton, M.P.; Brody, E.M. Assessment of Older People: Self-Maintaining and Instrumental Activities of Daily Living. Gerontologist 1969, 9, 179–186. [Google Scholar] [CrossRef] [PubMed]
  60. Matomo Glossary of Analytics Terms—Web & Mobile Analytics. Available online: (accessed on 9 March 2019).
  61. Wang, R.; Blackburn, G.; Desai, M.; Phelan, D.; Gillinov, L.; Houghtaling, P.; Gillinov, M. Accuracy of Wrist-Worn Heart Rate Monitors. JAMA Cardiol. 2017, 2, 104–106. [Google Scholar] [CrossRef] [Green Version]
  62. World Health Organization. Global Recommendations on Physical Activity for Health; WHO Press: Geneva, Switzerland, 2010; ISBN 978-92-4-159-997-9. [Google Scholar]
  63. Pate, R.R. Physical Activity and Public Health. JAMA 1995, 273, 402–407. [Google Scholar] [CrossRef] [PubMed]
  64. Trukeschitz, B.; Blüher, M. Usability of “CARIMO” after Initial Training and Over Time. The Home Care Service Users’ Perspective in Austria and Italy; Vienna University of Economics and Business: Vienna, Austria, 2018. [Google Scholar]
  65. Perro, J. Mobile Apps: What’s a Good Retention Rate? Available online: (accessed on 11 March 2019).
  66. Garmendia, M.L.; Dangour, A.D.; Albala, C.; Eguiguren, P.; Allen, E.; Uauy, R. Adherence to a physical activity intervention among older adults in a post-transitional middle income country: A quantitative and qualitative analysis. J. Nutr. Health Aging 2013, 17, 466–471. [Google Scholar] [CrossRef] [Green Version]
  67. Macera, C.A.; Ham, S.A.; Yore, M.M.; Jones, D.A.; Ainsworth, B.E.; Kimsey, C.D.; Kohl, H.W. Prevalence of physical activity in the United States: Behavioral Risk Factor Surveillance System, 2001. Prev. Chronic Dis. 2005, 2, A17. [Google Scholar]
  68. Coroiu, A.D.C.A.; Coroiu, A. Interchangeability of Kinect and Orbbec Sensors for Gesture Recognition. In Proceedings of the 2018 IEEE 14th International Conference on Intelligent Computer Communication and Processing (ICCP), Cluj-Napoca, Romania, 6–8 September 2018; pp. 309–315. [Google Scholar]
Figure 1. CARIMO tablet (CARIMO home screen) and fitness bracelet. Source: Salzburg Research.
Figure 1. CARIMO tablet (CARIMO home screen) and fitness bracelet. Source: Salzburg Research.
Applsci 10 00038 g001
Figure 2. Usage data evaluation framework.
Figure 2. Usage data evaluation framework.
Applsci 10 00038 g002
Table 1. Sample description of the trial participants (test group).
Table 1. Sample description of the trial participants (test group).
Test Month12345678
Number of participants10497948989858585
Age group
 <60 years22222222
 60–69 years2928272727262626
 70–79 years5147454141393939
 >79 years2220201919181818
Dependency level
 Can do some (I) ADLs with help only (help needed)2825242323212121
 Can do some (I) ADLs with difficulty but manage on their own (difficulty)4945444040393939
 Can do all (I) ADLs without help (independent)2424232323232323
Notes: Source: SRFG (Salzburg Research), CiM usage data 2017/2018, and WU (Vienna University of Economics and Business), CiM effectiveness survey 2017/2018.
Table 2. Usage groups for body-related features.
Table 2. Usage groups for body-related features.
User GroupFitness Exercises
(Per Month)
(Per Month)
Body-Related Functions
(Per Month)
Frequent user8 times or more25 times or more
  • Fitness exercises = frequent AND activities = frequent
  • Fitness exercises = frequent AND activities = regular
  • fitness exercises = regular AND activities = frequent
Regular userBetween 4 and 7 timesBetween 9 and 24 times
  • All other possibilities
Infrequent userBetween 1 and 3 timesBetween 1 and 8 times
  • Fitness exercises = infrequent AND activities = infrequent
  • Fitness exercises = infrequent AND activities = non-user
  • Fitness exercises = non AND activities = infrequent
  • Fitness exercises = non AND activities = non-user
Table 3. Usage groups for mind-related features.
Table 3. Usage groups for mind-related features.
Usage GroupsNewspapers
(Per Month)
(Per Month)
(Per Month)
Internet & Games (I & G)
(Per Month)
Mind-Related Functions
(Newspapers, Internet & Games)
(Per Month)
Frequent user20 times or more16 times or more15 times or more
  • Games = frequent AND Internet = frequent
  • Games = frequent AND Internet = regular
  • Games = regular AND Internet = frequent
  • Newspapers = frequent AND I & G = frequent
  • Newspapers = frequent AND I & G = regular
  • Newspapers = regular AND I & G = frequent
Regular userBetween 10 and 19 timesBetween 7 and 15 timesBetween 8 and 14 times
  • All other possibilities
  • All other possibilities
Infrequent userBetween 1 and 9 timesBetween 1 and 6 timesBetween 1 and 7 times
  • Games = infrequent AND Internet = infrequent
  • Games = infrequent AND Internet = non-user
  • Games = non-user AND Internet = infrequent
  • Newspapers = infrequent AND I & G = infrequent
  • Newspapers = infrequent AND I & G = non
  • Newspapers = non-user AND I & G = infrequent
  • Games = non-user AND Internet = non-user
  • Newspapers = non-user AND I & G = non-user
Table 4. General use of CARIMO per test month.
Table 4. General use of CARIMO per test month.
Test Month12345678
Total users 10497948989858585
Users per country
Median usage of CARIMO (per user/month)
CARIMO app: median visits3236273029273329
Activity tacker; median days14181520171530
Median usage of CARIMO in Austria (per user/month)
CARIMO app median visits4445383839303335
Activity tracker median days17202028222016.50
Median usage of CARIMO in Italy (per user/month)
CARIMO app median visits1818182326321732
Activity tracker median days6105119300.5
Notes: Source: SRFG, CiM usage data 2017/2018.
Table 5. Usage groups and retention rates for body-related and mind-related features of the CARIMO app per test month.
Table 5. Usage groups and retention rates for body-related and mind-related features of the CARIMO app per test month.
Body-Related Features (Aggregation of ‘Fitness Exercises’ and ‘Activities’)CARIMO FeaturesMind-Realted Features (Aggregation of ‘Newspapers’ and ‘Internet and Games’)
12345678Test month12345678
44.2% (46)44.3% (43)45.7% (43)51.7% (46)41.6% (37)37.6% (32)40.0% (34)32.9% (28)Frequent users18.3% (19)16.5% (16)11.7% (11)18.0% (16)14.6% (13)16.5% (14)18.8% (16)15.3% (13)
22.1% (23)25.8% (25)24.5% (23)23.6% (21)24.7% (22)29.4% (25)30.6% (26)29.4% (25)Regular users36.5% (38)39.2% (38)40.4% (38)36.0% (32)40.4% (36)49.4% (42)44.7% (38)37.6% (32)
22.1% (23)8.2% (8)17.0% (16)10.1% (9)18.0% (16)15.3% (13)11.8% (10)10.6% (9)Infrequent users37.5% (39)32.0% (31)36.2% (34)32.6% (29)30.3% (27)24.7% (21)29.4% (25)29.4% (25)
11.5% (12)21.6% (21)12.8% (12)14.6% (13)15.7% (14)17.6% (15)17.6% (15)27.1% (23)Non-users7.7% (8)12.4% (12)11.7% (11)13.5% (12)14.6% (13)9.4% (8)7.1 (6)17.6% (15)
54.7% (35)55.0% (33)64.9% (37)65.5% (36)52.7% (29)50.0% (27)48.1% (26)40.7% (22)Frequent users25.0% (16)21.7% (13)12.3% (7)20.0% (11)14.5% (8)18.5% (10)16.7% (9)14.8% (8)
26.6% (17)28.3% (17)17.5% (10)20.0% (11)21.8% (12)25.9% (14)37.0% (20)33.3% (18)Regular users40.6% (26)45.0% (27)49.1% (28)36.4% (20)40.0% (22)48.1% (26)46.3% (25)37.0% (20)
14.1% (9)5.0% (3)7.0% (4)5.5% (3)14.5% (8)11.1% (6)5.6% (3)5.6% (3)Infrequent users28.1% (18)25.0% (15)26.3% (15)27.3% (15)25.5% (14)24.1% (13)31.5% (17)29.6% (16)
4.7% (3)11.7% (7)10.5% (6)9.1% (5)10.9% (6)13.0% (7)9.3% (5)20.4% (11)Non-users6.3% (4)8.3% (5)12.3% (7)16.4% (9)20.0% (11)9.3% (5)5.6% (3)18.5% (10)
27.5% (11)27.0% (10)16.2% (6)29.4% (10)23.5% (8)16.1% (5)25.8% (8)19.4% (6)Frequent users7.5% (3)8.1% (3)10.8% (4)14.7% (5)14.7% (5)12.9% (4)22.6% (7)16.1% (5)
15.0% (6)21.6% (8)35.1% (13)29.4% (10)29.4% (10)35.5% (11)19.4% (6)22.6% (7)Regular users30.0% (12)29.7% (11)27.0% (10)35.3% (12)41.2% (14)51.6% (16)41.9% (13)38.7% (12)
35.0% (14)13.5% (5)32.4% (12)17.6% (6)23.5% (8)22.6% (7)22.6% (7)19.4% (6)Infrequent users52.5% (21)43.2% (16)51.4% (19)41.2% (14)38.2% (13)25.8% (8)25.8% (8)29.0% (9)
22.5% (9)37.8% (14)16.2% (6)23.5% (8)23.5% (8)25.8% (8)32.3% (10)38.7% (12)Non-users10.0% (4)18.9% (7)10.8% (4)8.8% (3)5.9% (2)9.7% (3)9.7% (3)16.1% (5)
Notes: Source: SRFG, CiM usage data 2017/2018.

Share and Cite

MDPI and ACS Style

Schneider, C.; Trukeschitz, B.; Rieser, H. Measuring the Use of the Active and Assisted Living Prototype CARIMO for Home Care Service Users: Evaluation Framework and Results. Appl. Sci. 2020, 10, 38.

AMA Style

Schneider C, Trukeschitz B, Rieser H. Measuring the Use of the Active and Assisted Living Prototype CARIMO for Home Care Service Users: Evaluation Framework and Results. Applied Sciences. 2020; 10(1):38.

Chicago/Turabian Style

Schneider, Cornelia, Birgit Trukeschitz, and Harald Rieser. 2020. "Measuring the Use of the Active and Assisted Living Prototype CARIMO for Home Care Service Users: Evaluation Framework and Results" Applied Sciences 10, no. 1: 38.

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop