Intelligent Monitoring of Affective Factors Underlying Sport Performance by Means of Wearable and Mobile Technology ^†

Abstract

The fluctuation of affective states is a contributing factor to sport performance variability. The context surrounding athletes during their daily life and the evolution of their physiological variables beyond sport events are relevant factors, as they modulate the affective state of the subject over time. However, traditional procedures to assess the affective state are limited to self-reported questionnaires within controlled settings, thus removing the impact of the context. This work proposes a multimodal, context-aware platform that combines the data acquired through smartphones and wearable sensors to assess the affective state of the athlete. The platform is aimed at ubiquitously monitoring the fluctuations of affective states during longitudinal studies within naturalistic environments, overcoming the limitations of previous studies and allowing for the complete evaluation of the factors that could modulate the affective state. This system will also facilitate and expedite the analysis of the relationship between affective states and sport performance.

1. Introduction

Affective states are the existential background of our lives. They provide the affective context to what we experience and what we do [1]. Moreover, they play an active role in the determination of both the cognition content—how we think or remember and the cognitive processes—how we reason.

Within a sport context, affective states are inherent to practices and competitions, and they are widely recognized to be a contributing factor to sport performance variability [2,3]. Athletes commonly experience fluctuations in their performance which may not be due to physical causes, but to behavioral and emotional ones. Positive affects have typically been related to increased performance outcomes, while negative affects have been associated to malfunctioning and decreased performance [4]. For that reason, research into the link between emotions and sport performance is gaining momentum. A SportDiscus [5] search for the keyword ‘emotion’ in the title from 2010 to 2018 produced almost the same results (672) than the previous two decades combined (669 results from 1990 to 2009).

However, this research has been traditionally focused on analyzing affective states during the competitive events, without considering daily life factors that could modulate them. Non-competitive affects have the potential to impact on emotional well-being and therefore, sporting performance. That is why researchers are encouraged to perform longitudinal studies among professional athletes, monitoring the change of affective states and profiles over time, in order to enable early detection of decreased performance [6]. Furthermore, not only within-subject factors should be addressed when monitoring daily life affective states, but also the context surrounding the athlete. Identifying out-of-sport events that could lead to a change in the individual’s affective profile is also a fundamental task [7]. Almost any study aimed to explore the relationship between affective states and sport performance is based on experiments carried out within controlled environments, dependent on subjective self-reports of the athletes through questionnaires [6,7,8].

In the light of the present challenges and limitations, this work, as a part of an ongoing project, describes the factors leading to the variability of affective states, and presents a proposal for a methodology to explore the relationship between affective states variability and sport performance level within naturalistic environments. In order to model the fluctuations of affective states in athletes, we propose a multimodal, context-aware platform which combines the data of several sensors, measured continuously in an ubiquitous, non-invasive way, during both sport events and daily life. This paper is structured as follows. Section 2 provides an overall picture of the factors that modulate affective states, and how they can be measured using mobile and wearable technologies. Section 3 presents the architecture and actual implementation of the monitoring platform. The improvements introduced by this methodology within a sport context will be discussed in Section 4. Finally, Section 5 summarizes final conclusions and highlights future work and prospects of the system.

2. Smart Monitoring of Factors Modulating Affective States

The affective profile of an individual does not remain constant, but fluctuates and changes over time. These variations may be caused by several factors, which can be divided into two groups: exogenous or external, and endogenous or internal. External factors are the short-term situational or environmental variables which trigger a significant transient fluctuation on the affective state. Environmental aspects as temperature or humidity, the intake of certain substances and food, or daily events are commonly considered examples [9]. Research has also shown the relationship between out-of-sport physical exercise and affective states [10]. Internal factors, by contrast, refer to biological variables. There is evidence supporting that hormone levels, body temperature, sleep time or circadian rhythms induce an important part of the variation of affective states [11]. The bidirectional communication between brain and heart, as a representation of the link between central and peripheral nervous systems, has also raised as a key factor in affective processing [12]. This constant communication, basis of the interoception, has traditionally been measured in an indirect fashion through the heart-rate variability (HRV). However, as a direct measurement, the heartbeat-evoked potentials (HEP) have taken a lead role during last years [13]. The HEP are electroencephalographic (EEG) responses synchronized with heart beats. Their amplitude provide an objective measurement of the conscious perception of heart beats, which has proved to be a relevant indicator of affective processing [14].

The objective monitoring of those factors which influence each athlete offer a huge challenge, almost unreachable thus far. With the recent emergence of mobile and wearable technologies, a new breed of intelligent mechanisms can be raised to ubiquitously monitor affective and emotional processes [15]. Among the various available systems, noteworthy is the use of smartphones. They are presumably the richest devices in terms of sensors and computing resources, within our reach. They are part of our daily life and their use is widespread among every segment of population. Within the emotional field, smartphones have been usually employed to record affective data through digital questionnaires [16]. However, their applications have also been explored in a more sophisticated manner to estimate the affective state based on the measurement of exogenous factors. For example, speech recognition techniques are used to determine the voice tone using the microphone, and classification techniques are also used to interpret the content of text messages, in order to infer the individual affective state [17]. Accelerometers and GPS sensors of smartphones are also used to measure the movement and activity patterns of the individual [16].

Likewise, wearable devices are presented as a low-cost, ubiquitous technology capable of acquiring endogenous data that complement those obtained through smartphones. They are able to record a wide spectrum of physiological variables to estimate the affective state in a non-invasive and continuous way [18]. Multiple wearables can also be joined in a Body Area Network (BAN) to acquire different body signals simultaneously. Within the available devices, photoplethysmography (PPG) sensors are used to estimate the previously mentioned HRV, and electromyography (EMG) sensors are also used to measure the electrical activity of skeletal muscles. In [19], various emotions are estimated based on the activation of certain facial muscles. The electrodermal activity (EDA) and the temperature (ST) of the skin are also commonly used internal factors, in that affective states with a high level of activation cause abrupt fluctuations in skin temperature and conductance. In [20], these two sensors are integrated in a flexible patch used to monitor the stress level.

Brain-heart communication has also been measured using wearables. Traditional EEG/ECG signal acquisition devices restrict their use to laboratory-based settings, so wearables become the appropriate solution to extend the monitoring of HEP to naturalistic studies. A recently developed example is the multimodal database DREAMER [21]. It includes a range of EEG and ECG signals acquired through low-cost wearable devices, which are aimed at being used for emotion recognition.

3. Proposed Monitoring Platform

In view of the clear importance of affective states in sport performance, the lack of longitudinal studies of their dynamic changes, and the limited range of modulating factors analyzed, this work presents a first approach towards a methodology for exploring the variability of affective states within the sport performance context, using the advantages of wearable and mobile technologies. In particular, we propose the architecture of an intelligent platform for monitoring and modeling the evolution of internal and external variables and their interrelationship, in a holistic and continuous way. In order to obtain reliable and objective results, we have followed the key methodological prerequisites for within-subject associations between physical activity and momentary affective states proposed in [22]: measurement of affective states should be instantaneous rather than based on a retrospective recall at the end of the day or week; and questionnaire-based methods need to objectively record the time of the response, to avoid “back-filling” responses just before the end of the study, which is easily achieved using mobile technologies.

3.1. Platform Architecture

Figure 1 shows the high-level architecture of the proposed platform. The core entity of the system is the smartphone of the athlete. It serves as a coordinator of both internal and external sensors, setting up their operation and storing the data gathered. Internal smartphone sensors are primarily used to measure exogenous variables. They capture the context information, which is directly stored in the smartphone. Wearable sensors are connected to the smartphone and send the acquired data periodically. They are intended to measure endogenous variables during the daily life of the athlete, and to assess the level of performance during training sessions and events.

The data of both internal and external sensors is collected by the smartphone and sent to a server. There, it is unified in a dataset after a preprocessing stage, along with the affective state and sport performance data gathered. This dataset is aimed to be the starting point of several researches. First of all, it will allow us to model the variability of affective states in terms of the previously measured variables. Then, the link between the fluctuation of affective states and the measured sport performance could be analyzed.

3.2. Platform Implementation

Up until now, the lower levels of the architecture have been implemented. This section describes the sensors deployed and its operation within the smartphone framework. In order to develop the aforementioned functionalities, the platform builds on AWARE, an Android-based open-source context instrumentation framework [23]. It provides a client-server mobile framework that supports the collection of unobtrusive passive sensor data and is licensed under the Apache Software License 2.0, so it allows for performing changes and adding extensions to the main code. It follows a modular approach: the AWARE client app abstracts the communication with the sensors and the acquisition of data; then, data is used to generate context through customizable code extensions named plugins. The proposed platform uses and extends this approach. The specialist sets on the server which sensors and plugins are going to be used, and once the athlete has joined the study—which is as simple as scanning a QR code through the AWARE client app–, plugins are automatically installed, and data acquisition begins. Then, installed plugins manage each sensor’s acquisition process and convert data into interpretable context information in real-time, as each sensor has its own sampling rate, which can be continuous—e.g., accelerometer, screen usage, temperature, etc.—or intermittent—EEG, ECG, etc.

3.2.1. Affective State

To measure the evolution of the affective state of the athlete, the system also takes advantage of user-provided context using the Experience Sampling Method (ESM) [24] through the smartphone. It allows for the collection of real-time subjective data about affective states in the natural environment repeatedly over time. The use of ESM beyond traditional text-based questionnaires has been successfully explored in recent works [25]. In order to take advantage of ESM, we have developed an AWARE plugin which triggers a short questionnaire that evaluates the user’s affective state at this precise instant. The ESM is triggered several times per day during the course of the study, since we need to record the evolution of the affective state as accurate as possible. It can be scheduled either at hours or at random hours, setting an interval during which the questionnaire is randomly triggered. The ESM is thrown as a push notification in the smartphone, which the athlete must click to start the questionnaire. Questions show up sequentially, and the ESM screen disappears once every question is answered, without leaving any background process. Both notification timeout and ESM screen expiration threshold can be previously configured.

The affective state is measured using Russell’s circumplex model [26]. This dimensional model crosses two dimensions: valence and arousal, and affective states are constructed as combinations of varying degrees of these two independent dimensions. Valence is understood as the self-perceived level of pleasure of the actual affective state, from highly unpleasant to highly pleasant. Arousal, for its part, is the self-perceived level of activation, ranging from low—calmness—to high. Figure 2 shows some of the affective states which can be inferred using those dimensions. Although more dimensional models of emotion can be found in the literature—as the vector model [27] or the PANA model [28], the circumplex model’s easiness of use and reduced set of dimensions make it more suitable for a first approach of the affective state sampling.

Currently, the questionnaire designed includes two questions, measuring valence and arousal, respectively. Traditionally, these dimensions have been measured using a Likert scale [8] to select the level of the dimension. However, some studies make use of adapted response scales to allow for a greater variability of possible scores and improve usability [6]. Our approach is the use of a 100-point bipolar scale which is selected using a scroll bar. However, instead of showing the numeric value of the scale, we show simple face icons whose expressions represent the extreme values of the scale (Figure 3). For example, when measuring valence, the scale is represented with a sad face at the beggining and a smiling face at the end. In the case of arousal, a calm face represents the lower level of arousal, and an excited face, the higher one. Through this method, the anchoring problems of numeric scales are avoided. Emoticon-based ESMs have been proved to be more intuitive and user-friendly than numeric-based ones [29]. Since the ESM is triggered several times per day, the scale cursor is configured to start at a random point fo the scale rather than fixed point like the middle or the beggining, thereby avoiding fixed answers.

Although the AWARE client app supports basic ESM, the developed plugin extends its functionalities using a client-server approach. Instead of setting the ESM questions and schedules within the plugin’s code, they are constructed remotely through an XML form. The ESM question queue, its features and schedules are defined in an XML file that follows a particular XML schema known by the plugin, and the file is stored in a secure server. The plugin periodically retrieves, reads and deserializes the file, so that the ESM questionnaire can be constructed. This approach allows us to make real-time changes to the questions and schedules without recompiling the plugin.

When each question is answered or canceled, an intent with the hidden numeric value of the scale is sent by the plugin, and captured by the main app. The ESM status, device id, user answer, starting point of the scroll bar, time instance of the response and waiting time of the notification are stored in a central SQL database, which is periodically synchronized with the server.

3.2.2. Modulating Factors

To measure the modulating factors, both smartphone and wearable sensors are used. Within the smartphone, accelerometer, GPS and screen usage sensors are enabled. Accelerometer and GPS data are used to estimate the activity performed by the athlete outside training hours [30], by means of the AWARE Activity Recognition plugin. This plugin provides a human-readable activity name—seated, walking, running, etc.—and the confidence level of the estimation, with a previously configured sampling rate. These data is also used along with the detected screen usage to estimate the sleeping hours, as well as long periods of inactivity [31]. In terms of wearable sensors, up to now we have focused on the acquisition of EEG and ECG signals, in order to compute the value of the HEP. To that end, RABio w8 [32], an 8-channel wireless biosignal acquisition system has been used along with an electrode cap for EEG. Simultaneous EEG and ECG acquisition is allowed using different channels, and the synchronization is inherent to this approach, as both signals are measured by the same device. Both gain and sampling rate can be configured individually for each channel. Using the same procedure as the ESMs, a push notification is sent to the athlete’s smartphone when the HEP has to be measured. Then, a diagram shows up explaining how to put on the ECG electrodes and the EEG cap. Once the device is connected, it start sending the acquired data, sample by sample, to the smartphone via Bluetooth Low Energy. BLE data frames are gathered and stored by means of the AWARE Bluetooth plugin. This measure is aimed to be performed several times a day during resting hours. However, the measure can be extended to training sessions applying suitable techniques for movement artifact removal.

4. Discussion

This work presents a proposal for a multimodal, context-aware platform which integrates the data acquisition of numerous sensors within naturalistic environments. The system is aimed at monitoring the evolution of affective states of athletes during longitudinal studies using ubiquitous and non-invasive technologies. The core of the system is a smartphone, whose sensors and functionalities allow us to continuously model the context surrounding the athlete, thus identifying out-of-sport internal and external factors that could modulate the affective state. By doing so, this work overcomes the limitations of previous studies, where the affective state was modeled using only within-subject measurements at isolated moments. This approach also allows for conducting the study during the daily life of the athlete, rather than measuring the affective state within controlled environments. At this stage, we have developed a prototype of the monitoring system. Among the functionalities implemented within the smartphone, we make use of ESMs to collect the values of valence and arousal of the user at certain instants, thus estimating the affective state. The developed plugin schedules a short questionnaire which is thrown as a push notification at the smartphone at the requested time, thereby avoiding “back filling” issues of paper-based questionnaires. Moreover, the questions of the ESM have been modified to display a visual scale with a changing emoticon instead of a numeric value. This provides a more intuitive and user-friendly interface, which helps athletes to identify their affective state more precisely than traditional numerical scales. Wearable sensors can also be connected to the platform for measuring biosignals and internal variables relative to affective states. In this primary design, a wearable acquisition system is used to measure EEG and ECG simultaneously, for a further computing of the heartbeat-evoked potentials. Its small size and low power consumption allows for the measurement of HEP out of laboratory-based settings, and explore its effect over the fluctuations of the affective state. The use of wearable technologies is also in line with the requirements of assessing performance during sport sessions, where the freedom of movement is mandatory. Further studies are needed to validate the developed platforms, which will be the next step of our research.

5. Conclusions

The affective state is a contributing factor to sport performance variability. Traditional research in affective states within a sport context has focused on cross-sectional, laboratory-based analyses during the physical activity. Furthermore, the assessment of affective states have been almost exclusively based on athletes’ self-reports through questionnaires. This work presents a monitoring platform to perform objective analyses of the affective state during longitudinal studies within naturalistic environments. The platform uses smartphones and wearable sensors to measure internal and external factors that modulate the affective state, overcoming the limitations of previous studies. This work is now being extended to include a higher range of sensors. A case study will also be performed to validate the system against the gold-standards. The final idea behind this work is to apply this platform to obtain a dataset of affect-related measurements out of sport-related events, in order to extract knowledge about the relevance of each factor within the fluctuations of the affective state during daily life. Once the variability of affective states is modeled, this data will also be related to sport performance fluctuations, in order to develop a prediction model of performance in terms of affective states.