Analysis of Cognitive Load Using EEG when Interacting with Mobile Devices ^†

Abstract

The study of cognitive responses and processes while using applications is a critical field in human–computer interaction. This paper aims to determine the mental effort required for different typical tasks with smartphones. Mental effort is typically associated with the concept of cognitive load, and has been studied by analyzing electroencephalography (EEG) signals. Thus, this paper shows the results of analyzing the cognitive load of a set of characteristic tasks on smartphones. To determine the set of tasks to analyze, this paper proposes a taxonomy of smartphone-based actions defined after considering the related proposals in the literature and identifying the significant characteristics of the tasks to classify them. The EEG data was obtained through an experiment with real users doing tasks from the aforementioned taxonomy. The results show significant differences in the cognitive load of each task category and identify those tasks that involve a higher degree of mental effort. The results will be the starting point of the M⁴S project that aims to contribute to the early diagnosis of mild cognitive impairment through monitoring everyday dual-tasking in terms of interaction with smartphones.

1. Introduction

The widespread use of smartphones and mobile devices today allows us to provide applications and services to address many issues, such as those in healthcare. The smartphone is a great source of information; its sensors provide us with data about location, movement, voice, battery, and application use, among others; this is being used to assess the behavioral aspects of users in their daily life [1]. Concerning health, for example, the analysis of smartphone use allows us to track GPS locations and trajectories followed by users and can be used, for example, to measure anxiety levels as a prelude to mental health problems [2]. Additionally, data coming from the interaction between users and their own mobile devices provide us with valuable information related to human behavior. In fact, the smartphone can be an ally for diagnostic purposes [3], although it should serve a complementary role in the doctor–patient relationship. Particularly, in dementia, Blanka Klimova [4] evidenced the potential of mobile apps for diagnosis support, minimizing examiner bias, increasing patient independence, cutting hospitalization costs, and improving the overall quality of life of elders. As we can see, analyses of interactions (explicit or implicit) with smartphones can be invaluable in the field of human–computer interactions, as well as in the field of health, with diagnosis and treatment purposes.

This paper is part of the project “Mobile computing-based Multitasking for Mild cognitive impairment Monitoring and early Screening (M⁴S)” that aims to contribute to the early diagnosis of mild cognitive impairment through monitoring everyday dual-tasking in terms of interactions with smartphones on the move. An initial stage of this project is to determine the cognitive load required for various, typical tasks performed on a mobile device, i.e., the main objective of this paper. The findings of this paper not only contribute to the aforementioned project, but also to the research community in the field, giving rise to a better understanding of cognitive processes which are associated with the use of mobile devices.

In order to analyze the cognitive load (i.e., the amount of working memory resources or “mental effort” associated with a specific task, concepts explored in depth in Section 2.1), we analyzed the EEG activity of users performing a set of typical tasks with a smartphone. The fundamentals of the EEG-based cognitive load analysis are also described in Section 2.1. To determine the set of tasks to analyze, this paper proposes a taxonomy of smartphone-based actions. We considered the related proposals in the literature (Section 2.2), and identified the significant characteristics of the tasks to classify them, in order to propose the HuSBIT-10 taxonomy: Human-Smartphone Basic Interactions Taxonomy for 10-s tasks (Section 3). An experiment with real users was conducted with the dual objective of (i) studying the cognitive load of different typical tasks with the smartphone, and (ii) validating the classification made in the taxonomy in terms of the mental effort associated with the identified task categories. The protocol, material, and methods of the experiment are explained in Section 4, and the obtained results are presented in Section 5. Finally, Section 6 concludes the paper.

2. Fundamentals and Background

2.1. Cognitive Load

2.1.1. Cognitive Load Fundamentals

Each person has a specific way of responding to external stimulus, but the brain processes this information following a common pattern for most people. One similar point is usually the mental effort and cognitive load, which are closely-related but different concepts; the first occurs in interactions between the characteristics of a task and the characteristics of the subject. Each task has a different load, i.e., it can be more complex or simple (depending on the steps or the level of precision required in order to perform the action), and each subject processes it differently, according to their skills and aptitudes [5]. On the other hand, mental effort is related to the cognitive resources we use to undertake a specific task.

With these basic concepts in mind, we can mention the Sweller’s [6] theory of cognitive load, which focuses on working memory, and specifically, on Mayer’s [7] theory of multimedia learning. These theories are part of the cognitive sciences that seek to improve multimedia environments [8] within the information processing paradigm, taking it as a “natural information processing system” [9].

In our work, tasks with smartphones have different mental burdens and are related to some external stimulus, therefore the tasks are translated into cognitive information. For this, we use several sensitive channels. In our case, the auditory and visual channels trigger working memory. The processing of information in working memory is related to the activity we are consciously carrying out [10]. In addition, recent research has shown that working memory is divided into three processors or channels [8]. The information processed in this memory is distributed between two partially independent processors, i.e., the auditory and the visual, which manipulate verbal and pictorial information, respectively. Additionally, there is a third processor known as the central-executive, which is responsible for coordinating the processing of information that enters and leaves the working memory.

For this reason, we must consider the presentation of information to avoid overloading these channels. In addition, it is critical to consider whether the information is new, so that it can be acquired only if the subject’s mental activity can relate it to mental schemes previously stored in long-term memory [7,11]. The difference here is that if a person has done a task repeatedly, their processing is different because they have response patterns associated with that task and the execution is faster or easier to do. This is achieved with practice time and depends on the intuitiveness of the tasks, which has been considered in the development of our work.

All the previous fundamentals are considered in this paper to define the taxonomy tasks with smartphone and to define the experiment protocol, as we describe in next sections.

2.1.2. EEG-Based Cognitive Load Analysis

Attending to the EEG activity, four main areas of the brain have been discussed in the literature to study neurological activity: parietal, occipital, temporal and frontal [4,12]. This neurological activity has been observed to produce a range of electrical waves per second at different frequencies with greater or lesser level of coverage that depends on the task being performed.

A clear example of the differences that occur in the electrical response of the brain associated with neurological activity can be seen in the electrical oscillations emitted during sleep compared to those made when awake. The brain produces very low-frequency electrical waves (<1 Hz) in the electroencephalogram (EEG) of sleep stages, between the 0.55–0.95 Hz range and with peaks at 0.7–0.8 Hz in the frequency band known as delta [13]. On the other hand, higher frequencies and faster waves predominate in waking conditions, where bands oscillate between 0.5–40 Hz. The intervals that correspond to each band are as follows: 0.5–4 Hz (delta band), 4–8 Hz (theta band), 8–13 Hz (alpha band), 13–30 (beta band), and finally 30–40 Hz (gamma band). As stated before, the composition of the electrical response strongly depends on the cognitive task.

EEG techniques can capture the electrical response of the brain by means of electrodes placed on the scalp. These electrical signals are generated by ionic movements in and around neurons during the activation and deactivation of neurons involved in a cognitive task. EEG measures the fluctuating voltages in these electrical signals. There is not a straightforward way to estimate cognitive load from EEG electrical signals, however, some approaches can be found in the literature. The three most commonly used analysis techniques are: (i) event-related desynchronization (ERD), (ii) theta-alpha ratio (TAR) and (iii) those techniques based on machine learning. In relation to detecting changes in cognitive load using the ERD technique, Klismech found that the spectral power in the theta band increases while the spectral power in the alpha band decreases [14]. Further relevant contributions have studied the use of ERD from alpha and theta bands to measure cognitive load. For example, Antonenko et al. have applied ERD technique in two different case studies related to the learning context [15].

On the other hand, some recent studies have explored the use of TAR technique as a measure of cognitive load [16,17,18]. In particular, Trammell at el. have found associations between age and estimated cognitive load by using this technique. TAR is obtained by dividing the spectral power of theta band in the middle frontal area (Fz) by the spectral power of alpha band in central parietal area (Pz).

Other novel and powerful approaches to estimate cognitive load are those based on machine learning. There are many research works using these techniques for this particular purpose [19] which uses Naïve-Bayes, and [20], which uses deep convolutional neural networks. Through machine learning models, robust and useful metrics can be extracted from EEG signals, although they have some problems related to the sample size and data gathered from the acquisition trials. Specifically, (i) it takes a large number of participants to adequately train a classifier or fit a regression model that can be able to work properly on EEG data from anyone; (ii) the studies found are mostly based on supervised learning, therefore, a big labeled dataset is required to train the model. Such considerations have made us discard machine learning techniques as a method for this work.

2.2. Mobile-Device Interaction Background

Interaction with mobile devices has been widely studied in the literature. Today, many projects and research apply a user-centered design and development, highlighting the role of usability and user experience in terms of mobile device interactions. According to Hoober [21], users interact with their mobile devices in three different ways: (1) using only one hand, (2) using both hands, and (3) in a passive manner. Likewise, this study also indicates three types of human-device interactions: active use passive use, and talking, as well as the body posture that the users have when they interact with their mobile phones: walking, standing and sitting.

In 2005, Karam and Shraefel carried out a broad study leading to the creation of a general taxonomy of gestures in human computer interactions [22]. In this work, authors also presented a review of possible interactions with any device, not just mobile. Focusing on mobile phones, the most common inputs were: camera, touch surface and sensors-on-body (e.g., accelerometer, GPS). The last input is considered as a pervasive or implicit way of interaction with the mobile device. In case of interaction with touch screens, Wroblewski [23] proposed a reference guide very popular as standardized guide about gestures in these kind of displays.

Furthermore, there is a vast amount of works which are focused on analyzing user-Smartphone interactions in different domains, using a variety of measurement mechanisms, and facing multiple purposes. Today, works such as the one presented by Hinckley et al. [3] show new ways to detect interaction with smartphone screens before it happens. which is referred to as “pre-touch sensing”. Cameras and vision-based systems are also useful to analyze interactions with mobile applications. Authors such as Souza [4] and Chang [5] highlight the importance of eye-tracking data for usability studies, comparing them with traditional techniques.

The analysis of population behaviors can also be studied by observing interactions between users and their mobile phones. In this regard, new usage and behavioral patterns can be found [6], as well as different types of smartphone users [7]. In the field of psychology, Harari et al. [8] determine that the use of smartphones is an important observation tool in psychological science, considering all data provided by these mobile devices. Measuring and analyzing patterns for smartphone addiction is also possible by interacting with the mobile phone [9,10].

All interactions with mobile phones occur at different levels. We can study these interactions with the operating system, built-in sensors, and physical device buttons, as well as with the installed applications. Given our objectives, and considering the related work, in which we did not find research regarding the classification of smartphone tasts on a cognitive level, we propose a specific taxonomy of basic tasks related to the most common types of interaction with mobile phones in the following section.

3. Proposed Taxonomy: HuSBIT-10

According to the objectives of our study, we need to define a set of usual tasks focused on the user-smartphone interaction. Thesetasks are quick and simple tasks that require less than 10 s. The name of the taxonomy is HuSBIT-10: Human-Smartphone Basic Interactions Taxonomy for 10-s tasks.

First, we have identified four types of interactions that a user could carry out with his mobile phone: (τ) touch, (ι) look, (ς) speak, and (η) hear. All of them are closely related to human senses, critical to analyze the cognitive load and information processing [8]. In addition, considering some approaches from the literature, the interaction types can be classified in two categories: (α) active and (ρ) passive, depending on whether the user explicitly interacts with the device or not. With this, we can determine if a specific interaction type from the first four types previously mentioned is active or passive.

Moreover, any interaction task with the mobile phone could employ one or several types of interaction from the above. Hence, we have defined AMPEC-10 as a term to group the five types of tasks that a user can carry out with the smartphone in a maximum time of 10 s (limit obtained experimentally), making use of the four interaction types. According to the acronym AMPEC-10, the tasks have been grouped into the following types:

• (A) Automated. This represents the tasks with or without a minimal cognitive effort that we typically perform automatically or unconsciously.
• (M) psychoMotor. This kind of task requires a quick or direct interaction with the smartphone, where the main difficulty is to perform a touching interaction carefully or with proper accuracy.
• (P) Production. It includes tasks which require basic content creation, requiring creative skills to produce new content.
• (E) Exploration. This kind of tasks requires the analysis of a set of data to obtain specific information.
• (C) Consumption. It defines the tasks that require content consumption.

The prevalence of interaction types (touch, look, speak, and hear) on these task types is another factor to consider. A first approach reveals that touch and look interaction types are the most common interactions between user and smartphone. Likewise, as we saw in Section 2.2, speak and hear occur less often. This fact confirms what other studies in the literature, as studied in [24,25].

With these assumptions, we have modeled a new taxonomy approach from scratch, called HuSBIT-10 to classify any task with a duration under 10 s, that users perform with their smartphones. In

Table 1. AMPEC-10 tasks classification according to HuSBIT-10 approach.

Task Category	Id	Task Type	Characteristics	Examples
Automated	A1	Query an item	(α) (τ, ι)	Check time/Check if there are notifications/Check if I have WiFi
	A2	Action on any physical button	(α) (τ)	Turn on-off device/Turn up-down Volume
psychoMotor	M1	Pattern	(α) (τ, ι)	Device unlock (with unlock pattern)
	M2	Move	(α) (τ, ι)	Add and move a shortcut
	M3	Dismiss	(α) (τ, ι)	Close opened apps, Close notification preview
	M4	Copy & Paste	(α) (τ, ι)	Share information among applications
	M5	Select	(α) (τ, ι)	Select a part of a text
Production	P1	Text Production	(α) (τ, ι)	Add a new contact/Set an alarm/Write a message/Reminder
	P2	Voice Production	(α) (τ, ς)	Make a call/Make a voice command/Create voice message
	P3	Visual Production	(α) (τ, ι)	Take a photo
Exploration	E1	Search on a textual set	(α) (τ, ι)	Search for a contact/Search for a song/Search for date in the calendar/Last call made to someone
	E2	Search on a visual set	(α) (τ, ι)	Search for a specific application/Browse images/Change direct-access settings (e.g., airplane mode)
	E3	Analysis of textual contents	(α) (τ, ι)	Change settings details (e.g., data roaming)/Do a search in an Internet Browser
	E4	Analysis of visual contents	(α) (τ, ι)	Search for a route/site on a map
Consumption	C1	Text Consumption	(ρ) (ι)	View/Read notifications, Read a text message
	C2	Audio Consumption	(ρ) (η)	Listen to an audio message/Listen to a podcast
	C3	Media Consumption	(ρ) (ι, η)	Watch a video

, an overview of identified tasks (classified by task type) and some examples are shown.

The aim of HusBIT-10 approach is to provide support to classify the AMPEC-10 tasks in terms of planning and cognitive load from a bidimensional perspective, as well as promoting replicability in other trials and experiments.

4. Experiment: Cognitive Load in Smartphone Interactions

4.1. Experiment Protocol and Method

The study was conducted in the MAmI research lab at the University of Castilla–La Mancha, a group focused on health informatics and human-computer interaction (http://mami.uclm.es). The participants were informed about the scope and goals of this research and about the collected data. The work was conducted with six participants, from 22 to 31 years old who received and signed the information sheet and consent form, which provided detailed information about the study’s objective, procedures, and types of data to be collected. All participants had the opportunity to consider their participation before making a final decision. Thereby, the preservation of the dignity and autonomy of the end-users was ensured by their voluntary participation and the fact that they could leave the study at any time without any consequences.

This study followed the empirical method for gathering evidence regarding EEG data while user interact with a smartphone. The followed protocol can be summarized as follows: (1) all participants received an instruction sheet with the actions to do with the smartphone and ensuring they fully understood it (Figure 1d). The actions were randomly sorted for each participant; (2) all participants were wearing the EEG headset (Figure 1a) and sat at a desk with the smartphone (Figure 1c); (3) participants were required to perform the EEG calibration with the Xavier TechBench Software^TM (www.emotiv.com/product/emotivpro/) (Figure 1b); (4) the participants, without receiving any additional instruction, performed all the tasks of the sheet. This entire process for each participant took approximately 25 min.

4.2. Material

The cognitive load is studied using a device for capturing EEG signals for scientific purposes, model Emotiv EPOC+. This device has 14 EEG channels and two references for positioning and accurate spatial resolution. These channels have eight frontal electrodes (AF3, F7, F8 and FC5 on the left hemisphere, and FC6, F4, F8 and AF4 on the right hemisphere), two temporal electrodes (T7 and T8), two occipital electrodes (O1 and O2) and two parietal electrodes (P7 and P8). The headset uses a sequential sampling method at a rate of 128 samples per second. To collect the EEG raw data, we used the Xavier TechBench Software^TM. For the EEG data processing, there is a specialized software developed to process EEG data, called eeglib, which can also applied to other kind of data sources [26,27]. eeglib is actually a Python-based library for EEG processing that provides some data structures to help for that purpose. This library can load CSV and EDF files that are typical formats in which EEG is stored, and allows the user to import the data from Python and NumPy data structures. It can apply three different pre-processing techniques to the signals: bandpass filtering, z-scores normalization and Independent Component Analysis. It also includes a set of processing techniques to extract features from data: FFT, Higuchi Fractal Dimension, Petrosian Fractal Dimension, Hjorth parameters, Detrended Fluctuation Analysis, Lempel-Ziv Complexity, Multiscale Sample Entropy, Synchronization Likelihood, and Pearson Cross Correlation Coefficient. The library includes a tool to generate datasets (in pandas DataFrame format) that can be used easily to apply machine learning techniques or to perform statistical analysis.

The smartphone used in the experiment was a Samsung J6 with the operating system Android 9.0 Pie. The list of specific tasks to perform is shown in

Table 2. List of tasks performed in the experiment according to HuSBIT-10 taxonomy.

Task Category	Task Type	Specific Task in the Experiment
Consumption	C1	Read a message that contains a poem by Espronceda
	C2	Listen to a podcast from the daily news
	C3	Watch a video
Exploration	E1	Search for a given date in the calendar
	E3	Switch off the data roaming in the device settings
	E4	Search how to reach a given place (about 500 m away) in the map from the current location
psychoMotor	M2	Add and move an app shortcut (2 times)
	M4	Copy a message into the browser search box (Google widget)
	M5	Select one word, then two and, finally, two and a half words in a Wikipedia article
Production	P1	Write down the places where you would go in a zombie apocalypse
	P2	Create a voice message with the list of objects you would collect in a zombie apocalypse
	P3	Take an artistic photo of one object around you

. There are three tasks per category, omitting the category Automated due to the very low cognitive load associated to unconscious or mechanic tasks. Thus, the total number of evaluated tasks were 12. The design of the list of tasks follows the considerations and fundamentals about cognitive load in Section 2.1.

4.3. EEG Data Processing

Participants performed all the tasks described in Table 2, specifically, three defined tasks per category considered in the HuSBIT-10 proposed taxonomy. EEG activity was recorded during each task for a 10-s interval (EEG segment). The recorded EEG data can be found in the link in Section Supplementary Material.

The EEG segments, denoted as <Participant_Id, Task Type>, were then analyzed to estimate cognitive load in accordance with the following procedure: first (i), each EEG segment was preprocessed applying a 2–15 Hz bandpass filter to remove frequencies that were not under analysis (neither alpha nor theta band were removed); then (ii), zScore normalization was performed on the filtered EEG signals to made possible relative comparisons across EEG segments for all the participants in the next steps. After pre-processing (iii), each normalized EEG segment was split into 1-s windows with 50% overlapping; later (iv), a variation of the TAR index was computed for each analysis window within the EEG segment, as indicated below:

TAR = (theta_F3 + theta_F4)/(alpha_P7 + alpha_P8)

(1)

where theta_F3 and theta_F4 were the spectral power of the theta band in the F3 and F4 electrodes (frontal area in both brain hemispheres); and alpha_P7 and alpha_P8 the spectral power of the alpha band in the P7 and P8 electrodes (parietal area in both brain hemispheres). Once theta-alpha ratio values were obtained for each analysis window (v), the TAR index was averaged for the entire EEG segment that enclosed them all. This average TAR index (for the current EEG segment) was considered as a cognitive load estimation for a particular task performed by one participant (<Participant_Id, Task Type>). If the average TAR indices across all participants are grouped by task and again averaged, cognitive load estimations for each particular task will be obtained.

4.4. Result

The experiment results show differences among the cognitive load associated with each of the categories of smartphone tasks. These finding should be considered preliminary due to the reduced number of subjects in the experiment, the slight differences, and the wide standard deviation of the data, as shown in

Table 3. Results of cognitive load in each task performed in the experiment according to HuSBIT-10 taxonomy.

Task Category	Averaged Cog. Load	Standard Deviation	Task Types	Averaged Cog. Load
Consumption			C1	1.206
	1.253	0.251	C2	1.273
			C3	1.280
Exploration			E1	1.097
	1.394	0.247	E3	1.597
			E4	1.488
psychoMotor			M2	1.224
	1.247	0.297	M4	1.241
			M5	1.277
Production			P1	1.287
	1.284	0.255	P2	1.240
			P3	1.324

. This table summarizes the average cognitive load data for each category, considering all the users. It can be observed that the tasks with the highest cognitive load are those of the Exploration category with a significant difference, followed by Production. The ones that show the least cognitive load are the psychoMotor and Consumption tasks, with very similar cognitive load measures.

Observing the average value obtained in each of the 12 evaluated tasks, we see two cases of tasks that have a significantly higher cognitive load than the rest of tasks: analysis of textual contents (E3) and analysis of visual content (E4). The task regarding analysis of textual contents has been carried out by means of asking to switch off the data roaming in the device settings. This task is complex because users must navigate into the numerous categories in the smartphone setting and localize one element through understanding the setting organization. The second case regarding analysis of visual content corresponds to the search for a route in a map, a more complex exploration task than the search for an element as a whole. The lowest cognitive load is observed in the task about searching into a set of textual elements, in the experiment case, to search for a given date in the calendar. Figure 2 shows the graphical representation of the average values of each task.

5. Conclusions

The present work is the beginning of a new research line that aims to help in the evaluation of cognitive abilities and contribute to the early diagnosis of mild cognitive impairment through the continuous analysis of the interaction with smartphones. The first steps have focused on two aspects that correspond to the main contributions of this article: the taxonomy of tasks with smartphones named HuSBIT-10, and the analysis of the cognitive load of typical tasks. The taxonomy is based on similar classifications focused on other devices found in the literature, as well as on the cognitive components related to each of the tasks. This taxonomy can support researchers in human-computer interaction to have a model that classifies the types of interactions with smartphones. Secondly, the data obtained after the experiment to analyze the cognitive load of different tasks provides us with information about the associated mental effort of each one. It will serve as a starting framework to evaluate their performance over time and associate it with cognitive impairment. It is important to notice that the EEG headset used in this study has no medical or scientific purpose, so, although it is reliable, its accuracy is not very high. Also, the population used in this experiment can be insufficient to be statistically significant. Future work will focus on improving this experiment, both with a more accurate EEG measuring devices, and by increasing the population and the set of tasks to be performed. Moreover, future work will involve analyzing cognitive load when interacting with specific mobile application for people with special needs. Examples of this are augmented reality for guiding people with dementia [28,29], mobile-based biomedical signals measurement [30], and avatar-based apps for emotion management [31].