Human Behaviour Analysis through Smartphones †

: Human behaviour analysis through smartphone devices has been an active ﬁeld for more than a decade and there are still a lot of key aspects to be addressed. This paper surveys the state-of-the-art in human behaviour analysis based on smartphones. We categorise prior works into four main sensing modalities related to physical, cognitive, emotional and social behaviour. Finally, we conclude with the outcomes of this survey and we illustrate our ideas for future research in the area of human behaviour understanding.


Introduction
Human behaviour understanding has become one of the most promising research areas in healthcare.Many researchers have proved that following a healthy lifestyle behaviour can promote health and prevent diseases, while lifestyle risk behaviours (such as physical inactivity, poor diet, anxiety, and sleep disorders) are responsible for the likelihood of diseases and premature mortality [1].Thus, detecting and monitoring human behaviours can play a major role in promoting active and healthy ageing, which is a complex process that depends on physical, psychological and social changes [2].
The recent technological advances have enabled the recognition of human behaviour, unobtrusively.Smartphones have been one of the most promising devices to track and detect human behaviour, but also to provide coaching strategies [2].Smartphone-based research is gaining more and more popularity, enabling new potentials for monitoring people's health.In 2016, there were 2.1 billion of smartphone users worldwide, while this number is expected to rise to 2.5 billion by 2020 [2].Smartphone devices contain multiple sensors including motion sensors (e.g., accelerometer, gyroscope) that can track the motion, orientation and position of the user's body but also sensors that provide information related to the user-smartphone interaction such as screen-(un)locks or text typing.
In order to detect human behaviour, it is important to define the types of behaviour but also the different types of data that can be acquired, allowing reliable and valid measurements.According to Marc H. Bornstein et al. [3], human behaviour is described as "the potential and expressed capacity for physical, mental and social activity during the phases of human life".Behavioural lifestyle data contain information about the user's daily activities, mobility patterns, mental abilities and social interactions, which can be used to detect and predict physical, emotional, social and cognitive behaviour.For instance, physical behaviour is related to physical health issues (such as obesity and heart failure).Cognitive behaviour is related to individual performance of processing information and can be used to monitor user's skills in different fields (such as occupational and academic).Emotional behaviour is related to mental health issues (such as stress and depression) and social behaviour is related to user's social interactions issues (such as loneliness).
The main aim of this article is to review some of the most prominent works addressing the sensing and detection of human behaviour through smartphones.In order to analyze the studies on human behaviour, we divided human behaviour into physical, cognitive, emotional and social behaviour.Thus, this survey focuses on examining the existing studies based on the four aforementioned sensing modalities.Specifically, physical behaviour includes a range of human bodily movements such as physical activities (e.g., walking, standing, sitting, cycling), transitions (e.g., sitting-to-standing), and hand movements (e.g., eating a sandwich, hand waving).Cognitive behaviour stands for a range of mental actions or states related to perception, attention, memory, language skills, visuospatial processing and executive function (e.g., reading or problem solving).Emotional behaviour consists of a range of mental action or states related to mood (e.g., happiness, sadness, boredom).Social behaviour is related to a range of communicative interactions that take place through physical (e.g., face-to-face) and virtual (e.g., social networks, phone calls) means.
The succeeding sections are structured in the following way.Section 2 presents the methods that are used for this survey, concerning the review of the state-of-the-art literature.Section 3 elaborates, thoroughly, the results of this survey for smartphone-based studies, while Section 4 presents the analysis of our review.Finally, we outline our conclusions and remarks in Section 5.

Methods
The literature review was performed using two of the most popular web search engines for academic and scientific articles: 'Scopus' and 'Web of Science'.Some of the primary keywords for searching in these search engines were 'human behaviour', 'physical behaviour', 'social behaviour', 'cognitive behaviour' and 'emotional behaviour'.These keywords were also combined with 'smartphone' in order to limit the search to those works where smartphones are present.A significant number of published articles, journals, papers and other reports exist already in the field of human behaviour understanding.In order to limit the number of returned results, the following search query was used: (activit* OR emotion* OR cognit* OR social) AND (sensing OR monitoring OR behavio*) AND (smartphone* OR (smart AND phone*) OR (mobile AND phone*) OR (cell AND phone*)).
The initially yielded results were more than 5000 articles per database.In order to downsize this number, the range of publication year was defined from 2010 to 2017 (the articles were selected up to 20 December 2017).Furthermore, the document types were limited to articles, while the publication categories were adjusted, excluding no-related fields such as economics, chemistry and environmental studies.However, the exclusion criteria affected only the 30% of the yielded articles, resulting to more than 3000 articles for review.Consequently, a systematic review could not be thoroughly performed for the selected inclusion criteria.For this reason, priority was given to some of the most highly cited articles, but also to other relevant publications based on authors' judgment, resulting to 55 articles for definitive review.

Results
The recent technological advances enabled the release of smartphone devices with powerful specifications and enormous sensing possibilities.These devices have been increasingly used in healthcare and behavioural research in order to collect health data and deliver comprehensive healthcare information to practitioners, researchers and patients, by enhancing the ability to diagnose and track diseases [4].The main aim of this section is to elaborate on the sensing modalities that can be provided through a smartphone device in order to measure behaviour.Four sensing modalities exploited to date are presented in the following subsections, including physical activity, cognitive, emotional and social sensing.An overview of the existing studies on human behaviour analysis through smartphones is presented in Figure 1.Different types of sensor data are clustered based on the four sensing modalities, regarding the physical, cognitive, emotional and social behaviour.It is worth mentioning that phone activity stands for the interaction of a user with the device and varies per study.Overall, phone activity includes features related to battery level, ratio of incoming/outcoming calls, sent/received text messages, screen (un)locks and applications usage.

Physical Activity Sensing
The research on physical activity recognition has gained much attention during the recent years as an essential descriptor of human behaviour.For more than two decades, researchers have intensively explored the use of inertial sensors, such as accelerometers (ACC) and gyroscopes [5], to fairly quantify physical activity in epidemiological, surveillance, and intervention medicine [6].These devices fundamentally consist of an accelerometer, a small inertial sensor that records the movement of the body where the device is placed (e.g., wrist, arm, chest, hip, thigh, etc.).Smartphones, which natively incorporate these types of inertial sensors, have been studied for estimating human physical activity in both controlled and uncontrolled settings [7].Most studies place smartphones on different unobtrusive locations on body, either on trousers' and shirts' pockets or inside bags and backpacks, while other studies use unconventionally obtrusive straps to firmly attach smartphones to body locations (e.g., hip, wrist, arm, chest).
The use of mobile phones as stand-alone physical activity monitors has been explored during the last decade, as a technological follow-up of traditional accelerometer-based mechanisms.The first systems on physical activity monitoring through mobile phones were based on simple pedometers and step counters building on the acceleration measured through the built-in sensors [8,9].However, the use of more sophisticated machine learning techniques has enabled the extraction of more meaningful data for activity detection.For example, sedentary, ambulatory and commuting activities (e.g., walking, jogging, running, cycling, ascending/descending stairs, sitting, standing) have been successfully identified in [10][11][12][13][14]. Furthermore, the detection of more complex physical behaviours, including housework and other everyday activities, has also been proven feasible in a number of works [15,16].Some works have explored the use of these sensors to detect commuting activities and even determine the transportation means by analysing the natural vibration of the vehicle [17].Activity recognition systems based on inertial sensor data are shown to be dependent on the specific location and placement of the sensors [18,19].These limitations have been overcome in a number of works by either exploiting the use of location-independent features [13,20,21] or identifying in the first place the actual location of the smartphone to use a customised activity identification model [22].
The monitoring of people's location and mobility patterns has represented a paramount topic in mobile computing since its early days.Phithakkitnukoon, Horanont, Lorenzo, Shibasaki & Ratti [37] mentioned that GPS sensors have been possibly the primal source for mobility analysis.Despite their widespread use, the accuracy of these sensors has been quite limited in indoor settings, where the connection between mobile phone and satellites is normally lost.Accordingly, researchers proposed some alternate options to track the location of users indoor.One of the most effective ones is the so-called pedestrian dead reckoning (PDR), a path integration process based on calculating one's current position by using a previously determined position.The mobile phones built-in accelerometers, gyroscopes and magnetometers are normally used to this end [38][39][40].Some other approaches elaborate on the analysis of call logs [41], but also the signal strength of the Wi-Fi and Bluetooth network [42,43] in order to track the mobility of a user or group of users.Consequently, it is possible to detect the location and the user's current physical activity outdoors, through accelerometer and GPS traces for example, but also indoors through Wi-Fi and Bluetooth signal strength.

Cognitive Sensing
Cognitive sensing is possibly the most challenging task since there is not a smartphone sensor that can directly measure cognition-related processes.Humans' cognitive functions can be divided into six main categories, including perception, attention, memory, language skills, visuospatial processing and executive functioning, which can be monitored by assessing the performance at specific tasks [44].Specifically, the attentional system of a user can be evaluated by assessing the user's alertness, which modulates sensory, motor, and cognitive processing [45].For instance, fatigue, a diminished state of alertness and attention, can be related to motor vehicle accidents and other occupational mistakes [46].Thus, cognitive performance can vary significantly during the day and among different users, and can be affected by multiple individual factors, such as the need to sleep at a specific time and based on the body clock, or by user's social obligations.
Abdullah et al. [47] showed that alertness can oscillate approximately 30% depending on time and body clock type (circadian rhythms), while other factors such as the daylight saving time (practice of advancing clocks during summer), hours slept, and stimulant intake can influence alertness as well.For instance, a high level of alertness was noticed in their study when participants checked their phones more frequently and only for shorter periods of time, while low levels of alertness were noticed when participants engaged in more sustained use.Furthermore, Pijnenborg et al. [48] studied the use of SMS text messages to compensate for the effects of cognitive impairments in subjects with schizophrenia.
It is also possible to track user's behaviour and cognitive state by evaluating the screen touch events while using a smartphone device.Specifically, data on screen touches can be used to evaluate the speed and the response time to smartphone surveys in order to track short term cognitive states, such as attention and alertness [47,49].Additionally, screen touch patterns can be employed to detect quality metrics, sleeping duration and whether the subject uses the phone during night-time, leading to the monitoring of long-term cognitive states in many mental disorders, such as schizophrenia and depression [42].A recent work has explored the use of so-called cognitive experience sampling methods, which go beyond traditional clinical questionnaires while introducing a myriad of tasks that users can perform on the phone, ubiquitously and opportunistically, in order to measure the cognitive functioning of the user [50].

Emotional Sensing
There is a great number of studies using smartphone technologies for detecting emotional states and mood disorders.The most common way to detect emotional states on a smartphone is based on collecting user's self-reported data through an application, sometimes also including cognitive and physical behavioural states [42].For instance, MONARCA [51] is a therapeutic application for bipolar disorder patients using a simple questionnaire through a smartphone device for self-reporting user's mood score.
Smartphone devices can also be used to automatically sample data for the monitoring and treatment of many mental disorders.Specifically, assumptions can be made about the subject's current mood through statistical analysis of some mobile sensing modalities, such as variations in phone usage patterns, texting and calling [42].For instance, an unusual increased number of outgoing phone calls could be related to a change in the mental state of patient.Muaremi, Gravenhorst, Grunerbl, Arnrich & Troster [52] examined the phone call parameters and showed that the phone call duration and the accumulated talking time can be used to predict bipolar disorder episodes, significantly.Furthermore, many studies consider audio and voice recognition methods in order to extract user's mood and detect short-term emotional behaviour through sound and phone call features.Examples of phone call features are the number of calls, the number of involved ID phone callers, and the sum, average and standard deviation length of calls [53].Sound features consist of speech and voice features acquired through phone's microphone including ,for example, features related to phone call interaction (e.g., the average speaking length and duration) and audio recordings (detect emotions based on user's voice) [53].Vanello et al. [54] used speech signals to extract voice features related to pitch and pitch changes, combined with acquired ECG signals from external device, in order to characterise the mood of bipolar patients to depressive, hypomanic and euthymic state.
Another category of smartphone applications that run quietly in the background, without demanding an active request information from the user, is this for tracking text messages.Tausczik and Pennebaker [55] studied the content of SMS messages, using classification tools to scan the words from a SMS text and interpret it to psychologically meaningful terms for estimating the subject's emotions and mood.Similarly, Rutland, Sheets, and Young [56] studied the mood modification and the addictive attitudes towards SMS, by analysing the number of sent and received SMS messages.
Additionally, other smartphone sensing modalities such as GPS, accelerometer and Wi-Fi signal strength, most frequently used for physical activity sensing, can also provide here valuable information about the patients' state [42].Lomranz, Bergman, Eyal & Shmotkin [57] showed that the outdoor behaviour of patients is related to their mental states.Gravenhorst et al. [42] explained that depressed patients can be isolated into their rooms, spending most of the time in bed, while manic patients tend to travel long distances in an unusual way.Similarly, Osmani et al. [58] found that the activity level of bipolar patients is related to their mental state; a reduced activity level could be interpreted as a depressed episode, while an enhanced activity level could indicate the beginning of a manic episode.Burns et al. [43] developed the Mobilyze!system for patients with depression, sampling data from the accelerometer, GPS, Bluetooth, Wi-Fi, ambient light and phone usage analytics in order to target depression by detecting user's location and emotional states, such as sadness, happiness, anger and anxiety.
Similar to the ongoing research on detecting the different mental states through a smartphone device, there are many studies focusing on the recognition of emotional states, such as happiness and boredom, as well.Pielot, Dingler, Pedro & Oliver [59] developed a more sophisticated machine learning approach to automatically identify user's boredom situations from mobile phone usage, context and demographics.Other emotional states such as happiness have been found to be detectable through the analysis of mobile phone data [52].Bogomolov, Lepri & Pianesi [60] used an extensive set of indicators obtained while using a mobile phone, such as call logs, SMS and Bluetooth proximity data, plus environmental (mainly weather) data and personality traits, which can be intelligently combined to estimate people's daily happiness.
The use of these applications enables medicine prescription, where the psychiatrists have access to the patients' database, being able to continuously adjust the prescriptions [42].Thus, the psychiatric is able to involve and adjust the medicine on time, when the patient goes from a depressed to a manic state for example, by monitoring the short-term emotional behaviour.Moreover, physical activity levels measured through the smartphone's acceleration sensors and GPS traces have been proved adequate for psychiatric assessment of depression, when long-term data series over a period of days or weeks are considered [42,58].

Social Sensing
Social networking sites, such as Twitter and Facebook, have been used to study users' social behaviour through computers, smartphones and tablets.Users' actions, posts, comments and sentiments have been examined in order to extract useful information through the internet regarding their mood but also their social activity and interaction with other users [61][62][63].Additionally, there is an ongoing research on sampling social behaviour data through smartphone devices, especially focusing on the correlation of social inactivity with many mental disorders.Burns et al. [43] developed a system for patients with depression in order to detect user's location and measure the interaction level with friends.They sampled data for emotional but also for social sensing, using sensors such as accelerometer, GPS, Wi-Fi, Bluetooth and other parameters from phone usage.Frost et al. [64] studied the number of ingoing and outgoing phone calls and text messages in order to detect short term social behaviour and detect the current mood of patients.For instance, mentally ill patients tend to get talkative in a manic phase [42], while a low level of social interaction is correlated with depression [65].
Furthermore, Lane et al. [36] measured social isolation based on the total duration of ambient conversations, by using the mobile phone microphone.Moturu et al. [66] studied the Bluetooth proximity detection to estimate the sociability level of people and predict their mood.Mobile devices can also be used to measure long-term behavioural cues and social signals and reveal relevant determinants of health.For example, the analysis of aggregated and anonymized call records captured from the mobile phone infrastructure is sufficient to characterize human behaviour in dense areas [67] and during critical events such as natural disasters [68].

Discussion
In order to detect human behaviour, data from different sensors have to be processed and analyzed.An overview of these sensors is presented in Table 1.Motion sensors, such as accelerometer and gyroscope, but also other sensors, such as GPS, Wi-Fi and Bluetooth (signal strength), have been used to detect the user's activity and movement (for example sitting, walking, running) [42].Data related to screen touch events and user's response time can be used to track short term cognitive states, such as attention and alertness [49].Phone calls and text messages can be used to monitor the user's social life [64], while audio and microphone signals, combined with the user's physical activity, can be used to detect user's mood (boredom, happiness, anxiety, etc.) [52].One of the outcomes of this review is that most of the studies focus on the detection of physical activities, while there are only a few for detecting cognitive and social behaviour.To the best of our knowledge, there is not a study based on a holistic approach for detecting physical, cognitive, emotional and social behaviour at the same time.Furthermore, there is not a clear distinction on behaviours at different levels of granularity and their impact on health outcomes.For instance, the detection of short-infrequent calls can be associated to social behaviour (e.g., being socially isolated), but also to cognitive behaviour (e.g., suffering from dementia).
Additionally, even though smartphones can be used to record behavioural data continuously, there is not a clear investigation for detecting behaviours over different periods of time (e.g., hours, days, weeks, months).For instance, detecting users' short-term social behaviours of being socially inactive over a long period of time, consecutively, could foresee mood states and disorders, such as loneliness and depression (which might also affect long-term emotional and cognitive behaviour).Thus, a distinction between short and long term behaviours has to be examined, as well.
Many studies explore the optimal placement of the smartphone on the body in order to detect physical activities accurately, while a few focus on the energy expenditure and battery consumption for recording smartphone data, continuously.However, continuous smartphone monitoring might have an impact on the device battery life, but also to the device storage capacity, and needs to be further investigated.Moreover, audio recordings, but also text messages, might contain sensitive information that may arise privacy concerns.Thus, privacy and security is another important issue that has to be thoroughly addressed, combined with the perceptions of end-users.Consequently, there are still many issues that should be addressed in future works for human behaviour analysis through smartphones, regarding the smartphone placement on the body, the limitations due to the battery drain, the arising of privacy and security concerns, but also the end-users' perception.

Conclusions
Despite the fact that human behaviour understanding cannot be easily addressed, it is still a challenging research area with a great impact on healthcare.Based on the existing studies in human behaviour understanding, it is clear that combining all the smartphone sensing modalities has not been investigated thoroughly.Specifically, we have observed that each component of behaviour has been mostly addressed individually.Hence, given the complex nature of human behaviour, we consider future works should address this problem in a more holistic approach where multiple behavioural aspects are measured at once.

Figure 1 .
Figure 1.An overview of the state-of-the-art literature on smartphone-based human behaviour analysis.Different types of sensor data are clustered based on the four sensing modalities, regarding the physical, cognitive, emotional and social behaviour.It is worth mentioning that phone activity stands for the interaction of a user with the device and varies per study.Overall, phone activity includes features related to battery level, ratio of incoming/outcoming calls, sent/received text messages, screen (un)locks and applications usage.

Table 1 .
Human behaviour analysis through smartphones.