Next Article in Journal
Evitapp: Persuasive Application for Physical Activity and Smoking Cessation
Previous Article in Journal
Control of a Boost Converter to Improve the Performance of a Photovoltaic System in a Microgrid
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Proceedings
Volume 2
Issue 19
10.3390/proceedings2191243
proceedings-logo
Submit to this Journal
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Proceeding Paper

Human Behaviour Analysis through Smartphones †

by
Kostas Konsolakis
1,*,
Hermie Hermens
1,2,
Claudia Villalonga
3,
Miriam Vollenbroek-Hutten
1,4 and
Oresti Banos
1,5
1
Biomedical Signals and Systems Group, University of Twente, 7522 NB Enschede, The Netherlands
2
Telemedicine Group, Roessingh Research and Development, 7500 AH Enschede, The Netherlands
3
School of Engineering and Technology, Universidad Internacional de La Rioja, 26006 Logroño, Spain
4
Ziekenhuisgroep Twente, 7609 PP Almelo, The Netherlands
5
Department of Computer Architecture & Computer Technology, University of Granada, E-18071 Granada, Spain
*
Author to whom correspondence should be addressed.
†
Presented at the 12th International Conference on Ubiquitous Computing and Ambient Intelligence (UCAmI 2018), Punta Cana, Dominican Republic, 4–7 December 2018.
Proceedings 2018, 2(19), 1243; https://doi.org/10.3390/proceedings2191243
Published: 30 October 2018
(This article belongs to the Proceedings of UCAmI 2018)
Browse Figure
Versions Notes

Abstract

:
Human behaviour analysis through smartphone devices has been an active field for more than a decade and there are still a lot of key aspects to be addressed. This paper surveys the state-of-the-art in human behaviour analysis based on smartphones. We categorise prior works into four main sensing modalities related to physical, cognitive, emotional and social behaviour. Finally, we conclude with the outcomes of this survey and we illustrate our ideas for future research in the area of human behaviour understanding.

1. Introduction

Human behaviour understanding has become one of the most promising research areas in healthcare. Many researchers have proved that following a healthy lifestyle behaviour can promote health and prevent diseases, while lifestyle risk behaviours (such as physical inactivity, poor diet, anxiety, and sleep disorders) are responsible for the likelihood of diseases and premature mortality [1]. Thus, detecting and monitoring human behaviours can play a major role in promoting active and healthy ageing, which is a complex process that depends on physical, psychological and social changes [2].
The recent technological advances have enabled the recognition of human behaviour, unobtrusively. Smartphones have been one of the most promising devices to track and detect human behaviour, but also to provide coaching strategies [2]. Smartphone-based research is gaining more and more popularity, enabling new potentials for monitoring people’s health. In 2016, there were 2.1 billion of smartphone users worldwide, while this number is expected to rise to 2.5 billion by 2020 [2]. Smartphone devices contain multiple sensors including motion sensors (e.g., accelerometer, gyroscope) that can track the motion, orientation and position of the user’s body but also sensors that provide information related to the user-smartphone interaction such as screen-(un)locks or text typing.
In order to detect human behaviour, it is important to define the types of behaviour but also the different types of data that can be acquired, allowing reliable and valid measurements. According to Marc H. Bornstein et al. [3], human behaviour is described as “the potential and expressed capacity for physical, mental and social activity during the phases of human life”. Behavioural lifestyle data contain information about the user’s daily activities, mobility patterns, mental abilities and social interactions, which can be used to detect and predict physical, emotional, social and cognitive behaviour. For instance, physical behaviour is related to physical health issues (such as obesity and heart failure). Cognitive behaviour is related to individual performance of processing information and can be used to monitor user’s skills in different fields (such as occupational and academic). Emotional behaviour is related to mental health issues (such as stress and depression) and social behaviour is related to user’s social interactions issues (such as loneliness).
The main aim of this article is to review some of the most prominent works addressing the sensing and detection of human behaviour through smartphones. In order to analyze the studies on human behaviour, we divided human behaviour into physical, cognitive, emotional and social behaviour. Thus, this survey focuses on examining the existing studies based on the four aforementioned sensing modalities. Specifically, physical behaviour includes a range of human bodily movements such as physical activities (e.g., walking, standing, sitting, cycling), transitions (e.g., sitting-to-standing), and hand movements (e.g., eating a sandwich, hand waving). Cognitive behaviour stands for a range of mental actions or states related to perception, attention, memory, language skills, visuospatial processing and executive function (e.g., reading or problem solving). Emotional behaviour consists of a range of mental action or states related to mood (e.g., happiness, sadness, boredom). Social behaviour is related to a range of communicative interactions that take place through physical (e.g., face-to-face) and virtual (e.g., social networks, phone calls) means.
The succeeding sections are structured in the following way. Section 2 presents the methods that are used for this survey, concerning the review of the state-of-the-art literature. Section 3 elaborates, thoroughly, the results of this survey for smartphone-based studies, while Section 4 presents the analysis of our review. Finally, we outline our conclusions and remarks in Section 5.

2. Methods

The literature review was performed using two of the most popular web search engines for academic and scientific articles: `Scopus’ and `Web of Science’. Some of the primary keywords for searching in these search engines were `human behaviour’, `physical behaviour’, `social behaviour’, `cognitive behaviour’ and `emotional behaviour’. These keywords were also combined with `smartphone’ in order to limit the search to those works where smartphones are present. A significant number of published articles, journals, papers and other reports exist already in the field of human behaviour understanding. In order to limit the number of returned results, the following search query was used: (activit* OR emotion* OR cognit* OR social) AND (sensing OR monitoring OR behavio*) AND (smartphone* OR (smart AND phone*) OR (mobile AND phone*) OR (cell AND phone*)).
The initially yielded results were more than 5000 articles per database. In order to downsize this number, the range of publication year was defined from 2010 to 2017 (the articles were selected up to 20 December 2017). Furthermore, the document types were limited to articles, while the publication categories were adjusted, excluding no-related fields such as economics, chemistry and environmental studies. However, the exclusion criteria affected only the 30% of the yielded articles, resulting to more than 3000 articles for review. Consequently, a systematic review could not be thoroughly performed for the selected inclusion criteria. For this reason, priority was given to some of the most highly cited articles, but also to other relevant publications based on authors’ judgment, resulting to 55 articles for definitive review.

3. Results

The recent technological advances enabled the release of smartphone devices with powerful specifications and enormous sensing possibilities. These devices have been increasingly used in healthcare and behavioural research in order to collect health data and deliver comprehensive healthcare information to practitioners, researchers and patients, by enhancing the ability to diagnose and track diseases [4]. The main aim of this section is to elaborate on the sensing modalities that can be provided through a smartphone device in order to measure behaviour. Four sensing modalities exploited to date are presented in the following subsections, including physical activity, cognitive, emotional and social sensing. An overview of the existing studies on human behaviour analysis through smartphones is presented in Figure 1.

3.1. Physical Activity Sensing

The research on physical activity recognition has gained much attention during the recent years as an essential descriptor of human behaviour. For more than two decades, researchers have intensively explored the use of inertial sensors, such as accelerometers (ACC) and gyroscopes [5], to fairly quantify physical activity in epidemiological, surveillance, and intervention medicine [6]. These devices fundamentally consist of an accelerometer, a small inertial sensor that records the movement of the body where the device is placed (e.g., wrist, arm, chest, hip, thigh, etc.). Smartphones, which natively incorporate these types of inertial sensors, have been studied for estimating human physical activity in both controlled and uncontrolled settings [7]. Most studies place smartphones on different unobtrusive locations on body, either on trousers’ and shirts’ pockets or inside bags and backpacks, while other studies use unconventionally obtrusive straps to firmly attach smartphones to body locations (e.g., hip, wrist, arm, chest).
The use of mobile phones as stand-alone physical activity monitors has been explored during the last decade, as a technological follow-up of traditional accelerometer-based mechanisms. The first systems on physical activity monitoring through mobile phones were based on simple pedometers and step counters building on the acceleration measured through the built-in sensors [8,9]. However, the use of more sophisticated machine learning techniques has enabled the extraction of more meaningful data for activity detection. For example, sedentary, ambulatory and commuting activities (e.g., walking, jogging, running, cycling, ascending/descending stairs, sitting, standing) have been successfully identified in [10,11,12,13,14]. Furthermore, the detection of more complex physical behaviours, including housework and other everyday activities, has also been proven feasible in a number of works [15,16]. Some works have explored the use of these sensors to detect commuting activities and even determine the transportation means by analysing the natural vibration of the vehicle [17]. Activity recognition systems based on inertial sensor data are shown to be dependent on the specific location and placement of the sensors [18,19]. These limitations have been overcome in a number of works by either exploiting the use of location-independent features [13,20,21] or identifying in the first place the actual location of the smartphone to use a customised activity identification model [22].
Mobile phones have not only been used to detect body movement and orientation but also to identify abnormal physical behaviour, and especially gait [12,23,24,25], fall [26,27,28,29], movement transitions [30,31] and postural instability or balance [32,33,34]. Furthermore, specific activities of daily relevance such as sleeping have also been inferred by analyzing mobile phone usage patterns [35,36].
The monitoring of people’s location and mobility patterns has represented a paramount topic in mobile computing since its early days. Phithakkitnukoon, Horanont, Lorenzo, Shibasaki & Ratti [37] mentioned that GPS sensors have been possibly the primal source for mobility analysis. Despite their widespread use, the accuracy of these sensors has been quite limited in indoor settings, where the connection between mobile phone and satellites is normally lost. Accordingly, researchers proposed some alternate options to track the location of users indoor. One of the most effective ones is the so-called pedestrian dead reckoning (PDR), a path integration process based on calculating one’s current position by using a previously determined position. The mobile phones built-in accelerometers, gyroscopes and magnetometers are normally used to this end [38,39,40]. Some other approaches elaborate on the analysis of call logs [41], but also the signal strength of the Wi-Fi and Bluetooth network [42,43] in order to track the mobility of a user or group of users. Consequently, it is possible to detect the location and the user’s current physical activity outdoors, through accelerometer and GPS traces for example, but also indoors through Wi-Fi and Bluetooth signal strength.

3.2. Cognitive Sensing

Cognitive sensing is possibly the most challenging task since there is not a smartphone sensor that can directly measure cognition-related processes. Humans’ cognitive functions can be divided into six main categories, including perception, attention, memory, language skills, visuospatial processing and executive functioning, which can be monitored by assessing the performance at specific tasks [44]. Specifically, the attentional system of a user can be evaluated by assessing the user’s alertness, which modulates sensory, motor, and cognitive processing [45]. For instance, fatigue, a diminished state of alertness and attention, can be related to motor vehicle accidents and other occupational mistakes [46]. Thus, cognitive performance can vary significantly during the day and among different users, and can be affected by multiple individual factors, such as the need to sleep at a specific time and based on the body clock, or by user’s social obligations.
Abdullah et al. [47] showed that alertness can oscillate approximately 30% depending on time and body clock type (circadian rhythms), while other factors such as the daylight saving time (practice of advancing clocks during summer), hours slept, and stimulant intake can influence alertness as well. For instance, a high level of alertness was noticed in their study when participants checked their phones more frequently and only for shorter periods of time, while low levels of alertness were noticed when participants engaged in more sustained use. Furthermore, Pijnenborg et al. [48] studied the use of SMS text messages to compensate for the effects of cognitive impairments in subjects with schizophrenia.
It is also possible to track user’s behaviour and cognitive state by evaluating the screen touch events while using a smartphone device. Specifically, data on screen touches can be used to evaluate the speed and the response time to smartphone surveys in order to track short term cognitive states, such as attention and alertness [47,49]. Additionally, screen touch patterns can be employed to detect quality metrics, sleeping duration and whether the subject uses the phone during night-time, leading to the monitoring of long-term cognitive states in many mental disorders, such as schizophrenia and depression [42]. A recent work has explored the use of so-called cognitive experience sampling methods, which go beyond traditional clinical questionnaires while introducing a myriad of tasks that users can perform on the phone, ubiquitously and opportunistically, in order to measure the cognitive functioning of the user [50].

3.3. Emotional Sensing

There is a great number of studies using smartphone technologies for detecting emotional states and mood disorders. The most common way to detect emotional states on a smartphone is based on collecting user’s self-reported data through an application, sometimes also including cognitive and physical behavioural states [42]. For instance, MONARCA [51] is a therapeutic application for bipolar disorder patients using a simple questionnaire through a smartphone device for self-reporting user’s mood score.
Smartphone devices can also be used to automatically sample data for the monitoring and treatment of many mental disorders. Specifically, assumptions can be made about the subject’s current mood through statistical analysis of some mobile sensing modalities, such as variations in phone usage patterns, texting and calling [42]. For instance, an unusual increased number of outgoing phone calls could be related to a change in the mental state of patient. Muaremi, Gravenhorst, Grunerbl, Arnrich & Troster [52] examined the phone call parameters and showed that the phone call duration and the accumulated talking time can be used to predict bipolar disorder episodes, significantly. Furthermore, many studies consider audio and voice recognition methods in order to extract user’s mood and detect short-term emotional behaviour through sound and phone call features. Examples of phone call features are the number of calls, the number of involved ID phone callers, and the sum, average and standard deviation length of calls [53]. Sound features consist of speech and voice features acquired through phone’s microphone including ,for example, features related to phone call interaction (e.g., the average speaking length and duration) and audio recordings (detect emotions based on user’s voice) [53]. Vanello et al. [54] used speech signals to extract voice features related to pitch and pitch changes, combined with acquired ECG signals from external device, in order to characterise the mood of bipolar patients to depressive, hypomanic and euthymic state.
Another category of smartphone applications that run quietly in the background, without demanding an active request information from the user, is this for tracking text messages. Tausczik and Pennebaker [55] studied the content of SMS messages, using classification tools to scan the words from a SMS text and interpret it to psychologically meaningful terms for estimating the subject’s emotions and mood. Similarly, Rutland, Sheets, and Young [56] studied the mood modification and the addictive attitudes towards SMS, by analysing the number of sent and received SMS messages.
Additionally, other smartphone sensing modalities such as GPS, accelerometer and Wi-Fi signal strength, most frequently used for physical activity sensing, can also provide here valuable information about the patients’ state [42]. Lomranz, Bergman, Eyal & Shmotkin [57] showed that the outdoor behaviour of patients is related to their mental states. Gravenhorst et al. [42] explained that depressed patients can be isolated into their rooms, spending most of the time in bed, while manic patients tend to travel long distances in an unusual way. Similarly, Osmani et al. [58] found that the activity level of bipolar patients is related to their mental state; a reduced activity level could be interpreted as a depressed episode, while an enhanced activity level could indicate the beginning of a manic episode. Burns et al. [43] developed the Mobilyze! system for patients with depression, sampling data from the accelerometer, GPS, Bluetooth, Wi-Fi, ambient light and phone usage analytics in order to target depression by detecting user’s location and emotional states, such as sadness, happiness, anger and anxiety.
Similar to the ongoing research on detecting the different mental states through a smartphone device, there are many studies focusing on the recognition of emotional states, such as happiness and boredom, as well. Pielot, Dingler, Pedro & Oliver [59] developed a more sophisticated machine learning approach to automatically identify user’s boredom situations from mobile phone usage, context and demographics. Other emotional states such as happiness have been found to be detectable through the analysis of mobile phone data [52]. Bogomolov, Lepri & Pianesi [60] used an extensive set of indicators obtained while using a mobile phone, such as call logs, SMS and Bluetooth proximity data, plus environmental (mainly weather) data and personality traits, which can be intelligently combined to estimate people’s daily happiness.
The use of these applications enables medicine prescription, where the psychiatrists have access to the patients’ database, being able to continuously adjust the prescriptions [42]. Thus, the psychiatric is able to involve and adjust the medicine on time, when the patient goes from a depressed to a manic state for example, by monitoring the short-term emotional behaviour. Moreover, physical activity levels measured through the smartphone’s acceleration sensors and GPS traces have been proved adequate for psychiatric assessment of depression, when long-term data series over a period of days or weeks are considered [42,58].

3.4. Social Sensing

Social networking sites, such as Twitter and Facebook, have been used to study users’ social behaviour through computers, smartphones and tablets. Users’ actions, posts, comments and sentiments have been examined in order to extract useful information through the internet regarding their mood but also their social activity and interaction with other users [61,62,63]. Additionally, there is an ongoing research on sampling social behaviour data through smartphone devices, especially focusing on the correlation of social inactivity with many mental disorders. Burns et al. [43] developed a system for patients with depression in order to detect user’s location and measure the interaction level with friends. They sampled data for emotional but also for social sensing, using sensors such as accelerometer, GPS, Wi-Fi, Bluetooth and other parameters from phone usage. Frost et al. [64] studied the number of ingoing and outgoing phone calls and text messages in order to detect short term social behaviour and detect the current mood of patients. For instance, mentally ill patients tend to get talkative in a manic phase [42], while a low level of social interaction is correlated with depression [65].
Furthermore, Lane et al. [36] measured social isolation based on the total duration of ambient conversations, by using the mobile phone microphone. Moturu et al. [66] studied the Bluetooth proximity detection to estimate the sociability level of people and predict their mood. Mobile devices can also be used to measure long-term behavioural cues and social signals and reveal relevant determinants of health. For example, the analysis of aggregated and anonymized call records captured from the mobile phone infrastructure is sufficient to characterize human behaviour in dense areas [67] and during critical events such as natural disasters [68].

4. Discussion

In order to detect human behaviour, data from different sensors have to be processed and analyzed. An overview of these sensors is presented in Table 1. Motion sensors, such as accelerometer and gyroscope, but also other sensors, such as GPS, Wi-Fi and Bluetooth (signal strength), have been used to detect the user’s activity and movement (for example sitting, walking, running) [42]. Data related to screen touch events and user’s response time can be used to track short term cognitive states, such as attention and alertness [49]. Phone calls and text messages can be used to monitor the user’s social life [64], while audio and microphone signals, combined with the user’s physical activity, can be used to detect user’s mood (boredom, happiness, anxiety, etc.) [52].
One of the outcomes of this review is that most of the studies focus on the detection of physical activities, while there are only a few for detecting cognitive and social behaviour. To the best of our knowledge, there is not a study based on a holistic approach for detecting physical, cognitive, emotional and social behaviour at the same time. Furthermore, there is not a clear distinction on behaviours at different levels of granularity and their impact on health outcomes. For instance, the detection of short-infrequent calls can be associated to social behaviour (e.g., being socially isolated), but also to cognitive behaviour (e.g., suffering from dementia).
Additionally, even though smartphones can be used to record behavioural data continuously, there is not a clear investigation for detecting behaviours over different periods of time (e.g., hours, days, weeks, months). For instance, detecting users’ short-term social behaviours of being socially inactive over a long period of time, consecutively, could foresee mood states and disorders, such as loneliness and depression (which might also affect long-term emotional and cognitive behaviour). Thus, a distinction between short and long term behaviours has to be examined, as well.
Many studies explore the optimal placement of the smartphone on the body in order to detect physical activities accurately, while a few focus on the energy expenditure and battery consumption for recording smartphone data, continuously. However, continuous smartphone monitoring might have an impact on the device battery life, but also to the device storage capacity, and needs to be further investigated. Moreover, audio recordings, but also text messages, might contain sensitive information that may arise privacy concerns. Thus, privacy and security is another important issue that has to be thoroughly addressed, combined with the perceptions of end-users. Consequently, there are still many issues that should be addressed in future works for human behaviour analysis through smartphones, regarding the smartphone placement on the body, the limitations due to the battery drain, the arising of privacy and security concerns, but also the end-users’ perception.

5. Conclusions

Despite the fact that human behaviour understanding cannot be easily addressed, it is still a challenging research area with a great impact on healthcare. Based on the existing studies in human behaviour understanding, it is clear that combining all the smartphone sensing modalities has not been investigated thoroughly. Specifically, we have observed that each component of behaviour has been mostly addressed individually. Hence, given the complex nature of human behaviour, we consider future works should address this problem in a more holistic approach where multiple behavioural aspects are measured at once.

Author Contributions

O.B. conceived this work. K.K. performed the literature search and analysis. H.H., M.V. and C.V. contributed to the interpretation of results and discussion of findings. K.K. and O.B. wrote the paper. O.B. reviewed the paper.

Acknowledgments

This work has received funding from the European Union’s Horizon 2020 research and innovation programme under Grant Agreement #769553. This result only reflects the author’s view and the EU is not responsible for any use that may be made of the information it contains. This work has also been supported by the Dutch UT-CTIT project HoliBehave and in collaboration with the research project “Progress in Computer Architectures for Automatic Learning using Heterogeneous Sources: Health and Well-Being Applications” (TIN2015-71873-R).

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Conner, M.; Norman, P. Health behaviour: Current issues and challenges. Psychol. Health 2017, 895–906. [Google Scholar] [CrossRef] [PubMed]
  2. Helbostad, J.L.; Vereijken, B.; Becker, C.; Todd, C.; Taraldsen, K.; Pijnappels, M.; Aminian, K.; Mellone, S. Mobile Health Applications to Promote Active and Healthy Ageing. Sensors 2017, 17, 622. [Google Scholar] [CrossRef] [PubMed]
  3. Bornstein, M.H.; Kagan, J.; Lerner, R.M. “Human Behaviour.” Encyclopædia Britannica, Encyclopædia Britannica, Inc. Available online: http://www.britannica.com/topic/human-behavior (accessed on 24 April 2018).
  4. Silva, B.M.C.; Rodrigues, J.J.P.C.; de la Torre Díez, I.; López-Coronado, M.; Saleem, K. Mobile-health: A review of current state in 2015. J. Biomed. Inf. 2015, 56, 265–272. [Google Scholar] [CrossRef]
  5. John, D.; Freedson, P. ActiGraph and Actical Physical Activity Monitors. Med. Sci. Sport. Exerc. 2012, 44, S86–S89. [Google Scholar] [CrossRef] [PubMed]
  6. Buman, M.P.; Hekler, E.B.; Haskell, W.L.; Pruitt, L.; Conway, T.L.; Cain, K.L.; Sallis, J.F.; Saelens, B.E.; Frank, L.D.; King, A.C. Objective light-intensity physical activity associations with rated health in older adults. Am. J. Epidemiol. 2010, 172, 1155–1165. [Google Scholar] [CrossRef] [PubMed]
  7. Hekler, E.B.; Buman, M.P.; Grieco, L.; Rosenberger, M.; Winter, S.J.; Haskell, W.; King, A.C. Validation of Physical Activity Tracking via Android Smartphones Compared to ActiGraph Accelerometer: Laboratory-Based and Free-Living Validation Studies. JMIR mHealth uHealth 2015, 3, e36. [Google Scholar] [CrossRef]
  8. Garcia, E.; Ding, H.; Sarela, A.; Karunanithi, M. Can a mobile phone be used as a pedometer in an outpatient cardiac rehabilitation program? In Proceedings of the 2010 IEEE/ICME International Conference on Complex Medical Engineering, Gold Coast, Australia, 13–15 July 2010; pp. 250–253. [Google Scholar]
  9. Mladenov, M.; Mock, M. A step counter service for Java-enabled devices using a built-in accelerometer. In Proceedings of the 1st International Workshop on Context-Aware Middleware and Services Affiliated with the 4th International Conference on Communication System Software and Middleware (COMSWARE 2009)- CAMS ’09, Dublin, Ireland, 16 June 2009; p. 1. [Google Scholar]
  10. Kwapisz, J.R.; Weiss, G.M.; Moore, S.A. Activity recognition using cell phone accelerometers. SIGKDD Explor. Newsl. 2011, 12, 74–82. [Google Scholar] [CrossRef]
  11. Lau, S.L.; David, K. Movement recognition using the accelerometer in smartphones. In 2010 Future Network and Mobile Summit; IEEE: Piscataway Township, NJ, USA, 2010; pp. 1–9. [Google Scholar]
  12. Martín, H.; Bernardos, A.M.; Iglesias, J.; Casar, J.R. Activity logging using lightweight classification techniques in mobile devices. Pers. Ubiquitous Comput. 2013, 17, 675–695. [Google Scholar] [CrossRef]
  13. Sun, L.; Zhang, D.; Li, B.; Guo, B.; Li, S. Activity recognition on an accelerometer embedded mobile phone with varying positions and orientations. In International Conference on Ubiquitous Intelligence and Computing; Springer: Berlin/Heidelberg, Germany, 2010; Volume 6406, pp. 548–562. [Google Scholar]
  14. Banos, O.; Villalonga, C.; Garcia, R.; Saez, A.; Damas, M.; Holgado-Terriza, J.A.; Lee, S.; Pomares, H.; Rojas, I. Design, implementation and validation of a novel open framework for agile development of mobile health applications. Biomed. Eng. Online 2015, 14, S6. [Google Scholar] [CrossRef]
  15. Khan, A.M.; Tufail, A.; Khattak, A.M.; Laine, T.H. Activity recognition on smartphones via sensor-fusion and KDA-based SVMs. Int. J. Distrib. Sens. Netw. 2014, 10, 503291. [Google Scholar] [CrossRef]
  16. Ouchi, K.; Doi, M. Indoor-outdoor activity recognition by a smartphone. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing, Pittsburgh, PA, USA, 5–8 September 2012. [Google Scholar]
  17. Hur, T.; Bang, J.; Kim, D.; Banos, O.; Lee, S. Smartphone location-independent physical activity recognition based on transportation natural vibration analysis. Sensors 2017, 17, 931. [Google Scholar] [CrossRef] [PubMed]
  18. Banos, O.; Damas, M.; Pomares, H.; Rojas, I. On the use of sensor fusion to reduce the impact of rotational and additive noise in human activity recognition. Sensors 2012, 12, 8039–8054. [Google Scholar] [CrossRef] [PubMed]
  19. Banos, O.; Toth, M.A.; Damas, M.; Pomares, H.; Rojas, I. Dealing with the effects of sensor displacement in wearable activity recognition. Sensors 2014, 14, 9995–10023. [Google Scholar] [CrossRef]
  20. Han, M.; Vinh, L.T.; Lee, Y.K.; Lee, S. Comprehensive context recognizer based on multimodal sensors in a smartphone. Sensors 2012, 12, 12588–12605. [Google Scholar] [CrossRef]
  21. Han, M.; Bang, J.H.; Nugent, C.; McClean, S.; Lee, S. A Lightweight Hierarchical Activity Recognition Framework Using Smartphone Sensors. Sensors 2014, 14, 16181–16195. [Google Scholar] [CrossRef]
  22. Guiry, J.J.; Karr, C.J.; van de Ven, P.; Nelson, J.; Begale, M. A single vs. multi-sensor approach to enhanced detection of smartphone placement. In Proceedings of the 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Chicago, IL, USA, 26–30 August 2014; pp. 3691–3694. [Google Scholar]
  23. Isho, T.; Tashiro, H.; Usuda, S. Accelerometry-based gait characteristics evaluated using a smartphone and their association with fall risk in people with chronic stroke. J. Stroke Cerebrovasc. Dis. 2015, 24, 1305–1311. [Google Scholar] [CrossRef]
  24. Mazilu, S.; Hardegger, M.; Zhu, Z.; Roggen, D.; Troster, G.; Plotnik, M.; Hausdorff, J.M. Online detection of freezing of gait with smartphones and machine learning techniques. In Proceedings of the 2012 6th International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth) and Workshops, San Diego, CA, USA, 21–24 May 2012; pp. 123–130. [Google Scholar]
  25. Yamada, M.; Aoyama, T.; Mori, S.; Nishiguchi, S.; Okamoto, K.; Ito, T.; Muto, S.; Ishihara, T.; Yoshitomi, H.; Ito, H. Objective assessment of abnormal gait in patients with rheumatoid arthritis using a smartphone. Rheumatol. Int. 2012, 32, 3869–3874. [Google Scholar] [CrossRef]
  26. Abbate, S.; Avvenuti, M.; Bonatesta, F.; Cola, G.; Corsini, P.; Vecchio, A. A smartphone-based fall detection system. Pervasive Mob. Comput. 2012, 8, 883–899. [Google Scholar] [CrossRef]
  27. Dai, J.; Bai, X.; Yang, Z.; Shen, Z.; Xuan, D. Mobile phone-based pervasive fall detection. Pers. Ubiquitous Comput. 2010, 14, 633–643. [Google Scholar] [CrossRef]
  28. Mehner, S.; Klauck, R.; Koenig, H. Location-independent fall detection with smartphone. In Proceedings of the 6th International Conference on PErvasive Technologies Related to Assistive Environments, Rhodes, Greece, 29–31 May 2013. [Google Scholar]
  29. Mulcahy, M.K.; Kurkovsky, S. Automatic fall detection using mobile devices. In Proceedings of the 2015 12th International Conference on Information Technology—New Generations, Las Vegas, NV, USA, 13–15 April 2015; pp. 586–588. [Google Scholar]
  30. Bieber, G.; Koldrack, P.; Sablowski, C.; Peter, C.; Urban, B. Mobile physical activity recognition of stand-up and sit-down transitions for user behavior analysis. In Proceedings of the 3rd International Conference on PErvasive Technologies Related to Assistive Environments, Samos, Greece, 23–25 June 2010. [Google Scholar]
  31. Mellone, S.; Tacconi, C.; Chiari, L. Validity of a Smartphone-based instrumented Timed Up and Go. Gait Posture 2012, 36, 163–165. [Google Scholar] [CrossRef]
  32. Lee, B.C.; Kim, J.; Chen, S.; Sienko, K.H. Cell phone based balance trainer. J. Neuroeng. Rehabil. 2012, 9, 10. [Google Scholar] [CrossRef] [PubMed]
  33. Patterson, J.A.; Amick, R.Z.; Thummar, T.; Rogers, M.E. Validation of measures from the smartphone sway balance application: A pilot study. Int. J. Sports Phys. Ther. 2014, 9, 135–139. [Google Scholar] [PubMed]
  34. Aung, A.; Wai, P.; Duc, P.D.; Syin, C.; Haihong, Z. iBEST: Intelligent Balance Assessment and Stability Training System using Smartphone. In Proceedings of the 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Chicago, IL, USA, 26–30 August 2014; pp. 3683–3686. [Google Scholar]
  35. DeMasi, O.; Feygin, S.; Dembo, A.; Aguilera, A.; Recht, B. Well-Being Tracking via Smartphone-Measured Activity and Sleep: Cohort Study. JMIR mHealth uHealth 2017, 5, e137. [Google Scholar] [CrossRef] [PubMed]
  36. Lane, N.D.; Lin, M.; Mohammod, M.; Yang, X.; Lu, H.; Cardone, G.; Ali, S.; Doryab, A.; Berke, E.; Campbell, A.T.; et al. BeWell: Sensing Sleep, Physical Activities and Social Interactions to Promote Wellbeing. Mob. Netw. Appl. 2014, 19, 345–359. [Google Scholar] [CrossRef]
  37. Phithakkitnukoon, S.; Horanont, T.; di Lorenzo, G.; Shibasaki, R.; Ratti, C. Activity-aware map: Identifying human daily activity pattern using mobile phone data. Lect. Notes Comput. Sci. 2010, 6219, 14–25. [Google Scholar]
  38. Pratama, A.R.; Hidayat, R. Smartphone-based Pedestrian Dead Reckoning as an indoor positioning system. In Proceedings of the 2012 International Conference on System Engineering and Technology (ICSET), Bandung, Indonesia, 11–12 September 2012; pp. 1–6. [Google Scholar]
  39. Tian, Z.; Zhang, Y.; Zhou, M.; Liu, Y. Pedestrian dead reckoning for MARG navigation using a smartphone. EURASIP J. Adv. Signal Process. 2014, 2014, 65. [Google Scholar] [CrossRef]
  40. Kang, W.; Han, Y. SmartPDR: Smartphone-Based Pedestrian Dead Reckoning for Indoor Localization. IEEE Sens. J. 2015, 15, 2906–2916. [Google Scholar] [CrossRef]
  41. Oliver, N.; Matic, A.; Frias-Martinez, E. Mobile Network Data for Public Health: Opportunities and Challenges. Front. Public Health 2015, 3, 189. [Google Scholar] [CrossRef]
  42. Gravenhorst, F.; Muaremi, A.; Bardram, J.; Grünerbl, A.; Mayora, O.; Wurzer, G.; Frost, M.; Osmani, V.; Arnrich, B.; Lukowicz, P.; et al. Mobile phones as medical devices in mental disorder treatment: An overview. Pers. Ubiquitous Comput. 2015, 19, 335–353. [Google Scholar] [CrossRef]
  43. Burns, M.N.; Begale, M.; Duffecy, J.; Gergle, D.; Karr, C.J.; Giangrande, E.; Mohr, D.C. Harnessing Context Sensing to Develop a Mobile Intervention for Depression. J. Med. Internet. Res. 2011, 13, e55. [Google Scholar] [CrossRef]
  44. Schmidt, C.; Collette, F.; Cajochen, C.; Peigneux, P. A time to think: circadian rhythms in human cognition. Cogn. Neuropsychol. 2007, 24, 755–789. [Google Scholar] [CrossRef] [PubMed]
  45. van Dongen, H.P.; Dinges, D.F. Sleep, circadian rhythms, and psychomotor vigilance. Clin. Sports Med. 2005, 24, 237–49. [Google Scholar] [CrossRef] [PubMed]
  46. Dinges, D.F. An overview of sleepiness and accidents. J. Sleep Res. 1995, 4, 4–14. [Google Scholar] [CrossRef] [PubMed]
  47. Abdullah, S.; Murnane, E.L.; Matthews, M.; Kay, M.; Kientz, J.A.; Gay, G.; Choudhury, T. Cognitive rhythms: Unobtrusive and continuous sensing of alertness using a mobile phone. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, Heidelberg, Germany, 12–16 September 2016. [Google Scholar]
  48. Pijnenborg, G.H.; Withaar, F.K.; Brouwer, W.H.; Timmerman, M.E.; van den Bosch, R.J.; Evans, J.J. The efficacy of SMS text messages to compensate for the effects of cognitive impairments in schizophrenia. Br. J. Clin. Psychol. 2010, 49, 259–274. [Google Scholar] [CrossRef]
  49. Torous, J.; Kiang, M.V.; Lorme, J.; Onnela, J.P. New Tools for New Research in Psychiatry: A Scalable and Customizable Platform to Empower Data Driven Smartphone Research. JMIR Ment. Health 2016, 3, e16. [Google Scholar] [CrossRef]
  50. Wohlfahrt-Laymann, J.; Hermens, H.; Villalonga, C.; Vollenbroek-Hutten, M.; Banos, O. MobileCogniTracker. J. Ambient Intell. Hum. Comput. 2018, 1868–5145, 1–8. [Google Scholar] [CrossRef]
  51. Bardram, J.E.; Frost, M.; Szántó, K.; Faurholt-Jepsen, M.; Vinberg, M.; Kessing, L.V. Designing mobile health technology for bipolar disorder: A field trial of the monarca system. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Paris, France, 27 April–2 May 2013. [Google Scholar]
  52. Muaremi, A.; Gravenhorst, F.; Grünerbl, A.; Arnrich, B.; Tröster, G. Assessing Bipolar Episodes Using Speech Cues Derived from Phone Calls. In Pervasive Computing Paradigms for Mental Health; Cipresso, P., Matic, A., Lopez, G., Eds.; Springer: Cham, Switzerland, 2014; pp. 103–114. [Google Scholar]
  53. Grünerbl, A.; Muaremi, A.; Osmani, V.; Bahle, G.; Öhler, S.; Tröster, G.; Mayora, O.; Haring, C.; Lukowicz, P. Smartphone-based recognition of states and state changes in bipolar disorder patients. IEEE J. Biomed. Health Inform. 2015, 19, 140–148. [Google Scholar] [CrossRef]
  54. Vanello, N.; Guidi, A.; Gentili, C.; Werner, S.; Bertschy, G.; Valenza, G.; Lanata, A.; Scilingo, E.P. Speech analysis for mood state characterization in bipolar patients. Conf. Proc. IEEE Eng. Med. Biol. Soc. 2012, 2012, 2104–2107. [Google Scholar]
  55. Tausczik, Y.R.; Pennebaker, J.W. The Psychological Meaning of Words: LIWC and Computerized Text Analysis Methods. J. Lang. Soc. Psychol. 2010, 29, 24–54. [Google Scholar] [CrossRef]
  56. Rutland, J.B.; Sheets, T.; Young, T. Development of a scale to measure problem use of short message service: The SMS Problem Use Diagnostic Questionnaire. Cyberpsychol. Behav. 2007, 10, 841–843. [Google Scholar] [CrossRef]
  57. Lomranz, J.; Bergman, S.; Eyal, N.; Shmotkin, D. Indoor and outdoor activities of aged women and men as related to depression and well-being. Int. J. Aging Hum. Dev. 1988, 26, 303–314. [Google Scholar] [CrossRef] [PubMed]
  58. Osmani, V.; Maxhuni, A.; Grünerbl, A.; Lukowicz, P.; Haring, C.; Mayora, O. Monitoring activity of patients with bipolar disorder using smart phones. In Proceedings of the International Conference on Advances in Mobile Computing Multimedia, Vienna, Austria, 2–4 December 2013. [Google Scholar]
  59. Pielot, M.; Dingler, T.; Pedro, J.S.; Oliver, N. When attention is not scarce—Detecting boredom from mobile phone usage. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing, Osaka, Japan, 7–11 September 2015. [Google Scholar]
  60. Bogomolov, A.; Lepri, B.; Pianesi, F. Happiness Recognition from Mobile Phone Data. In Proceedings of the 2013 International Conference on Social Computing, Alexandria, VA, USA, 8–14 September 2013; pp. 790–795. [Google Scholar]
  61. Lachmar, E.M.; Wittenborn, A.K.; Bogen, K.W.; McCauley, H.L. MyDepressionLooksLike: Examining Public Discourse About Depression on Twitter. JMIR Ment. Health 2017, 4, e43. [Google Scholar] [CrossRef] [PubMed]
  62. Xu, K.; Qi, G.; Huang, J.; Wu, T.; Fu, X. Detecting bursts in sentiment-aware topics from social media. Knowl.-Based Syst. 2018, 141, 44–54. [Google Scholar] [CrossRef]
  63. Alp, Z.Z.; Öğüdücü, Ş.G. Identifying topical influencers on twitter based on user behavior and network topology. Knowl.-Based Syst. 2018, 141, 211–221. [Google Scholar]
  64. Frost, M.; Doryab, A.; Faurholt-Jepsen, M.; Kessing, L.V.; Bardram, J.E. Supporting disease insight through data analysis: Refinements of the monarca self-assessment system. In Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing, Zurich, Switzerland, 8–12 September 2013. [Google Scholar]
  65. George, L.K.; Blazer, D.G.; Hughes, D.C.; Fowler, N. Social support and the outcome of major depression. Br. J. Psychiatry 1989, 154, 478–485. [Google Scholar] [CrossRef] [PubMed]
  66. Moturu, S.T.; Khayal, I.; Aharony, N.; Pan, W.; Pentland, A. Using social sensing to understand the links between sleep, mood, and sociability. In Proceedings of the 2011 IEEE International Conference on Privacy, Security, Risk and Trust and IEEE International Conference on Social Computing, PASSAT/SocialCom 2011, Boston, MA, USA, 9–11 October 2011; pp. 208–214. [Google Scholar]
  67. Vieira, M.R.; Frias-Martinez, V.; Oliver, N.; Frias-Martinez, E. Characterizing Dense Urban Areas from Mobile Phone-Call Data: Discovery and Social Dynamics. In Proceedings of the 2010 IEEE Second International Conference on Social Computing, Minneapolis, MN, USA, 20–22 August 2010; pp. 241–248. [Google Scholar]
  68. Pastor-Escuredo, D.; Morales-Guzmán, A.; Torres-Fernández, Y.; Bauer, J.; Wadhwa, A.; Castro-Correa, C.; Romanoff, L.; Lee, J.G.; Rutherford, A.; Frias-Martinez, V.; et al. Flooding through the lens of mobile phone activity. In Proceedings of the 4th IEEE Global Humanitarian Technology Conference, GHTC 2014, San Jose, CA, USA, 10–13 October 2014; pp. 279–286. [Google Scholar]
Proceedings 02 01243 g001 550
Figure 1. An overview of the state-of-the-art literature on smartphone-based human behaviour analysis. Different types of sensor data are clustered based on the four sensing modalities, regarding the physical, cognitive, emotional and social behaviour. It is worth mentioning that phone activity stands for the interaction of a user with the device and varies per study. Overall, phone activity includes features related to battery level, ratio of incoming/outcoming calls, sent/received text messages, screen (un)locks and applications usage.
Figure 1. An overview of the state-of-the-art literature on smartphone-based human behaviour analysis. Different types of sensor data are clustered based on the four sensing modalities, regarding the physical, cognitive, emotional and social behaviour. It is worth mentioning that phone activity stands for the interaction of a user with the device and varies per study. Overall, phone activity includes features related to battery level, ratio of incoming/outcoming calls, sent/received text messages, screen (un)locks and applications usage.
Proceedings 02 01243 g001
Table
Table 1. Human behaviour analysis through smartphones.
Table 1. Human behaviour analysis through smartphones.
Smartphone SensorsFeaturesSensing ModalityBehaviours
motion
(e.g., accelerometer
and/or gyroscope)		time-domain
(e.g., mean, variance,
correlation)
frequency-domain
(e.g., FFT components)		physical activitycounting steps and/or
simple physical activities
(e.g., walking, sitting, standing)
and/or complicated activities
(e.g., shopping, eating, working)
and/or gait analysis and/or
fall detection and/or
sleep duration and/or
body balance and/or
hand movements
outdoor location
(e.g., GPS)		amount of time spent outdoors,
distances travelled,
frequency of visited places,
regularity of daily habits, etc.		physical &
social activity		location detection and
movement analysis and/or
social interaction/isolation
indoor location
(e.g., Bluetooth,
and/or Wi-Fi signals)		the amount and signal
strength of visible Wi-Fi
or Bluetooth stations
device usage patternsspeed of reaction timecognitive activityalertness, attendance,
fatigue assessment
SMS, phone calls,
audio and microphone		content,
number of
ingoing/outgoings calls
and texts, etc.		emotional &
social activity		emotional states and
mood and/or
social interaction/isolation
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Konsolakis, K.; Hermens, H.; Villalonga, C.; Vollenbroek-Hutten, M.; Banos, O. Human Behaviour Analysis through Smartphones. Proceedings 2018, 2, 1243. https://doi.org/10.3390/proceedings2191243

AMA Style

Konsolakis K, Hermens H, Villalonga C, Vollenbroek-Hutten M, Banos O. Human Behaviour Analysis through Smartphones. Proceedings. 2018; 2(19):1243. https://doi.org/10.3390/proceedings2191243

Chicago/Turabian Style

Konsolakis, Kostas, Hermie Hermens, Claudia Villalonga, Miriam Vollenbroek-Hutten, and Oresti Banos. 2018. "Human Behaviour Analysis through Smartphones" Proceedings 2, no. 19: 1243. https://doi.org/10.3390/proceedings2191243

Article Metrics

Back to TopTop