A Mini-Survey and Feasibility Study of Deep-Learning-Based Human Activity Recognition from Slight Feature Signals Obtained Using Privacy-Aware Environmental Sensors

Abstract

Numerous methods and applications have been proposed in human activity recognition (HAR). This paper presents a mini-survey of recent HAR studies and our originally developed benchmark datasets of two types using environmental sensors. For the first dataset, we specifically examine human pose estimation and slight motion recognition related to activities of daily living (ADL). Our proposed method employs OpenPose. It describes feature vectors without effects of objects or scene features, but with a convolutional neural network (CNN) with the VGG-16 backbone, which recognizes behavior patterns after classifying the obtained images into learning and verification subsets. The first dataset comprises time-series panoramic images obtained using a fisheye lens monocular camera with a wide field of view. We attempted to recognize five behavior patterns: eating, reading, operating a smartphone, operating a laptop computer, and sitting. Even when using panoramic images including distortions, results demonstrate the capability of recognizing properties and characteristics of slight motions and pose-based behavioral patterns. The second dataset was obtained using five environmental sensors: a thermopile sensor, a CO${}_2$ sensor, and air pressure, humidity, and temperature sensors. Our proposed sensor system obviates the need for constraint; it also preserves each subject’s privacy. Using a long short-term memory (LSTM) network combined with CNN, which is a deep-learning model dealing with time-series features, we recognized eight behavior patterns: eating, operating a laptop computer, operating a smartphone, playing a game, reading, exiting, taking a nap, and sitting. The recognition accuracy for the second dataset was lower than for the first dataset consisting of images, but we demonstrated recognition of behavior patterns from time-series of weak sensor signals. The recognition results for the first dataset, after accuracy evaluation, can be reused for automatically annotated labels applied to the second dataset. Our proposed method actualizes semi-automatic annotation, false recognized category detection, and sensor calibration. Feasibility study results show the new possibility of HAR used for ADL based on unique sensors of two types.

1. Introduction

1.1. Background and Scope

Human activity recognition (HAR) is a challenging task for pattern recognition and computer vision studies, especially when using off-the-shelf camera technology [1]. Numerous applications and their derivative variations exist in the areas of multimodal gesture recognition [2], consumption and consumer behavior analysis [3], human–robot interaction [4], robotics therapy [5], and body motion analysis in sports [6]. For industrial applications to a security service [7] and biometric authentication [8] in a public environment, vision-based systems have been proposed for identifying suspicious people based on characteristics of facial features, facial expressions [9], and body motions [10]. Moreover, development of a vision-based system [11] and an invisible sensor system [12] for bed-leaving detection and fall prevention at hospitals and nursing care sites has been reported. Such systems emphasize abnormality detection from the identification of behavior patterns.

In human living environments, real-time recognition of abnormal behavior patterns related to falls, injuries, and collisions is set as a high priority for reasons of safety and security [13]. Physical body motions in our daily life are classified roughly into three categories: standing, sitting, and lying down [14]. Standing motions are used during cooking, cleaning, household chores of various types, etc. Sitting motions are used during meals, when reading, playing games, watching television (TV), listening to music, and operating a smartphone, tablet, or laptop computer, etc. Classification and recognition of daily life behavior patterns based on image analyses is regarded as a challenging task for computer vision studies because the range of motion of represented features is limited and often inclusive of vague or meticulous movements, especially when examining sitting motions. Moreover, numerous systems use wearable devices [15] with embedded acceleration and gyro sensors without using a camera. They emphasize privacy in private environments such as living rooms.

1.2. Sensors and Human Behavior Patterns

For users, privacy concerns are relieved if image processing is completed inside a particularly closed device [16]. As one commercially available example, an up-to-date room air conditioner (Shirokuma-Kun series; Hitachi Ltd.; Tokyo, Japan) has a built-in wide field of view (FoV) monocular camera combined with a thermopile sensor [17]. Regarding this technical acceptance and social tendency, we consider that it will be acceptable for consumers to provide a comfortable environment using intelligent sensors. By contrast, wearable devices with physical restraints sometimes impose a mental burden on a wearer subject [18]. Moreover, the total cost for preparing, installing, and maintaining sensors installed in various places as ambient intelligence [19] is high, especially for healthcare applications [20]. For this study, we positively explore the use of cameras in daily life under the restriction that information is enclosed inside devices and systems considering mental burdens and sensor complexity. Intelligent sensor systems based on deep learning (DL) [21] provide such autonomous mechanisms and platforms.

The ultimate goal of this study is to infer an emotional state from recognized human behavior patterns in a living environment with unusual, non-routine, and extraordinary experiences. No specific definition exists for the respective patterns of usual daily living environments and unusual living environments. We assumed that unusual living environments include all locations except for those of our daily life, such as accommodations at a travel destination. Herein, university laboratories are mixed, including usual daily living environments for laboratory members and unusual living environments for non-members. Therefore, both environments are used freely depending on the observer background, situation, and context.

1.3. Contributions

As a preliminary study of emotion estimation applied in an unusual living environment, we developed original benchmark datasets of two types obtained in a laboratory, which is a mixed environment of usual and unusual living. For the first dataset, we used a fisheye lens camera to obtain time-series panoramic images. We specifically examine human pose estimation related to HAR as an approach that has no effect on objects because room sizes, vacant spaces, and arrangement of the furniture are generally varied and diverse in a laboratory. A particular characteristic of our method is that we employ a state-of-the-art pose estimation network to extract feature vectors from time-series panoramic images obtained using a fisheye lens camera. We also use a convolutional neural network (CNN) [22] to recognize five behavior patterns after classifying them as learning and verification subsets. To pursue measures to address privacy concerns, we used environmental sensors to obtain the second dataset. Experimentally obtained results demonstrate the possibility of recognizing behavior patterns of eight types from time-series tiny sensor signals using CNN combined with a short-term memory (LSTM) [23] network. Although the recognition accuracy for the second dataset was lower than that obtained for the first dataset, this paper presents an exploration of a new possibility of using HAR for activities of daily living (ADL) based on unique sensor systems and two original datasets. Accurately recognized results can be reused for automatic annotation of labels in another benchmark dataset for inferring emotion from behavior patterns. Here, the contributions of this study are as follows.

• A mini-survey presents existing HAR survey papers published between January 2016 and October 2021 and representative HAR studies reported between January 2019 and June 2021.
• Recent trends and developments of ADL studies are introduced, especially for the use of privacy-aware sensors.
• A dataset obtained using a fisheye lens monocular camera is provided with the aim reducing the burden of annotations.
• A dataset obtained using five privacy-aware environmental sensors is provided for long-term human monitoring.

The limitation of this study is that the respective dataset involves only one subject. However, we consider that this paper presents important characteristics and attributes of behavioral pattern recognition using DL-based methods.

1.4. Outline

This paper is structured as described below. In Section 2, we present a comprehensive survey HAR and ADL studies. For HAR, we review survey papers published between January 2016 and October 2021 and review state-of-the-art papers and proceedings published between January 2019 and June 2021. For ADL, we review details of eight representative studies. Based on investigation and analysis of these existing studies, we specifically examine tiny signal features of human motions related to HAR and ADL. Section 3 presents our first dataset, obtained using a fisheye lens camera. More than by a normal lens camera, humans are projected in small sizes. We applied two DL-based methods to five behavior patterns related to HAR. Subsequently, Section 4 presents our second dataset, obtained using air and thermometric sensors. We conducted evaluation experiments in a smaller room. We also used time-series feature changes to add three behavior patterns. Although the recognition accuracy was low compared with the first dataset, we demonstrated that these environmental sensors are useful for HAR and ADL tasks while also providing privacy and reliability. Finally, Section 5 concludes this report and highlights avenues for future work based on this feasibility study.

2. Mini Survey

We used Google Scholar to search for studies. All searches included the terms “human activity recognition,” or “HAR” or “activities of daily living” or “ADL”. First, we mainly selected open access papers. Then, we excluded articles that were less than four pages or articles with few citations published before 2021.

2.1. Existing HAR Surveys

For more than a quarter-century, HAR has been studied using various and numerous approaches [24]. Through that period, HAR studies have been boosted mainly by the rapid progress of computer and sensor technologies and their dissemination to our society. Numerous HAR studies have been conducted. Currently, HAR presents a challenging task because it extends to various technological domains that include, but which are not limited to, sensor selection, sensor calibration, recognition algorithm development, parameter optimization, real-time processing, improved accuracy, privacy consideration, and robustness for individual differences. Ultimately, HAR students examine human understanding from a behavioral perspective [25].

Numerous technical and academic papers describing HAR studies have been published each year. In addition, numerous survey papers have been presented since the advent of the first HAR studies [26,27,28].

Table 1. HAR survey papers between January 2016 and October 2021.

Pub.	Authors	Sensing Modality				Approach		Num.
Year	(A–Z)	Camera	SM	WS	ES	CML	DL	Ref.
2021	Biswal et al. [36]	√		√		√		48
2021	Bouchabou et al. [37]	√			√		√	113
2021	Chen et al. [38]		√	√	√		√	191
2021	Mihoub et al. [39]				√		√	64
2021	Muralidharan et al. [40]	√	√			√	√	115
2021	Shaikh et al. [41]	√					√	150
2021	Straczkiewicz et al. [42]		√	√		√	√	140
2020	Beddiar et al. [43]	√				√	√	237
2020	Carvalho et al. [44]		√	√	√			136
2020	Dang et al. [45]	√	√	√	√	√	√	236
2020	Demrozi et al. [46]		√	√		√	√	219
2020	Fu et al. [47]	√		√				183
2020	Hussain et al. [48]	√		√	√			165
2020	Jung [49]	√					√	60
2020	Sherafat [50]	√		√	√	√	√	132
Year	(A–Z)	Camera	SM	WS	ES	CML	DL	Ref.
2019	Dang et al. [51]		√	√				187
2019	Dhiman et al. [52]	√				√	√	208
2019	Elbasiony et al. [53]		√			√		48
2019	Hussain et al. [54]				√			141
2019	Jobanputra et al. [55]					√		9
2019	Li et al. [56]				√		√	106
2019	Lima et al. [57]		√	√			√	149
2019	Slim et al. [58]		√	√		√	√	119
2019	Wang et al. [59]			√	√		√	77
2018	Nweke et al. [60]			√			√	275
2018	Ramamurthy et al. [61]		√	√		√	√	83
2018	Shickel et al. [62]			√		√	√	63
2018	Wang et al. [63]	√					√	182
2017	Cornacchia et al. [64]		√	√		√		225
2017	Chen et al. [65]	√		√				78
2017	Morales et al. [66]		√	√				65
2017	Rault et al. [67]		√	√				88
2017	Vyas et al. [68]		√					39
2016	Dawn et al. [69]	√				√		66
2016	Onofri et al. [70]	√						90

presents a summary of HAR survey papers published between January 2016 and October 2021.

We classified sensing modalities into four categories: cameras, smartphones (SM), wearable sensors (WS), and environmental sensors (ES). Although cameras are included in ES, we specifically examine them as an independent category for the following three reasons. The first reason is that they are related to an actively researched pattern recognition problem in the computer vision field [29]. The second reason is that large amounts of high-dimensional pixel data can be obtained from cameras. In addition to still images, video sequences were used as spatiotemporal features for improved recognition accuracy [30]. Moreover, red, green, and blue–depth (RGB-D) cameras were widely used [31] to obtain depth data in each pixel. The third reason is the need for preservation and awareness of privacy. As a simple and convenient approach in the pursuit of privacy, image resolutions are reduced with degradation of spatial and temporal image quantity and quality. However, this measure will also degrade the resultant recognition accuracy [32].

We roughly divide recognition methods into two types: conventional machine learning (CML)-based methods and DL-based [21] methods. With respect to recognition accuracy, DL has appreciable performance and superiority over CML in numerous and varied applications [33]. The comparative application potential for real-time processing under a limited computing resource was the most important benefit of using CML. By virtue of the progress of edge computing technologies [34], cost-effective and advanced performance systems are available without using cloud computing, expensive graphics processing units (GPU), or a special analog-computing device [35].

2.2. State-of-the-Art HAR Studies

Although several survey papers [43,45,46,52,60,64] have cited more than 200 related research papers, these represent just a fraction of the numerous HAR studies that have been conducted to date. The numerous papers show the limited terms of years and fields of applications, including overlapping scopes among the survey papers. Moreover, several survey papers have reported earlier reports that present a comprehensive literature review. For this research field, numerous research and survey papers have been published over a short period. That trend is expected to continue. For our mini-survey,

Table 2. Representative HAR studies reported between January 2019 and June 2021.

Year	Authors	Methods	Datasets
2021	Fu et al. [71]	IPL-JPDA	original
2021	Gorji et al. [72]	CML (RF)	original
2021	Gul et al. [73]	YOLO+CNN	original
2021	Hussain et al. [74]	CML (RF)	[123]
2021	Mekruksavanich et al. [75]	4-CNN-LSTM	[124]
2021	Mekruksavanich et al. [76]	CNN-LSTM	[124,125]
2021	Moreira et al. [77]	ConvLSTM	original
2021	Nafea et al. [78]	CNN-BiLSTM	[124,126]
2021	Xiao et al. [79]	DIM-BLS	[124,126]
2020	Ahmed et al. [80]	CML (SVM)	[127]
2020	Ashry et al. [81]	BLSTM	original
2020	Debache et al. [82]	CML and CNN	original
2020	Ehatisham-Ul-Haq et al. [83]	CML	[128]
2020	Ferrari et al. [84]	Adaboost+CNN	[129,130,131]
2020	Hamad et al. [85]	CNN-LSTM	[132,133,134]
2020	Ihianle et al. [86]	MCBLSTM	[135,136]
2020	Irvine et al. [87]	EL	[123]
2020	Khannouz et al. [88]	CML	[135,137]
2020	Lawal et al. [89]	CNN	[138]
2020	Machot et al. [90]	ZSL	[139]
2020	Mukherjee et al. [91]	CNN (ResNet-101)	[140]+original
2020	Mutegeki et al. [92]	CNN-LSTM	[124]
2020	Pham et al. [93]	CNN-LSTM	[141,142,143]
2020	Popescu et al. [94]	CNN	[141,142,143]
2020	Qin et al. [95]	CNN (Fusion-ResNet)	[135,144]
2020	Shrestha et al. [96]	BLSTM	original
2020	Taylor et al. [97]	CML	original
2020	Tanberk et al. [98]	3D-CNN-LSTM	original
2020	Wan et al. [99]	CNN	[124]
2020	Wang et al. [100]	H-LSTM	[124]
2020	Xia et al. [101]	LSTM-CNN	[126,145,146]
2020	Xu et al. [102]	CNN (Fusion-Mdk-ResNet)	[124,126,147]
2019	Chung et al. [103]	LSTM	original
2019	Concone et al. [104]	HMM	original
2019	Ding et al. [105]	CNN+TL	[125,148]
2019	Ding et al. [106]	RNN (LSTM)	original
2019	Gumaei et al. [107]	RNN	[149]
2019	Ehatisham-Ul-Haq et al. [108]	CML (kNN, SVM)	[142]
2019	Gani et al. [109]	GMM	original
2019	Li et al. [110]	BLSTM	original
2019	Kim et al. [111]	HMM	original
2019	Naveed et al. [112]	CML (SVM)	[150,151]
2019	Qi et al. [113]	CNN	original
2019	Siirtola et al. [114]	EL	original
2019	Tian et al. [115]	CML (SVM+ELM)	original
2019	Voicu et al. [116]	CML (SVM)	original
2019	Xu et al. [117]	EL	[124]
2019	Xu et al. [118]	EL	[147]
2019	Yang et al. [119]	CNN (Inception)	[124,147,152]
2019	Zebin et al. [120]	CNN (LeNet)	original
2019	Zhang et al. [121]	U-Net	[126,145,146]+original
2019	Zhu et al. [122]	EL	original

presents representative HAR studies reported between January 2019 to June 2021. We specifically emphasized a review of recognition methods in the third column and benchmark datasets in the fourth column.

The most commonly used method is CNN-LSTM (e.g., 9 out of 51 studies): a model based on CNN with various backbones combined partially with LSTM [23] layers, including ConvLSTM [153]. Bidirectional LSTM (BLSTM) [154] was used solely or combined with CNN as CNN-BLSTM. Furthermore, CML-based and DL-based ensemble learning (EL), which comprise bagging, boosting, and stacking, are used commonly as HAR methods. For CML-based methods, support vector machines (SVM) [155], random forests (RF) [156], and k nearest neighbors (kNN) were used. Various CNN backbones have been used: LeNet [157], Inception [158], VGG-Net [159], Xception [160], ResNet [161], and ResNet’s derivative models in parentheses if they are indicated explicitly in the references. For other methods, various models and algorithms have been used: improved pseudo-labels joint probability domain adaptation (IPL-JPDA), you look only once (YOLO) [162], multi-channel bidirectional LSTM (MCBLSTM), extreme learning machine (ELM) [163], zero-shot learning (ZSL) [164], and U-Net [165].

Open datasets for HAR studies and competitions are well maintained for benchmarking [123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,166,167]. Advantages of using open benchmark datasets are not only that it saves time and costs for the development of sensor systems and data collection; it also facilitates performance comparison with other state-of-the-art methods. By contrast, because accuracy improvements have become saturated, the performance competition is becoming increasingly severe year by year. We regard original dataset developments for various scenarios, environments, and target persons as a fundamental and important research element for the wide varieties and scopes of HAR practical applications. Especially in recent years, the performance, duration, and size reduction in sensors are improving constantly. In addition to using commercially available sensors, the use of originally developed handmade sensors [12] can be expected to yield new applications according to applicable environments and restrictions. Although original benchmark datasets require a huge workload for data collection and their labeling for annotation, the development of an automatic data collection platform using an internet of things (IoT) [168] sensor devices provides a solution for workload reduction. Extension of studies and their applications can be anticipated in the furtherance of healthcare management [169], privacy consideration [170], and multimodal sensing [171] as advanced original system development [172].

2.3. Recent Trends and Developments of ADL Studies

Various methods for estimating dominant emotions [173] have been reported from various motions, actions, and behavior patterns. Nguyen et al. [174] widely surveyed video-based human behavior recognition studies related to ADL [175]. The results have been useful in supporting ambient assisted living systems for elderly people living independently. Generally, ADL patterns have included combing hair, applying makeup, brushing teeth, dental flossing, washing hands or face, drying hands or face, doing laundry, washing dishes, moving dishes, making tea, making coffee, drinking water or from a bottle, drinking water or from a tap, making a cold food snack, vacumming a floor, watching TV, using a computer, and using a cellular telephone [176]. To support ADL, they analyzed technical difficulties, such as occlusions and limited FoV, of conventional camera systems used in public environments. Moreover, they described that object recognition remained as a task for future study because intra-class variation occurred in unconstrained ADL scenarios. Therefore, they concluded that current system performance is far from adequate.

To enhance privacy and data capacity, accelerometers and gyro sensors can be used instead of cameras. Chelli et al. [177] developed an ML-based framework that is useful for fall detection and ADL recognition. For their study, ADL was set as comprising six categories: standing, sitting, walking, walking upstairs, walking downstairs, and lying down. They compared four popular CML algorithms: a multilayer perceptron (MLP), kNN, quadratic SVM, and ensemble bagged trees (EBT). The experimentally obtained results using a dataset produced for use with a smartphone revealed that the accuracy of fall detection reached 100% for both quadratic SVM and EBT algorithms.

Based on behavioral biometrics, Weiss et al. [136] proposed a recognition method for 18 physical activities including walking, jogging, using stairs, sitting, standing, kicking a soccer ball, dribbling a basketball, catching a tennis ball, typing, writing, clapping, brushing teeth, folding clothes, eating pasta, eating soup, eating a sandwich, eating chips, and drinking from a cup. The experimentally obtained results obtained with their original benchmark dataset using accelerometers and gyroscope sensors on a smartphone and a smartwatch revealed that zero-effort continuous biometrics based on normal activities and certain easy-to-perform activities of daily living are feasible and viable for gait-based biometrics. Nevertheless, their method involved wide-body motions, especially for hands and legs. Therefore, as a shortcoming, accuracy was markedly low for small and restricted motions done in a sitting posture such as eating, reading, and studying.

One important application of human motion recognition technology is for health-smart homes, which have various heterogeneous sensors. Diraco et al. [178] proposed a DL-based method for detecting early changes in human behaviors. The method specifically examines starting, duration, disappearing, swapping, locating, and heart rate changes. For actualizing long-term continuous monitoring, they used multiple sensors such as stereo cameras, monocular cameras, time of flight (TOF) cameras, microphones, pyroelectric infrared (PIR), sonar, light detection and ranging (LiDAR), ultra-wideband LiDAR, strain gauges, barometers, and vibration sensors. The experimentally obtained results demonstrated superiority in terms of accuracy and calculation time for detection and prediction. Unsupervised DL-based deep clustering (DC) methods were found to be superior to traditional supervised and semi-supervised methods using SVM including one-class SVM, CNN, stacked auto-encoders (SAE), and k-means clustering. Nevertheless, the burdens related to annotation for long-term continuous monitoring datasets remain difficult for practical application and dissemination.

To associate emotions and behavior patterns recognized from images, benchmark datasets were provided: interactive emotional dyadic motion capture (IEMO-CAP) datasets [179], the multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos (MOSI) [180], one-minute-gradual (OME) emotion datasets, emotion recognition in the wild named EmotiW [181], and the affect-net database [182]. For these datasets, the associated emotions comprise disgust, fear, happiness, surprise, sadness, and anger defined by Ekman [183]. Combined with these six basic emotion categories, Nojavanasghari et al. [184] added nine complex emotion categories: curiosity, uncertainty, excitement, attentiveness, exploration, confusion, anxiety, embarrassment, and frustration. Moreover, Weixuan et al. [185] added 11 complex emotional categories: amusement, contempt, contentment, embarrassment, excitement, guilt, pleasure, pride, relief, satisfaction, and shame.

As a smart classroom application at an elementary school, Kim et al. [186] proposed a system that includes affective sensing, DL-based emotion recognition, and real-time mobile-cloud computing networks. Their system yielded real-time suggestions with an in-class presenter to improve the quality and memorability of their allowed presentation. Moreover, their system yielded real-time adjustments and corrections to their non-verbal behavior, such as hand gestures, facial expressions, and body language. Their comprehensive study found computational requirements for their proposed system, which incorporates these technologies. Based on these requirements, they assess current difficulties and suggest some future directions in engineering and education disciplines for system deployment. They merely designed a conceptual system and its framework without specific classification or recognition of motion or behavior patterns.

Using a depth camera for three-dimensional (3D) human pose reconstruction, Marinoiu et al. [187] introduced a challenging task to recognize emotions from fine-grained actions. They developed non-staged videos recorded during robot-assisted therapy sessions for children with autism, along with non-standard camera viewpoints. Based on this development, they presented challenges: a large dataset with long videos, numerous highly variable actions, children who are only partially visible, and children of different ages who might show unpredictable actions. They investigated 3D human pose reconstruction methods for newly introduced tasks. Furthermore, they proposed extensions to adapt them to address these challenges. Moreover, for establishing several baselines and implications in the context of a child–robot interaction, they analyzed multiple approaches to action and emotion recognition from 3D human pose features.

For extraction of emotion-specific features from a vast number of human body motion descriptors, Ahmed et al. [188] introduced a double layer feature selection framework to classify emotions from a comprehensive list that included body motion features of 10 types. They used that feature selection framework to achieve accurate recognition of five basic emotions of happiness, sadness, fear, anger, and neutral, combined with three scenarios of walking, sitting, and action-independent scenarios. They validated their method using open datasets [189] obtained using a depth camera from 30 subjects. The experimentally obtained results revealed that their proposed emotion recognition system achieved excellent emotion recognition. In fact, it outperformed five widely used CML methods: SVM, linear discriminant analysis (LDA), naive Bayes (NB) classifier, kNN, and decision tree (DT). Nevertheless, its recognition accuracy was strongly dependent on the scenarios which were set in advance.

3. Vision-Based Approach Using a Fisheye Lens Monocular Camera

3.1. Proposed Sensor System and Recognition Method

Figure 1 depicts procedures used for our first proposed method. The procedures are divided roughly into three steps. Time-series panoramic images are obtained using a fisheye lens camera in the first step. For this measurement, we fixed the camera at a position x m distant from a subject. The obtained images were downsampled to n fps with a linear transformation to reduce the computation time and memory size.

The second step uses OpenPose [190] to extract feature vectors of pose motions. Herein, human two-dimensional (2D) pose estimation represents a challenging task for computer vision studies. Numerous methods [191,192,193,194,195,196,197,198,199] have been proposed to detect 2D poses from multiple people in a single image. Based on an earlier study reported by Noori et al. [200], we used OpenPose [190] because of its simple and easy implementation, high-speed processing, and high estimation accuracy. Regarding the accuracy verification of OpenPose, Kim et al. [201] employed 17 inertial sensors to evaluate 12 motion tasks: upright standing, trunk flexion 30${}^{\circ }$, sitting on a stool trunk flexed and rotated, placing an object on top, kneeling above the head, arms crossed, holding a box, legs crossed, sitting at the desk, simple lifting, and complex lifting.

OpenPose [190] employs a bottom-up representation of association scores via part affinity fields (PAFs) [202]. A set of 2D vector fields is encoded for the location and orientation of limbs over an image domain. To detect body parts, confidence maps are predicted from two-branch CNN. As an open library, OpenPose was trained using two benchmark datasets for multi-person pose estimation: The Max Planck Institut Informatik (MPII) human pose dataset [203] and the common objects in context (COCO) key-points challenge dataset [204]. For our implementation, we used the OpenPose library, which was pre-trained using the latter dataset.

As examined for behavior pattern classification and recognition used for labeling, we used the VGG-16 [159] backbone, which comprises 13 convolutional layers and three fully connected layers. The original VGG-16 was designed as a deep architecture to use $3\times 3$ convolution filters for each pixel. Using feature vectors produced from OpenPose, we applied transfer learning [205] to the upper layers of VGG-16, which were trained in advance using ImageNet [206]. Although ResNet [161] and its derivative models such as WideResNet [207], ResNeXt [208], Res2Net [209], and Inception-resnet (Inception-v4) [210] have already been proposed, we used the VGG-Net backbone because of our overall considerations of GPU time, speed, and memory consumption [211]. Actually, CNN backbones can be switched easily according to specifications and applications.

Table 3. VGG-16 parameters and their setting values.

Parameters	Setting Values
Learning iteration	50 epochs
Batch size	4
Validation split	0.2
Num. input layer	224 × 224 × 3 units
Num. output layer	2, 3, or 5 units
Optimization algorithms	Adam [213]

presents the major parameters of VGG-16. Based on our earlier study [212], the number of generations, batch size, and validation split were set, respectively, to 50 epochs, 4, and 0.2. The number of the input layer corresponds to the resized image size: 224 pixels × 224 pixels × 3 channels. The numbers of the output layer were set to two, three, and five, according to the number of target categories. For the optimization algorithm, we used Adam [213], which is an optimizer combined with Momentum [214] and RMSProp [215].

3.2. Setup and the Obtained First Dataset

Table 4. Major specifications of the fisheye lens camera (RealSense T265; Intel Corp.).

Parameters	Specifications
Imaging sensor	OV9282
Lens size	14 inch
Pixel size	3 × 3 $\mathsf{\mu }$m${}^2$
FoV	163 ± 5 deg
Resolution	W1280 × H800 pixels
Frame rate	120 fps
IMU	BMI055
Body dimensions	L12.5 × W108.0 × H24.5 mm
Weight	55 g
Mean power consumption	1.5 W

presents some major specifications of the fisheye lens camera (RealSense T265; Intel Corp.; Santa Clara, CA, USA) used for this study. The FoV of this camera is $63\pm 5$ deg, which is approximately twice that of an ordinary monocular camera. The camera body is 12.5 mm long × 108 mm wide × 24.5 mm high. This compact camera can reduce the psychological burdens of a subject that might result from awareness of a camera during an experiment. Nevertheless, because of this wide FoV, the subject becomes extremely small if the camera is in a distant position. In addition, image distortion is extensive at the image edges. For this property, we set $x=2.0$ in specific consideration of this camera.

We obtained images for the dataset in our university laboratory. The 100 m${}^2$ room, which has 17 students and 2 researchers, is an open space with no blocking structure except for 1.2 m high partitions. In a meeting corner of this room, we set up an environment for obtaining images.

The ultimate goal of this line of research is to infer a person’s emotional state based on their displayed behavior patterns. Human emotions have individual and often idiosyncratic differences that present implications about the person’s affect, well-being, and social relations [216]. For this study, we aim at developing an individual estimation model that is suitable and specialized for each subject without using a common and general estimation model.

The target behavior patterns represent five categories: eating (ET), reading (RD), operating a smartphone (SP), operating a laptop computer (LC), and sitting (ST). Herein, ST stands for a state of doing nothing while sitting. Moreover, ST might represent simple relaxation, taking a break, taking a nap, thinking, or dreaming. Recognition of human activities that involve great amounts of motion during standing has already been addressed in a report described by Noori et al. [200]. Therefore, for this study, we specifically examined slight motions that are made during sitting. After we explained the ethics and purpose of this experiment, the subject performed these five actions in order. The interval of each behavior was set as approximately 20 s.

Table 5. Details of our original benchmark dataset.

Behavior Pattern	Subset Name	Capture Time (s)	Total Images (Frames)	Valid Images (Frames)	Conversion Ratio
ET	ET1	21	218	218	1.00
	ET2	20	209	209	1.00
RD	RD1	20	200	200	1.00
	RD2	20	200	200	1.00
SP	SP1	20	202	202	1.00
	SP2	20	202	202	1.00
LC	LC1	20	202	202	1.00
	LC2	20	204	172	0.84
ST	ST1	18	181	138	0.76
	ST2	16	160	100	0.63
Total		199	1978	1843	0.93

presents details of the obtained images as our original dataset, which contains 10 subsets. Each subset was obtained twice for each behavior pattern. The behavior periods varied slightly because of the lack of a timer setting. Although all images included the subject, several unconvertible images of feature vectors occurred in LC and ST. All feature vectors were extracted from three categories: ET, RD, and SP. The conversion ratio for all datasets was 0.92 for this dataset.

Table 6. Major specifications of the computer used for this experiment.

Items	Specifications
OS	Windows 10 Professional 64 bit; Microsoft Corp.
CPU	Core i5-6200 (2.30 GHz); Intel Corp.
Memory	8192 MB
GPU	HD Graphics 520; Intel Corp.
VRAM	128 MB

presents major performance specifications of our computer, which was used to execute OpenPose [190] and VGG-16 [159]. For this study, we set $n=10$ with emphasis on our computer environment.

Table 7. Details of processing used to extract feature vectors using OpenPose.

Behavior Pattern	Subset Name	Processing Time (s)	Total Images (Frame)	Second per Frame (s)
ET	ET1	20,949	218	96.10
	ET2	21,659	209	103.63
RD	RD1	22,992	200	114.96
	RD2	12,056	200	60.28
SP	SP1	11,711	202	57.98
	SP2	11,757	202	58.20
LC	LC1	11,982	202	59.32
	LC2	11,750	172	68.31
ST	ST1	18,409	138	133.40
	ST2	13,599	100	135.99
Total		156,864	1843	85.11

presents details of the computational time necessary to calculate feature vectors using OpenPose. The total computational time for 2214 images was 182,241 s. The average computational time per frame was 85.11 s. The computational times differed depending on the respective action patterns. The shortest and longest were, respectively, 57.98 s in SP1 and 135.99 s in ST2.

Figure 2 depicts sample images of ET, RD, SP, LC, and ST. In each image, the left, center, and right panels, respectively, depict the original, pose detection, and feature vector sub-images. Except for ST, occlusion occurred because of a desk. Although the subject’s legs were not included, the skeleton position of the upper buttocks was estimated. Although the laptop display occluded the arms, the second joints were estimated. Images show that differences between the ET and SP postures are slight. The ET and SP images depict that differences between the postures are slight.

3.3. Experiment Results and Discussion

Using our original datasets combined with five behavior patterns, we conducted three experiments to evaluate the recognition accuracy and loss. Herein, A and L, respectively, denote accuracy and loss. Moreover, we set $A_{tra}$, $A_{val}$, $L_{tra}$, and $L_{val}$ as, respectively, representing accuracy for training datasets, accuracy for validation datasets, loss for training dataset, and loss for validation datasets.

Table 8. Combinations of behavior patterns for the respective experiments.

Experiment	ET	RD	SP	LC	ST
A	√	√
B	√	√	√
C	√	√	√	√	√

presents the combinations of behavior patterns for experiments among the designated Experiments A, B, and C. As the minimum combination, Experiment A assesses two behavior patterns: ET and RD. Experiment B represents a combination of Experiment A and SP. Experiment C targets all five behavior patterns. The first-round subsets, which comprise ET1, RD1, SP1, LC1, and ST1, are used for training. The second-round subsets, which comprise ET2, RD2, SP2, LC2, and ST2, are used for training. The number of training iterations was set as 10 epochs for all experiments.

The evaluation dataset used for Experiment A comprises ET and RD of two behavior patterns. Figure 3 presents transitions of $A_{tra}$, $A_{val}$, $L_{tra}$, and $L_{val}$ in 10 epochs. Overall, the data show that $A_{tra}$ increased steadily. The value of $A_{val}$ soared during the fourth epoch. Although a momentary drop occurred in the fifth epoch, $A_{val}$ increased steadily. Eventually, both $A_{tra}$ and $A_{val}$ reached 1.000 at the seventh epoch. By contrast, $L_{tra}$ and $L_{val}$ decreased steadily.

Evaluation datasets on Experiment B comprise ET, RD, and SP. This combination is equivalent to the addition of SP to Experiment A. Although the subject grasped different objects in RD and SP, the feature vectors obtained using OpenPose [190] were very similar. For that reason, we specifically examined the recognition and verification of RD and SP.

Figure 4 presents transitions of $A_{tra}$, $A_{val}$, $L_{tra}$, and $L_{val}$ in 10 epochs. Both $A_{tra}$ and $A_{val}$ increased steadily. The values of $L_{tra}$ and $L_{val}$ decreased, respectively, from the sixth epoch and the fourth epoch. After increasing $L_{val}$ in the fifth epoch, it decreased from the sixth epoch.

We conducted Experiment C using all images in the five categories. Figure 5 presents transitions of $A_{tra}$, $A_{val}$, $L_{tra}$, and $L_{val}$ in 10 epochs. The value of $A_{tra}$ increased steadily. The value of $A_{val}$ increased steadily, except for a slight drop in the fourth epoch. Both $L_{tra}$ and $L_{val}$ decreased steadily from the seventh epoch.

Table 9. Confusion matrix from Experiment C (frames).

Category	ET	RD	SP	LC	ST
ET	209	0	0	0	0
RD	1	199	0	0	0
SP	0	0	202	0	0
LC	0	0	1	171	0
ST	0	0	0	0	100

presents the confusion matrix obtained from Experiment C. One image of RD was falsely recognized as ET. One image of LC was falsely recognized as SP. All of the other 881 images were recognized correctly.

Table 10. Results of accuracy and loss obtained from the respective experiments (%).

Experiment	${\mathit{A}}_{\mathit{tra}}$	${\mathit{A}}_{\mathit{val}}$	${\mathit{L}}_{\mathit{tra}}$	${\mathit{L}}_{\mathit{val}}$
A	100.0	100.0	34.1	35.1
B	100.0	100.0	69.7	78.6
C	99.3	99.7	55.0	53.9

presents a summary of all results obtained for $A_{tra}$, $A_{val}$, $L_{tra}$, and $L_{val}$ from the respective experiments. Regarding A for all experiments, both $A_{tra}$ and $A_{val}$ were saturated around the seventh epoch. The L curve shows that this parameter tends to decrease if the number of iterations is increased by more than 10 epochs. Nevertheless, because of the decreased number of L, overfitting might occur if the number of iterations increases for training with sufficient accuracy of A. For this study, we obtained subsets of the dataset twice in a short time for one subject. Obtaining a new dataset with different dates and times is important to evaluate the overlearning and generalization that occur with our method.

4. Environmental Approach Using Air and Thermometric Sensors

4.1. Proposed Sensor System and Recognition Method

For the second approach, we aimed at developing a privacy-aware sensor system. Figure 6a depicts the overall configuration of our second proposed system. We selected five environmental sensors that measure air and temperature distributions. A single-board computer (SBC), Raspberry Pi 4 Model B, was used to save measurement signals to a flash memory device. This SBC is a commonly and widely used prototyping device available for scientific research and industrial application development [217]. Measurement signals obtained from two independent sensor modules are sent to an IoT cloud service called Ambient (AmbientData Inc.; Tokyo, Japan) for storage and visualization. The data-receiving interval of this free service is 1 min.

Figure 6b depicts details of the sensor module structure. Each sensor module comprises an SBC, which is used for sensor control, data storage, and wireless communication with an Ambient server, along with three sensor boards: a thermopile sensor board (D6T-44L-06; Omron Corp.; Kyoto, Japan), a CO${}_2$ sensor board (K30; Senseair AB; Delsbo, Sweden), and an air pressure, humidity, and temperature combined sensor board (BME280; Robert Bosch GmbH; Gerlingen, Germany). The SBC and these three sensor boards are wired by serial cables with serial communication protocols managed using a universal asynchronous receiver transmitter (UART) and an inter-integrated circuit (I2C). Figure 6c depicts images of data visualization results on browsers of a website provided by Ambient. We checked the sensor operating status periodically through the website during examination for data collection.

Table 11. Major specifications of D6T-44L-06.

Item	Specifications
Number of elements	16 ($4\times 4$)
Horizontal view angle	44.2${}^{\circ }$
Vertical view angle	45.7${}^{\circ }$
Object temperature detection range	5–50${}^{\circ }$
Ambient temperature detection range	5–45${}^{\circ }$
Object temperature output accuracy	$\pm 1.5^{\circ }$ (maximum)
Temperature resolution	0.06 ${}^{\circ }$C
Operating temperature	0–50 ${}^{\circ }$C
Operating humidity	20–95%
Sensor board dimensions	L11.6 × W12.0 × H10.7 mm

presents some major specifications of a thermopile sensor, D6T, which measures the temperature distribution of objects, especially for humans [218], and their surroundings. As a contactless sensor, the D6T sensor measures surface temperatures using thermocouple elements that receive radiant heat energy from objects and surroundings.

Table 12. Major specifications of K30.

Item	Specifications
Target gas	CO${}_2$
Operating principle	NDIR
Measurement range	0–5000 ppm
Accuracy	$\pm 30$ ppm $\pm 3$% of reading at 101.3 kPa
Response time	20 s diffusion time
Rate of measurement	0.5 Hz
Operating temperature	0–50 ${}^{\circ }$C
Operating humidity	0–95%
Sensor board dimensions	L51 × W58 × H12 mm

presents some major specifications of a CO${}_2$ sensor: K30. The operating principle of this sensor is a non-dispersive infrared (NDIR) technology based on microelectromechanical systems (MEMS) for a small, inexpensive, and lightweight gas sensor. We calibrated this sensor in our earlier study aimed at in-situ atmospheric measurements [219].

Table 13. Major specifications of BME280.

Item	Target	Specifications
Measurement range	Air pressure	300–1100 hPa
	Temperature	−40–85 ${}^{\circ }$C
	Humidity	0–100%
Resolution	Air pressure	0.18 hPa
	Temperature	0.01 ${}^{\circ }$C
	Humidity	0.008%
Accuracy tolerance	Air pressure	$\pm 1$ hPa
	Temperature	$\pm 1$%
	Humidity	$\pm 3$%
Sampling intervals		1 s
Sensor dimensions		L2.5 × W2.5 × H0.93 mm

presents some major specifications of an integrated sensor; BME280, which comprises an air pressure sensor, a humidity sensor, and a temperature sensor as a single sensor package. The measurement ranges for air pressure, temperature, humidity are, respectively, 300–1100 hPa, −40–85 ${}^{\circ }$C, and 0–100% with 0.18 hPa, 0.01 ${}^{\circ }$C, and 0.008% resolutions.

Based on our earlier study [219] and existing studies as shown in Table 2, we used LSTM embedded in VGG-16. Figure 7 portrays a typical LSTM network architecture of hidden layers. The internal LSTM structure comprises hidden layer units with memory cells called LSTM blocks, along with gates of three types: input gates, forgetting gates, and output gates. The vanishing gradient problem [220] is resolved using this mechanism. Input gates select enabled or disabled input feature signals. Forgetting gates select permission to reset the internal information stored in cells. Output gates determine the amount of information that is transmitted at the next phase. Based on recurrent neural network (RNN) algorithms, LSTM provides a one-step later prediction. The prediction is conducted from the input feature signals at the current time t and the feedback signals to the hidden layer at a previous time $t-1$. The memory cells save internal information for a long period, which provides an important benefit compared to RNN for modeling temporally distant dependences.

Letting $x_t$ and $c_t$, respectively, represent the input feature signal and output from memory cells, and letting $I_t$, $F_t$, and $O_t$, respectively, stand for the outputs of the input, forgetting, and output gates, then LSTM output $H_t$ is obtained as presented below.

$$H_t=o_t\otimes tanh\left(c_t\right),$$

(1)

$$c_t=c_{t-1}\otimes f_t+i_t\otimes tanh(W_z{x}_t+R_z{h}_{t-1}+b_z),$$

(2)

$$\left[\begin{array}{c}{I}_t\\ F_t\\ O_t\end{array}\right]={\sigma }_1\left[\begin{array}{c}{W}_i\\ W_f\\ W_o\end{array}\right]x_t+{\sigma }_2\left[\begin{array}{c}{R}_i\\ R_f\\ R_o\end{array}\right]h_{t-1}+{\sigma }_3\left[\begin{array}{c}{b}_i\\ b_f\\ b_o\end{array}\right].$$

(3)

where $W_{i,f,o,z}$, $R_{i,f,o,z}$, and $B_{i,f,o,z}$, respectively, stand for input weights, recurrent weights, and biases. Moreover, $\sigma ={\sigma }_1={\sigma }_2={\sigma }_3$ and ⊗, respectively, express the sigmoid function and the element-wise product.

The dominant role of $I_t$ is to update cell states. Propagation signals are controlled by $F_t$, which refers to a previous cell state $c_{t-1}$. Moreover, unnecessary signals are removed by $F_t$, which prevents excessive information from the earlier cell output split by short-term and long-term memories. The output gates control update values from hidden units. Similarly to the input gates, the output gates have a mechanism that is useful in avoiding inappropriate weight updates for redundant and undesired signals. In addition, the current cell state $c_t$ and $H_t$ are also used to calculate the subsequent input data at $t+1$. As a remarkable characteristic, LSTM has a dynamic adjustment mechanism that provides previous and forward signals while maintaining $c_t$ in addition to $H_t$.

Table 14. LSTM parameters and their setting values.

Parameters	Setting Values
Learning iteration	50 epochs
Batch size	2
Validation split	0.2
Num. input layer	20 or 40 units
Num. hidden layer	50 units
Num. output layer	8 units
Optimization algorithms	Adam [213]
Look-back	10

presents the major parameters of LSTM. Based on our earlier study [219], the numbers of generations, batch size, and validation split were set, respectively, to 100 epochs, 2, and 0.2. Each unit of the input layer was assigned to each sensor channel. The number of the input layer comprises two types: 20 units for the single sensor system and 40 units for the dual sensor system. The numbers of the hidden layer and the output layer were set, respectively, to 50 units and 8 units. Similar to the previous experiment, we used Adam [213] for the optimization algorithm between the LSTM layer and the output layer. Finally, we evaluated the look-back parameter, which represents the number of previous time steps to be considered as input, by changing it to three steps: 5, 10, 20, and 30. The optimal value for this parameter was 10.

4.2. Setup and the Obtained Second Dataset

We obtained another benchmark dataset using our originally developed sensor system, as depicted in Figure 6. Figure 8a depicts the room layout used in this experiment. The room is 8 m long and 3 m wide, with an area of approximately 24 m${}^2$. This room is used mainly for a meeting as our satellite laboratory. The two desks have different orientations. We installed the respective sensor modules on the walls at the centers of the longitudinal and latitudinal positions. The installation height was approximately 2.5 m from the floor. Figure 8b depicts an image after sensor installation on the wall.

We obtained time-series images using a hemispheric camera (PIXPRO SP360; Eastman Kodak Co., Rochester, NY, USA) for annotation. We used our proposed and evaluated method in Section 3 as a semi-automatic annotation method for labeling behavior patterns for this dataset. The behavior patterns comprise eight categories: eating (ET), operating a laptop computer (LC), operating a smartphone (SP), playing a game (GM), reading (RD), exiting the room (EX), taking a nap (NP), and sitting (ST). Of these categories, ET, LC, SP, RD, and ST are similar to the first dataset. We added GM, EX, and NP as new categories in the second dataset. We annotated them manually.

Table 15. Details of datasets for respective behavior patterns.

	ET	LC	SP	GM	RD	EX	NP	ST	Total
No. Signals	655	3142	1139	3951	1444	408	414	368	11,521
Ratio (%)	5.69	27.27	9.89	34.29	12.53	3.54	3.59	3.19	100.00

presents the number of signals in each behavior pattern. The total data amount is 11,521 signals. The quantities of data in each behavior are uneven because the subject spent with no restrictions in the room during the data recording. The number of GM data is the largest, accounting for one-third. The numbers of EX, NP, and ST data are small.

4.3. Experiment Results and Discussion

Time-series split cross-validation (TSSCV) was used to evaluate the recognition accuracy of behavior patterns. The number of divisions k is the most important parameter for TSSCV. We set $k=8$ as a result based on our preliminary experiment. Figure 9 depicts the distributions of recognition accuracies in each validation result. The distributions demonstrate that the difference in recognition accuracy is greater in each validation data subset.

Table 16. Mean recognition accuracies and standard deviations in the respective positions and sensors (%). Underlining shows the highest accuracy in each sensor combination.

Sensor Installed Position	All Sensors	D6T	BME280	K30
P1+P2	60.8 (12.0)	50.6 (17.7)	44.6 (17.7)	27.0 (12.7)
P1	49.8 (16.8)	41.6 (17.7)	30.8 (16.3)	18.2 (9.7)
P2	58.9 (13.1)	49.0 (16.2)	46.5 (15.6)	46.1 (18.5)

presents the mean recognition accuracies calculated from the result portrayed in Figure 9. The mean recognition accuracy and its standard deviation from all sensors of both modules are 60.8% and 12.0%, respectively. Reflecting the performance of the sensor module separately, the third and fourth rows in the table, respectively, show recognition accuracies for P1 or P2. Although the accuracy for P1 dropped 18.1%, the accuracy drop for P2 was 3.1%. This accuracy difference demonstrates that the effect of improved recognition accuracy of P2 was higher than that of P1. For the rectangular room depicted in Figure 8a, the sensor installation on the longitudinal side was more effective than that on the latitudinal side.

Subsequently, the third, fourth, and fifth columns of Table 16, respectively, present the recognition accuracies for the sole use of either D6T, BME280, or K30. Compared with the three sensors, D6T and K30, respectively, produced 50.6% as the highest accuracy and 18.2% as the lowest accuracy. This revealed tendency demonstrates that delay time gaps of measured feature changes affected feature changes of the measurement object. Moreover, these results demonstrate that feature dimensional differences of the respective sensors, such as 16-dimensional signals from D6T, 3D signals from BME280, and one-dimensional (1D) signals from K30, affected the recognition accuracy.

Figure 10 depicts recognition accuracies in each behavior pattern and sensor module. Among the eight behavior patterns, ET exhibited the highest accuracy of more than 90%. Subsequently, LC and GM demonstrated relatively high accuracies: 77.6%, 54.0%, 57.6%, 69.0%, 61.3%, and 77.9%. However, the accuracy difference can be large depending on the installation positions of the sensor modules. The accuracies of RD and EX are low compared with those of ET, LC, and GM. For ST, the accuracies of P1+P2, P1, and P2 are, respectively, 2.8%, 0.0%, and 0.0%. The respective accuracies of SP and NP are 0.0%. We analyzed these unbalanced results using confusion matrices.

Table 17. Confusion matrices for the results shown in Figure 10. Underlined values are the maximum numbers of signals in the respective recognition categories.

P1+P2	EX	LC	SP	GM	RD	EX	NP	ST
EX	408	23	4	0	0	10	0	0
LC	0	2437	0	62	552	73	0	17
SP	8	51	0	11	23	0	22	0
GM	0	439	1	2725	486	28	270	2
RD	536	0	468	0	627	0	0	113
EX	0	217	0	0	0	197	0	0
NP	0	0	0	0	408	0	0	0
ST	42	214	0	28	13	16	0	9
P1	EX	LC	SP	GM	RD	EX	NP	ST
EX	409	23	3	0	0	10	0	0
LC	0	1696	0	114	1324	7	0	0
SP	8	62	0	13	32	0	0	0
GM	0	889	0	2420	642	0	0	0
RD	463	0	471	0	481	0	0	29
EX	0	323	0	0	0	91	0	0
NP	0	0	0	0	408	0	0	0
ST	42	222	9	28	13	8	0	0
P2	EX	LC	SP	GM	RD	EX	NP	ST
EX	411	23	1	0	0	10	0	0
LC	42	1809	849	81	349	7	0	4
SP	8	51	0	25	31	0	0	0
GM	0	390	0	3078	483	0	0	0
RD	458	0	351	0	635	0	0	0
EX	0	318	0	0	0	96	0	0
NP	0	0	0	0	408	0	0	0
ST	42	80	9	28	155	8	0	0

present confusion matrices. These matrices demonstrated that EX was hindered by some slight confusion with LC in all combinations of the sensor module installation patterns. By contrast, numerous LC signals were confused with RD signals. Particularly, marked confusion occurred between LC and SP for P2 compared with those for P1+P2 and P1. For SP, confusion occurred in all categories, except for EX and ST. For GM, confusion among LC, RD, and NP occurred frequently. Confusion with the other categories occurred less. Although confusion among ET, SP, and ST occurred, the other categories showed less confusion. Numerous instances of confusion occurred between EX and LC. This tendency differs from the confusion property for LC, which caused a few instances of confusion against EX.

All the NP signals were recognized to RD. We consider that both characteristics were less similar because no confusion occurred from RD to NP. Experiments must be conducted to compare these results with those obtained using BLSTM [154] because temporal characteristics are assumed to have some influence. For ST, confusion with categories, except NP, occurred widely. Particularly, numerous ST signals were confused with LD and RD. The recognition output to the LC was large, approximately one-third of the total. Subsequently, GM and RD include numerous signals. SP, EX, NP, and ST are few: less than 5%. As presented in Table 15, GM accounts for the largest number of signals. This result demonstrated that the amount of data does not affect the recognition result distribution. We infer that the low accuracy of SP, EX, NP, and ST is attributable to the small amount of data. However, correlation between the number of data and the obtained accuracy is not a dominant factor because ET represents an exception to the overall tendencies found in this study.

5. Conclusions

As a feasibility study based on findings of an HAR mini-survey, this paper presented two original benchmark datasets used to recognize daily life behavior patterns from the assessment of tiny sensor signals generated from slight motions. The first dataset comprises feature vectors from time-series panoramic images obtained using a fisheye lens camera. We used OpenPose and CNN with the VGG-16 backbone to recognize behavior patterns of five types after first dividing them into learning and verification subsets. Our originally obtained first dataset comprises five categories: eating, reading, operating a smartphone, operating a personal computer, and sitting. Recognition evaluation experiments were conducted by changing the combination of subsets. The obtained results demonstrated characteristics of behavioral pattern recognition obtained from wide-field images taken using a fisheye lens camera, even for panoramic images that included distortions.

The second dataset was obtained using five environmental sensors. We were concerned not only with non-constraint but also with privacy awareness for this environmental sensor system. Using LSTM combined with CNN, we attempted to recognize eight behavior patterns. Comparing these datasets, the recognition accuracy of the second dataset was 38.9 percentage points lower than that of the first dataset. Nevertheless, we demonstrated the possibility of recognizing behavior patterns from time-series of weak sensor signals. We consider that the recognition results obtained from the first dataset after the accuracy evaluation can be reused for automatic annotation of labels to the second benchmark dataset.

As a subject of future work, we must develop an integrated system that is able to use the two datasets for automatic annotation and autonomous learning without using manually created labels. The limitation of this study is that respective dataset involve only one subject. We intend to develop a dataset accumulated from multiple subjects. Furthermore, intend to expand the categories of target behavior patterns, including data types and volumes. Moreover, our future studies are expected to clarify relations between emotions and behavior patterns through field tests conducted in unusual living environments such as accommodation facilities.