Human Posture Identification Using a MIMO Array

Abstract

The elderly are constantly in danger of falling and injuring themselves without anyone realizing it. A safety-monitoring system based on microwaves can ease these concerns. The authors have proposed safety-monitoring systems that use multiple-input multiple-output (MIMO) radar to localize persons by capturing their biological activities such as respiration. However, our studies to date have focused on localization, which is easier to achieve than an estimation of human postures. This paper proposes a human posture identification scheme based on height and a Doppler radar cross section (RCS) as estimated by a MIMO array. This scheme allows smart home applications to dispense with contact and wearable devices. Experiments demonstrate that this method can identify the supine position (i.e., after a fall) with 100% accuracy, and the average identification rate is 95.0%.

1. Introduction

Recent studies have targeted smart home systems for safety-monitoring and energy saving. However, the aging of society raises new social concerns such as lonely deaths and traffic accidents by the elderly. The increase in the number of elderly people demands safety-monitoring systems that can obviate these concerns by detecting the posture of the elderly. Existing solutions for indoor use, such as networked video cameras [1] and wearable sensors [2,3], employ IoT (Internet of things) devices. However, the former is an invasion of privacy, particularly in spaces such as the bathroom and restroom. Some people also have an aversion to being continuously watched. Furthermore, such a system provides only line-of-sight (LOS) coverage. The latter allows the state of the subject to be discerned. However, it forces the elderly to wear a device and thus places excessive physical and mental burdens on the user. Such a system is also unsuitable for the elderly because observation is not possible when the person forgets to wear the device.

To avoid these problems, living-body sensing systems [4,5,6] have been studied. The use of microwaves yields several key advances including privacy protection, contactless observation, and non-line-of-sight (NLOS) coverage. Examples of microwave-based monitoring techniques include direction of arrival (DOA) and direction of departure (DOD) estimation based on multiple-input multiple-output (MIMO) radar systems [7,8]. Though these methods can localize targets, they suffer from weak precision because the desired signal is buried by undesired waves due to the multi-path environment. To solve this problem, human localization methods suitable for multi-path environments have been proposed. There are three approaches to human localization: time difference of arrival (TDOA) estimation [9,10], object localization [11] based on the multiple signal classification (MUSIC) method [12], and the trigonometry methods based on DOA/DOD estimation using the MUSIC method [13,14,15]. Though the TDOA methods can quickly localize targets even in multi-path environments by using frequency-modulated continuous-wave (FMCW) radar, this method is expensive as it requires a wide bandwidth of 1.79 GHz (from 5.46 to 7.25 GHz). Additionally, they need static channels, i.e., the environment must be measured without human interference in advance. Moreover, the measurements must be repeated if the environment is changed, e.g., a piece of furniture is shifted. Localization based on MUSIC [11] uses a low-frequency band, 250 MHz, and estimates the target location by using spherical-mode MUSIC to process the oscillating return signal. However, the array aperture is comparable to the estimated distance because of the low frequency, and this method requires observation periods of over 10 s. Trigonometry-based localization [13,14,15] uses MIMO radar with DOA estimation by the fast Fourier transform (FFT) technique [16]. However, this method needs to observe the channel for several tens of seconds to accurately capture human activity information.

The authors have proposed a fast living-body localization algorithm where the time-differential channel is used to attain rapid DOA/DOD estimation in multi-path environments [17]. This algorithm identifies the differences among observation times that correspond to cyclic human body activities such as respiration and heartbeat. The living-body locations are estimated by applying the two-dimensional MUSIC method [18] to the time-differential channel. This algorithm has an observation period that corresponds to one cycle of biological activity. Its key feature is that this method does not require calibration to the environment in advance. However, the above techniques can estimate only target location. That is unfortunate as estimating human posture is very important in detecting safety-related events such as falls. Therefore, human posture detection is needed for comprehensive safety-monitoring systems.

In this paper, we propose a human posture identification scheme that uses a MIMO array. This method estimates a three-dimensionally target location by using the time-differential channel technique [17]; the Doppler radar cross section (RCS) is calculated from the power reflected from the target and the distance between the estimated location and the receiver/transmitter. The human posture is identified by applying the nearest neighbor algorithm [19] to the estimated height and the Doppler RCS information. The three-dimensional localization procedure and the Doppler RCS calculation are described below. Experiments are carried out in an actual indoor environment to demonstrate that the proposed method can accurately estimate the human location and identify human posture with over 90% accuracy.

2. Human Posture Identification Based on Height Doppler RCS of Subject Estimated by MIMO Array

The authors previously proposed a fast localization algorithm that estimates subject locations by using the time-variant channel in multi-path environments [17]. In this study, we apply the localization method to the observation channel, and the 3-dimensional location of the subject is estimated by MIMO radar. The Doppler RCS is calculated from the received power and the distance between transmitter/receiver and the estimated target location. Human posture is estimated from the height and the Doppler RCS of the subject as estimated by MIMO radar. The following text explains this method in detail.

2.1. Three Dimensional Localization of Human Subject

This study assumes a MIMO array consisting of an $M_r$ element array receiver and an $M_t$ element array transmitter. In a multi-path environment containing one person, the time-variant channel is generated by the fluctuation of the human body’s surface due to body motion, respiration, and heartbeat. We start by expressing the $M_r\times M_t$ time-variant MIMO channel as

$$\begin{array}{c}\hfill \mathit{H}(t)=\left(\begin{array}{ccc}{h}_{11}(t)& \dots & h_{1M_t}(t)\\ ⋮& \ddots & ⋮\\ h_{M_{r}1}(t)& \dots & h_{M_r{M}_t}(t)\end{array}\right),\end{array}$$

(1)

where $h_{ij}$ is the complex channel response from the j-th transmitter element to the i-th receiver element, and t represents the observation time. $M_r\times M_t$ MIMO radar can be considered to be an $M_r{M}_t\times 1$ virtual single-input multiple-output (SIMO) radar [7]. The $M_r{M}_t\times 1$ SIMO channel is expressed as

$$\begin{array}{c}\hfill \mathit{h}(t)={[h_{11}(t),\dots ,h_{M_{r}1}(t),\dots ,h_{M_r{M}_t}(t)]}^T\end{array}$$

(2)

where ${\{·\}}^T$ is transposition. Though DOA and DOD can be estimated using this virtual SIMO channel, unwanted path components disturb the estimation of the living-body location. Unwanted path components consist of the direct wave from transmitter to receiver and waves reflected from the walls, floor, and furniture, and these components are static. Therefore, we exclude the undesired components by applying the fast localization algorithm [17] to the converted SIMO channel; the time-differential channel is defined as

$$\begin{array}{c}\hfill {\mathit{h}}_{sb}(t,t_{sb})=\mathit{h}(t)-\mathit{h}(t+t_{sb})\end{array}$$

(3)

where $t_{sb}$ represents the time difference. The instantaneous correlation matrix, using the time-differential channel $h_{sb}(t,t_{sb})$ with observing time t and time difference $t_{sb}$, is defined as

$$\begin{array}{c}\hfill {\mathit{R}}_i(t,t_{sb})={\mathit{h}}_{sb}(t,t_{sb}){\mathit{h}}_{sb}{(t,t_{sb})}^H\end{array}$$

(4)

where ${\{·\}}^H$ means complex conjugate transposition. To recover eigenvalue rank, a correlation matrix is calculated by averaging over the time, where both t and $t_{sb}$ are swept. This is expressed as

$$\begin{array}{c}\hfill \mathit{R}=\overline{{\mathit{R}}_i(t,t_{sb})},(0\le t\le T_{\max }/2,T_{\min }/2\le t_{sb}\le T_{\max }/2)\end{array}$$

(5)

where $\overline{\{·\}}$ is the averaging operator, $T_{\max }$ and $T_{\min }$ are the maximum and minimum periods corresponding to the biological activities, respectively. By eigenvalue decomposition, the averaged correlation matrix $\mathit{R}$ is given by

$$\begin{array}{ccc}\hfill \mathit{R}& =& {\mathit{U}\mathbf{\Lambda }\mathit{U}}^H,\hfill \end{array}$$

(6)

$$\begin{array}{ccc}\hfill \mathit{U}& =& [{\mathit{u}}_1,\dots ,{\mathit{u}}_{M_r{M}_t}],\hfill \end{array}$$

(7)

$$\begin{array}{ccc}\hfill \mathbf{\Lambda }& =& \mathrm{diag}([{\lambda }_1,\dots ,{\lambda }_{M_r{M}_t}])\hfill \end{array}$$

(8)

where $\mathit{U}$ and $\mathbf{\Lambda }$ represent the eigenvector and the diagonal matrix representing eigenvalues, respectively. At this time, the eigenvalues, $\mathrm{\Lambda }$, are related as follows:

$$\begin{array}{c}\hfill {\lambda }_1>{\lambda }_2=\cdots ={\lambda }_{M_r{M}_t}={\sigma }_f^2\end{array}$$

(9)

where ${\sigma }_f^2$ represents the expected value of the energy of the channel fluctuation component due to noise. The eigenvector corresponding to noise, $[{\mathit{u}}_2,\cdots ,{\mathit{u}}_{M_r{M}_t}]$, is expressed as ${\mathit{U}}_N$. In this study, subject location is estimated via three-dimensional MUSIC with a spherical mode vector—the original MUSIC method [12] extended to cover the 3D domain. The three-dimensional spherical mode vector $\mathit{a}(x,y,z)$ is expressed as

$$\begin{array}{ccc}\hfill \mathit{a}(x,y,z)& =& {\mathit{a}}_t\otimes {\mathit{a}}_r,\hfill \end{array}$$

(10)

$$\begin{array}{ccc}\hfill {\mathit{a}}_t(x,y,z)& =& {[e^{-j2\pi {D_t}_1/\lambda },\dots ,e^{-j2\pi {D_t}_{M_t}/\lambda }]}^T\hfill \end{array}$$

(11)

$$\begin{array}{ccc}\hfill {\mathit{a}}_r(x,y,z)& =& {[e^{-j2\pi {D_r}_1/\lambda },\dots ,e^{-j2\pi {D_r}_{M_r}/\lambda }]}^T\hfill \end{array}$$

(12)

$$\begin{array}{ccc}\hfill D_{r_p}& =& \sqrt{{(x-x_{r_p})}^2+{(y-y_{r_p})}^2+{(z-z_{r_p})}^2}\hfill \end{array}$$

(13)

$$\begin{array}{ccc}\hfill D_{t_q}& =& \sqrt{{(x-x_{t_q})}^2+{(y-y_{t_q})}^2+{(z-z_{t_q})}^2}\hfill \end{array}$$

(14)

where ${\mathit{a}}_t(x,y,z)$ and ${\mathit{a}}_r(x,y,z)$ are the steering vectors at the transmitting and receiving side, respectively. ⊗ represents the Kronecker product, and $\lambda$ is wavelength. $D_{r_p}$ is the distance between position $(x,y,z)$ and the p-th receiver element, and $D_{t_q}$ is the distance between position $(x,y,z)$ and the q-th transmitter element, respectively. $(x_{r_p},y_{r_p},z_{r_p})$ and $(x_{t_q},y_{t_q},z_{t_q})$ are the positions of the p-th receiver element and the q-th transmitter element, respectively. Here, the evaluation function of the MUSIC method (MUSIC spectrum) is calculated as

$$\begin{array}{ccc}\hfill P(x,y,z)& =& \frac{{\mathit{a}}^H(x,y,z)\mathit{a}(x,y,z)}{{\mathit{a}}^H(x,y,z){\mathit{U}}_N{\mathit{U}}_N^H\mathit{a}(x,y,z)}.\hfill \end{array}$$

(15)

This MUSIC spectrum peak represents the estimated target location.

2.2. Doppler Radar Cross Section and Human Posture Identification

The first eigenvector, ${\mathit{u}}_1$ of Equation (7), corresponds to target location. The converted SIMO channel, $\mathit{h}(t)$, (2), and the first eigenvector, ${\mathit{u}}_1$, are multiplied to enhance the biological component of the target, and this signal $y(t)$ is expressed as

$$\begin{array}{c}\hfill y(t)={\mathit{u}}_1^H\mathit{h}(t).\end{array}$$

(16)

The observed signal, $y(t)$, is Fourier-transformed, and is defined as $F(\omega )$. The received power $P_r(\omega )$ is expressed as

$$\begin{array}{c}\hfill P_r(\omega )=\frac{{\left|F(\omega )\right|}^2}{M_r{M}_t}\end{array}$$

(17)

where $\omega$ represents frequency. Here, we define the Doppler radar cross section (RCS) by solving the radar range equation for $\sigma$; this is expressed as

$$\begin{array}{c}\hfill \sigma =\frac{{(4\pi )}^3{R}_r^2{R}_t^2}{G_r{G}_t{\lambda }^2{P}_t}{\int }_{f_1}^{f_2}{P}_r(\omega )d\omega \end{array}$$

(18)

where $R_r$ and $R_t$ represent the distances of the estimated target location from the centers of the receiver and transmitter, respectively. $P_t$ represents the transmitting power, and $G_r$ and $G_t$ are the gains of the receiving antenna and transmitting antenna, respectively. $f_1$ and $f_2$ define the frequency range that encompasses the vital sign effects.

First, we create training data of human posture. The dataset of the estimated height and the Doppler RCS with N trials are made. To exclude sample outliers, we use the data lying within 30–70% of the estimated height. The training data is made from every posture dataset. We evaluate the posture identification rate of the K-nearest neighbor (k-NN) algorithm [19]. In this study, k is set to 1, and the one nearest neighbor classifier is used.

3. Results

3.1. Experimental Condition and Measurement Setup

Table 1. Measurement conditions.

Antenna Element (Tx/Rx)	16-Element Patch Antenna
Distance between Tx and Rx, d	4 m
Height of the Tx/Rx, h	0.8 m
Frequency	2.47125 GHz
Tx power	−28 dBm
Snapshot rate	100 Hz
Database channel measurement time	10 s
Channel measurement time	2.56 s

and Figure 1 overview the measurement setup. The experiments used a $16\times 16$ MIMO configuration. As shown in Figure 1, a single-pole 64 throw (SP64T) switch was used at the transmitting side. Though the exact observation time is not the same for all elements in the MIMO channel matrix, the time differences among the elements are so short compared to the vital activity that they can be ignored. A continuous wave (CW) signal of 2.47125 GHz was used. The transmitting power at the antennas was set to −28 dBm. The CW signal was split to the receiver side since accurate synchronization between transmitting and receiving sides is needed. At the receiver side, received signals are input to a down-converter unit by way of a low-noise amplifier (LNA) unit. The down-converted baseband signals (${\mathrm{I}}_1$, ${\mathrm{Q}}_1$,∼${\mathrm{I}}_{16}$, and ${\mathrm{Q}}_{16}$ ) were digitized by a data-acquisition unit (DAQ) with a sampling frequency of 20 kHz. The snapshot rate of the MIMO channel is determined by the switching speed of the SP64T. In the experiments, the rate for taking a snapshot of the MIMO channel was set to 100 Hz.

Figure 2 shows a photo of the array antenna. The receiver and transmitter arrays have 16 patch antennas in a vertical 4 × 4 array. All array antennas used a PTFE substrate, and an antenna thickness, width, and height were 1.6, 60, and 240 mm, respectively. All antenna elements have vertical polarization. The element space of the arrays of receiver and transmitter was half wavelength. The array’s center was set to $h=0.8$ m, the trunk height of the subjects. The straight line distance between transmitting and receiving antennas was set to 4.0 m. The receiver and the transmitter faced the center of the room.

In this study, the observation time for localization and calculating Doppler RCS was set to 2.56 s. Thus, the number of snapshots was 256. The transmission components of S parameters were used as the propagation channel. The number of targets was determined following the MUSIC method. In this study, time difference $t_{sb}$ was set to 0.05 (s) $\le t_{\mathrm{sb}}\le$ 2.5 (s) for localization, while the range of frequency was set from 0.39 to 10.16 Hz in calculating the Doppler RCS; antenna gain $G_r$ and $G_t$ was 4.96 dB, and the averaged gain value was from $-{40}^{\circ }$ to ${40}^{\circ }$.

Figure 3 shows the experimental environment. The experiment was carried out in a room containing desks and shelves. The room had concrete walls and its width, depth, and height were 7.0, 6.0, and 2.7 m, respectively. One side of the room had four windows.

Figure 4 shows the subject posture when the channel was observed. When the channel was measured, the subject assumed positions in which he was standing (a), sitting on a chair (b), sitting on the floor (c), and lying on his back (d); the measurement location was set to $(X,Y)=(2.0,2.0)$ (m). The target faced the wall against which the antennas were set when the subject was standing and sitting. The subject lay down with his feet toward the wall against which the antennas were set, and the trunk of the subject was set at the measurement location. In all postures, the number of measurements was 500, and the number of trials of posture identification was 3000.

3.2. Results

3.2.1. Results of Measured Channel and Three Dimensional Localization

Figure 5 shows an example of the time-variant channel response, $h_{11}(t)$, of the observation channel, $\mathit{H}(t)$, for all postures. In this figure, “Static” indicates the channel response without a living-body. In comparison with the static channel, all time-variant channels with a living-body, Figure 5, exhibited change because the living-body’s activities altered the path. The variation demonstrates the periodicity of biological activities such as respiration. Additionally, the standing posture yielded the largest channel response. Moreover, standing created non-periodic components due to gross body motion. The other positions, on the other hand, yielded far cleaner patterns, as biological activity was the dominant factor.

Figure 6 shows an example of the MUSIC spectrum for living-body localization when the subject stood at $(X=2.0\mathrm{m}$, $Y=2.0\mathrm{m})$. Figure 6a,b are the XY plane and the ZX plane at the spectrum peak (i.e., estimated location), respectively. In these figures, the x-mark represents the spectrum peak, and the human silhouette is displayed on the MUSIC spectrum. In this case, the estimated location is $(X,Y,Z)=(2.01,1.83,0.88)$ (m). These figures confirm that the estimated location lies close to the human body, and the spectrum peak appears at the subject’s abdomen because the abdomen exhibits the largest surface fluctuation.

Figure 7a,b show examples of the MUSIC spectrum for living-body localization in the XY and ZX planes when the subject sat on a chair at $(X=2.0\mathrm{m}$, $Y=2.0\mathrm{m})$, respectively. In this result, the estimated location is $(X,Y,Z)=(1.98,1.76,0.74)$ (m). Figure 7a shows that the estimated location (x-mark) appears on the subject’s face. The result of Figure 7b shows that the spectrum peak tracks the subject’s abdomen, and the estimated height is lower than that of the standing position.

Figure 8 shows an example of the MUSIC spectrum for living-body localization when the subject sat on the floor at $(X=2.0\mathrm{m}$, $Y=2.0\mathrm{m})$. The estimated location is $(X,Y,Z)=(2.01,1.83,0.55)$ (m). Figure 8a shows that the spectrum peak appears at the subject’s location. Figure 8b shows that the estimated height was lower than that in the standing or sitting positions, and the peak appears close to the chest of the subject.

Figure 9 shows examples of the MUSIC spectrum for living-body localization in XY and ZX planes when the subject lay on his back at $(X=2.0\mathrm{m}$, $Y=2.0\mathrm{m})$, respectively. The spectrum peak appears at $(X,Y,Z)=(1.92,1.68,0.11)$ (m). In this case, the subject’s body looks large on the XY plane, the estimated location is again the abdomen of the subject. Additionally, the spectrum peak is the lowest among all states. Therefore, the results show that the estimated location depends on the subject’s posture, and the peak generally matches the abdomen of the target.

Figure 10 shows the cumulative distribution function (CDF) of the estimation error in the XY plane. The estimation error was defined by the Euclidean distance between the actual target location and the estimated location in the XY plane.

Table 2. All posture estimation error of the median, the 90% value, and the Root Mean Square Error (RMSE).

	Median Error (m)	90% Value (m)	RMSE (m)
Standing	0.177	0.271	0.201
Sitting on the chair	0.101	0.178	0.167
Sitting on the floor	0.124	0.160	0.202
Lying on his back	0.237	0.254	0.236

shows the results of Figure 10. The median, the 90% value, and the root mean square error (RMSE) of the estimation error are used as the evaluation metrics. For the standing position, the median, the 90% value, and the RMSE are 0.177 m, 0.271 m, and 0.201 m, respectively. The median, the 90% value, and the RMSE of the sitting-on-a-chair position are 0.101 m, 0.178 m, and 0.167 m, respectively. For the sitting-on-the-floor position, the median, the 90% value, and the RMSE are 0.124 m, 0.160 m, and 0.202 m, respectively. When the subject lay on his back, the median, the 90% value, and the RMSE of the estimation error were 0.237 m, 0.254 m, and 0.236 m, respectively. The subject has a width of about 0.4 m, and the center of the subject was always set on the vertical axis of the measurement location. Therefore, these results show that subject location can be accurately estimated for all postures.

3.2.2. Results of Human Posture Identification

Figure 11 shows the scatter diagram of the training data based on the estimated height and the Doppler RCS for all postures. These postures have non-overlapping distributions in this figure. In this figure, the estimated heights of the standing and the sitting-on-a-chair positions are nearly unchanged, but they do have large variance in the Doppler RCS. On the one hand, the sitting-on-the-floor position shows large variance in estimated height and in the Doppler RCS. The lying position yields the smallest distribution of all postures.

Figure 12 shows the confusion matrix of the identified posture classified by the proposed method. The number of the trials to identify each posture was 3000. In this figure, the true positive rate (TPR) of the standing, sitting-on-a-chair, sitting-on-the-floor, and lying positions were 92.4%, 96.5%, 91.2%, and 100%, respectively. The lying position yielded the highest TPR because its distribution is clearly different from the others, see Figure 11. Therefore, the proposed method can be used by safety-monitoring systems to detect that a person has fallen. Note that the sitting-on-the-floor position has the smallest TPR due to the large variation in height estimates. However, the TPR of the sitting-on-the-floor position, 91.2%, indicates high accuracy. The average TPR in our experiments was 95.0%. Therefore, it is confirmed that human posture can be accurately identified by the proposed method.

4. Conclusions

This paper proposes and demonstrates a human posture identification method that uses two microwave (2.47125 GHz) MIMO arrays. This method is a non-contact and non-wearable technique and thus well suits smart home applications. The MIMO antenna arrays measures the time-variant channel created by the subject, and the target location is estimated in three dimensions by using the time-differential channel technique and the Doppler RCS. The key is detecting the target’s vital signs from the power reflected from the target and the relationship between the estimated location and the receiver/transmitter. Human posture is identified by applying the nearest neighbor algorithm to the estimated height and the Doppler RCS information. Experiments were carried out in an indoor environment. The results showed that all postures had RMSE values of within 0.25 m. The results demonstrated that the proposed method identified the supine posture with 100% accuracy, and the average TPR was 95.0%. Therefore, we confirmed that the proposed method can identify human posture with very high accuracy.