Fall Detection Based on Continuous Wave Radar Sensor Using Binarized Neural Networks

Abstract

Accidents caused by falls among the elderly have become a significant social issue, making fall detection systems increasingly needed. Fall detection systems such as internet of things (IoT) devices must be affordable and compact because they must be installed in various locations around the house, such as bedrooms, living rooms, and bathrooms. In this study, we propose a lightweight fall detection method using a continuous-wave (CW) radar sensor and a binarized neural network (BNN) to meet these requirements. We used a CW radar sensor, which is more affordable than other types of radar sensors, and employed a BNN with binarized features and parameters to reduce memory usage and make the system lighter. The proposed method distinguishes movements using micro-Doppler signatures, and spectrogram is binarized as an input to the BNN. The proposed method achieved 93.1% accuracy in binary classification of five fall actions and six non-fall actions. The memory requirements for storing parameters were reduced to 11.9 KB, representing a reduction of up to 99.9% compared with previous studies.

1. Introduction

Owing to the recent decrease in birth rates and the increase in single and divorced households, the number of elderly people living alone is increasing. Elderly people living alone spend a lot of time by themselves, which can be very dangerous as they may be unable to receive immediate help if an accident occurs. Falls are serious accidents, especially among the elderly, and can lead to severe injury or even death. Therefore, systems that can quickly detect and respond to falls among elderly people in home environments are becoming increasingly important.

In recent years, active research has been conducted on fall detection systems using various sensors such as wearable sensors, camera sensors and radar sensors, and machine-learning algorithms. However, wearable sensors require regular battery charging, and are limited in detecting falls in situations where the sensor must be temporarily removed such as in wet environments, like a bathroom or swimming pool [1,2,3]. Camera sensors present a challenge in terms of detectability in dark environments, and significant concerns about privacy invasion arise [4,5,6]. Radar sensors are gaining increasing attention in fall-detection research because they are less affected by external environmental conditions. Radar sensors primarily used for fall detection include continuous-wave (CW) radar [7,8,9,10,11,12,13], frequency-modulated continuous-wave (FMCW) radar [14,15,16,17], and ultrawideband (UWB) radar [18,19,20,21,22]. Data acquired from radar sensors are used in machine-learning and deep-learning algorithms to detect falls.

In [7], six features—such as mean and variance—were extracted from the acquired data and input into a cubic support vector machine (SVM), achieving 100% accuracy in detecting falls. In [8], the frequency distribution trajectory (FDT) generated from movements was input into a hidden Markov model (HMM) to classify three fall and four non-fall actions with 95% accuracy. In [9], machine-learning algorithms such as SVM and random forest (RF) were connected in three stages, achieving 93.24% accuracy in detecting falls. In [10], a Kalman filter was used to extract features, and k-nearest neighbors (KNN) was employed to detect falls with 92.85% accuracy. Ref. [14] proposes the dynamic range-Doppler trajectory (DRDT) method. In this method, points with a high radar cross section (RCS) are extracted from the range-Doppler map of multiple frames. These points are then represented in a three-dimensional range-Doppler frame. Four types of features were extracted from the generated DRDT map, and KNN was used to detect falls. In [15], 29 features were extracted from the point cloud data acquired using two FMCW radars, and an RF classifier was used to detect falls with 92.2% accuracy. However, when using machine-learning algorithms instead of deep-learning algorithms, features must be manually extracted, and the accuracy varies depending on the features used.

In [11], a method was proposed to detect falls by inputting both thresholds and spectrograms generated from raw data, into a multistage system. This system consists of thresholding, a preliminary screening network (PSN), and a constraint reconstruction-based fall detector (CRFD). In [12], a short-time Fourier transform (STFT) was applied to the received signal generating a time-frequency spectrogram, that was converted to a grayscale image and input into a CNN to detect falls. In [16], a line kernel convolutional neural network (LKCNN) was proposed to process the received data directly without preprocessing. Additionally, a data sample generation method that utilizes multiple receiving channels and small pulse repetition times (PRT) was proposed to generate numerous samples. In [17], range-angle reflection heat maps were obtained using two fast Fourier transforms (FFTs) and then merged into one. After removing the spatial redundancy using the radar low dimension embedding (RLDE) algorithm, the data were input into an RNN for detection. In [18], the range bins corresponding to fast time in the raw data were summed to obtain time-series data for a slow time, which were then input into a deep convolutional neural network to detect falls. In [19], a method was proposed to detect falls by converting a spectrogram into a binary scale. A binary image was generated through k-means clustering, median filtering, and a morphological opening operation. The resulting image was input into a deep convolutional neural network, achieving 98.37% precision and 97.82% specificity in distinguishing between falls and non-falls. In [20], a time-frequency image was generated using the STFT operation and input into a CNN-based Capsule Network, achieving 94.22% accuracy, 95.66% precision, 93.99% sensitivity, and 94.55% specificity. In [21], FFT and singular value decomposition (SVD) filtering were used to extract frequency-domain and time-domain feature images. An adaptive channel selection algorithm was then proposed to distinguish activity from background. This output was then input into a deep convolutional neural network to classify three types of fall actions. In [13,22], a Convolutional Long Short-Term Memory Model was used to extract spatial and temporal features for fall detection. However, these deep-learning algorithms have large model sizes and many parameters that require a significant amount of memory. This can be a limitation in terms of size and cost of fall detection systems that need to be installed throughout the home in the form of IoT devices.

To solve these problems, we propose a lightweight fall detection method that utilizes a CW radar sensor and a BNN. Unlike other types of radar, CW radar does not compress pulses or modulate frequencies when generating transmission waves, making it cheaper. BNN has the same structure as CNN, but unlike CNN, where features and parameters are multi-bit floating-point, it uses +1 and −1. Since CNN uses multi-bit floating-point, it requires considerable memory to store features and parameters. On the other hand, BNN only uses +1 and −1, so it can express data as 1 bit, simplifying the computation process and reducing memory usage. In addition, when implemented as a very-large-scale integrated circuit (VLSI), matrix multiplication, which corresponds to the operations of the convolutional and fully-connected layers that compose the CNN-based architecture, can be parallelized to reduce the inference time. Therefore, both methods are suitable for fall detection systems in home environments that require lightweight solutions.

The remainder of this paper is organized as follows: Section 2 describes the experimental environment for measuring data and dataset. Section 3 explains the structure of the proposed method. Section 4 evaluates the performance of the proposed method and verifies its utility through a comparison with previous studies. Finally, Section 5 presents the conclusions of this study.

2. Experimental Setup and Measurement

Data measurements were performed with 24 GHz CW radar from Infineon’s Sense2GoL Development Kit [23]. The detailed specifications of the radar are listed in Table 1. The radar sensor and experimental environment are shown in Figure 1. The size of the experimental environment is 3 × 2.5 m². The radar was positioned 1.3 m above the ground. Further, because only micro-Doppler signatures were used, the distance between the radar and the participants was flexible. The distance was set between 1 and 4 m to account for the indoor environment while remaining within the radar’s maximum measurement range. Additionally, lighting conditions were adjusted to simulate both daytime and nighttime scenarios, and the indoor temperature was varied by opening or closing the window. Five male and female participants with varying heights, weights, and ages participated in the experiment. The Doppler effect, which measures the speed of radar, varies depending on the direction of motion of the target relative to the radar. However, as falls can occur in any direction, we configured the actions in various directions and measured data at different speeds, even within the same action. For example, when standing, a fall can occur in any direction. To account for this, the cases of 0° (forward), ±90° (left/right), and 180° (backward) were selected based on the radar’s line of sight. Conversely, when sitting on a chair or sofa, only the cases of 0° (forward) and ±90° (left/right) were considered, taking the backrest into account. Examples of the actions are shown in Figure 2. There were five fall actions, labeled (a)–(e), and six non-fall actions, labeled (f)–(k).

Table 1. Radar specifications.

Parameters	Value
Frequency	24.125 GHz
Minimum speed	0.5 km/h
Maximum speed	30 km/h
Maximum distance	15 m
Horizontal −3 dB beamwidth	80°
Elevation −3 dB beamwidth	29°

Figure 1. Radar sensor used and experimental environment: (a) radar sensor; (b) experimental environment.

Figure 2. Examples of actions: (a) standing and then falling forward; (b) standing and then falling to the left/right; (c) standing and then falling backward; (d) sitting and then falling forward; (e) sitting and then falling to the left/right; (f) walking slowly without moving arms; (g) walking quickly whiles winging arms; (h) squatting; (i) sitting on a chair; (j) standing up from sitting on a chair; (k) lying down and then lifting the upper body. Arrows indicate the direction of movement of the upper body or arm.

The results are summarized in Table 2. The total number of data points was 2192, with 1017 and 1175 representing fall and non-fall data, respectively. For (b), there were 306 data points, with 153 for falling to the left and the remaining 153 for falling to the right. Similarly, for (e), out of 270 data points, 135 fell to the left, and the remaining 135 fell to the right.

Table 2. Number of data.

Class	Action	No. of Data
Fall	(a) standing and then falling forward	153
	(b) standing and then falling to the left/right	306
	(c) standing and then falling backward	153
	(d) sitting and then falling forward	135
	(e) sitting and then falling to the left/right	270
Non-Fall	(f) walking slowly without moving arms	145
	(g) walking quickly while swinging arms	145
	(h) squatting	265
	(i) sitting on a chair	190
	(j) standing up from sitting on a chair	190
	(k) lying down and then lifting the upper body	240

3. Proposed Method

3.1. Overview

Figure 3 shows the overall structure of the proposed method. In the first step, the data acquired from the CW radar were preprocessed to generate a spectrogram that represented the micro-Doppler signature caused by human movement. The preprocessing consisted of STFT, min-max normalization, and thresholding. In the second step, the generated spectrogram was input into the BNN to detect falls. During this process, features were extracted and classified through the convolution layer (CL) and fully connected layer (FCL).

Figure 3. Architecture of the proposed method.

3.2. Pre-Processing

3.2.1. Short-Time Fourier Transform

When a target moves, the received and transmitted radar signals have different frequencies, which is known as the Doppler effect. The difference between the two frequencies is referred to as the Doppler frequency. If vibration or rotational movement is added to a moving object, an additional Doppler effect, called the micro-Doppler effect, occurs. FFT is a signal processing method that converts a signal from the time domain to the frequency domain. Because FFT is applied to the entire signal, it cannot detect the micro-Doppler signature, which is a subtle signal over short periods of time. Therefore, STFT, which divides the entire signal into intervals and performs FFT operations, was used. The calculation process is shown in Equation (1). Since STFT performs FFT on segmented signals, it provides time information and allows the signal to be represented as a spectrogram in the time-frequency domain. To perform STFT, three parameters must be determined: The first is the window size, which is the length of the signal on which the FFT is to be performed. The second is the window overlap ratio, which determines how much the windows overlap to perform the STFT. The third is the FFT point. The STFT parameters used in the proposed method were a window size of 128, window overlap ratio of 50%, and 128 FFT points. Examples of the spectrograms for actions (a)–(k) are shown in Figure 4.

$$X\left(m,w\right)=\sum _{n=-\infty }^{\infty }x\left[n\right]w\left[n-m\right]e^{-iwn}$$

(1)

Figure 4. Examples of the spectrograms for action (a–k). The colors in the spectrograms represent the signal strength and are expressed using Matlab’s Parula colormap.

3.2.2. Min–Max Normalization

Next, min-max normalization was performed. This process adjusts all pixel values within the spectrogram to a range between 0 and 1, as shown in Equation (2). Because fall detection uses the distribution of micro-Doppler signatures as a feature, signal intensity is not important. Therefore, there is no limit restricting the signal intensity to all actions within the same range. Additionally, it allows the system to process signals from various actions using the same algorithm, thereby contributing to a lightweight system design. In Equation (2), x represents the pixel value before normalization, Xmin refers to the minimum value among all the pixels in the spectrogram, and Xmax refers to the maximum value among all the pixels. Examples of normalized spectrograms are shown in Figure 5.

$$x_{scaled}={\displaystyle \frac{x-X_{min}}{X_{max}-X_{min}}}$$

(2)

Figure 5. Examples of the normalized spectrograms for action (a–k).

3.2.3. Thresholding

The normalized spectrogram is binarized through thresholding. A threshold value of 0.15 was used, where pixels smaller than the threshold were converted to −1 and the larger the threshold to 1. Since noise typically has a lower signal intensity than the target signal, this process can help remove noise. The threshold value to minimize noise was determined experimentally while preserving the micro-Doppler signatures. Table 3 shows the results of the experiments conducted to select the threshold value. Setting this threshold to 0.15 yields the highest accuracy. The region of interest (ROI) was extracted from the spectrogram with an input size of the network, 24 × 24. Because the spectrogram is composed of −1 and 1, the part with the largest sum of signals is the ROI, which contains the most signals. Examples of the binarized spectrograms are presented in Figure 6.

Table 3. Results of threshold selection experiment.

Threshold	0.1	0.125	0.15	0.175	0.2
Accuracy (%)	91.8	92.0	93.1	92.4	92.2

Figure 6. Examples of the binarized spectrograms for action (a–k).

3.3. Network

This section details the experiment for selecting the network used in the classifier and explains the network structure in detail. The network was based on a convolutional neural network (CNN). The network was selected through experiments by changing the number of convolutional layers, fully connected layers, output channels, and the presence or absence of pooling layers. Approximately 80% of the data was used for training, and the remaining 20% for testing. The cross-entropy function and Adam optimizer were used for training. The epoch and batch sizes were set to 200 and 32, respectively. The learning rates were set to 0.005, 0.001, 0.0005, 0.0001, 0.00005, and 0.00001 when the epochs were 0, 60, 100, 120, 140, and 160, respectively. NVIDIA GeForce RTX 2080 Ti GPU was used for training, requiring 1.5 s per epoch. Table 4 shows the detailed structure of the network used in the experiments, as well as the number of parameters and the accuracy of each network.

Table 4. Results of network configuration experiments.

Network	No. of Output Channels						No. of Par 1	Acc 2 (%)
	CL1	CL2	CL3	CL4	FCL1	FCL2
1	32	64	-	-	2	-	21,472	84.1
2	64	64	-	-	16	2	55,168	88.9
3	64	64	64	-	16	2	76,800	91.6
4	32	64	128	-	2	-	93,664	92.4
5	32	64	128	-	32	2	97,632	93.1
6	64	128	128	-	2	-	223,680	92.2
7	128	128	128	-	2	-	299,136	93.1
8	64	128	256	-	64	2	387,776	93.8
9	32	64	128	256	64	2	454,624	93.8

¹ Parameters; ² Accuracy.

Figure 7 shows the number of network parameters and accuracy trend. The experimental results showed that as the number of parameters increased, the accuracy increased and then converged. Network 5 performs much better than Networks 1–4 and slightly worse than Networks 7–9. However, Network 5 has much fewer parameters compared to the accuracy difference with networks 7–9. Therefore, this network is best suited for lightweight applications. The detailed structure of the proposed network is shown in Figure 8. The number of channels in the three convolution layers were 32, 64, and 128, and the kernel sizes were 5, 3, and 3. Batch normalization and max-pooling layers were applied after all convolution layers. A hard hyperbolic tangent was used as the activation function.

Figure 7. Number of parameters and accuracy of networks.

Figure 8. Structure of the proposed network.

4. Experiment Results and Discussion

In this section, we analyze the results of the experiment and compare its performance with those of previous studies. Table 5 presents the confusion matrix for the test results for the proposed network. Four performance metrics were obtained from the confusion matrix. The performance metrics used are shown in Equations (3)–(6). For the 452 data point experiment, the numbers of true positives (TP), false negatives (FN), false positives (FP), and true negatives (TN) were 193, 14, 17, and 228, respectively. Thus, the proposed method achieved a precision of 93.2%, a recall of 91.9%, an F1-score of 92.5, and an accuracy of 93.1%.

$$Precision={\displaystyle \frac{TP}{TP+FP}}$$

(3)

$$Recall={\displaystyle \frac{TP}{TP+FN}}$$

(4)

$$F1-score={\displaystyle \frac{2\times Precision\times Recall}{Precision+Recall}}$$

(5)

$$Accuracy={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}$$

(6)

Table 5. Confusion matrix of experimental results.

		Predicted Label
		Fall	Non-Fall
True label	Fall	193	14
	Non-Fall	17	228

Table 6 compares the performance of the proposed method with those of previous studies. Ref. [12] and the proposed method have almost similar performance with an accuracy difference of 0.3. However, our method has 2.75 times more actions and requires approximately 93% less memory. Ref. [16] has a 2.1% higher accuracy than the proposed method but has 36% fewer actions and requires approximately 91.9 times more memory. In addition, the cost of the overall system increases significantly owing to the use of FMCW radar. Compared with [18], the proposed method shows a higher accuracy, has 2.2 times more actions, and requires 99.4% less memory. Compared to [19], the proposed method is 2.7% less accurate and has 2.2 times more actions and requires approximately 99.9% less memory.

Table 6. Comparison with previous studies.

	[12]	[16]	[18]	[19]	This Work
Sensor	CW	FMCW	UWB	UWB	CW
Fall actions	2	4	3	3	5
Non-fall actions	2	3	2	2	6
Network	CNN	LKCNN 1	CNN	CNN	BNN
Memory usage (MB)	0.176	1.103	2.028	282.743	0.012
Accuracy (%)	93.4	95.2	92.7	95.8	93.1

¹ Line Kernel convolutional neural network.

5. Conclusions

This study proposes a lightweight fall detection method that uses CW radar and a BNN. We preprocessed the radar-acquired data to generate a spectrogram representing the micro-Doppler signature of human motion and used a deep-learning algorithm to classify it for fall detection. However, the many parameters of deep-learning algorithms and the resulting high memory requirements pose significant limitations for fall detection systems in home environments. To resolve this issue, a BNN was used to implement a lightweight fall detection method. The proposed method has a lower or similar overall performance compared with previous studies; however, it has a larger number of distinguishable motions and significantly reduced memory usage, making it more suitable for home-based fall detection systems. The proposed method achieved an accuracy of 93.1%, which is up to 2% lower than that in previous studies. However, it can distinguish up to 2.75 times more actions and uses up to 99.9% less memory (11.9 KB), confirming its suitability for home fall detection systems where light weight is essential.

In future work, we plan to implement the BNN on a VLSI to reduce power consumption and shorten the inference time, thereby developing a more advanced lightweight fall detection system.