Evaluating Feasibility of Pose Detection with Image Rotation for Monitoring Elderly People at Home ^†

Abstract

This study aims to enhance human pose detection performance through image preprocessing. With the growing global elderly population, supporting in-home elderly individuals has become increasingly crucial, especially in Japan, where approximately 90% of the elderly live at home. In this study, abnormal behaviors in in-home elderly individuals were detected using high-precision pose detection technologies, such as the MoveNet model, based on machine learning. However, there was a discrepancy in pose detection accuracy between the standing and lying positions. Hence, preprocessing techniques such as image rotation, flipping, resizing, and noise reduction were applied, and their effects were analyzed in detail. As a result, an improvement in pose detection accuracy for most body parts, except for specific regions, was observed. By constructing an optimal preprocessing pipeline, it is expected to reduce false detection and missed detection, contributing to the practical application of pose detection technology.

1. Introduction

As of 2021, approximately 80% of the world’s elderly population aged 65 and over live at home [1]. The proportion of elderly individuals living at home is particularly high in developed countries; for instance, in Japan, about 90% of the elderly population resides at home [2]. The number of dementia patients worldwide is estimated to reach 152 million by 2050 [3]. There is an increasing need to monitor dementia patients who are at higher risk of going missing or experiencing accidents [4]. The weakening of community ties and the decline in the caregiving capacity of families are contributing to the isolation of the elderly. Ensuring their safety and providing life support for isolated elderly individuals has become a critical issue. Consequently, the demand for monitoring services that support home care is rising [5].

Several systems and services utilizing engineering technology are currently available for monitoring elderly individuals living at home. Systems that detect the behavior and daily rhythms of the elderly using sensors and notify caregivers of any abnormalities have become widespread [6].

Services that employ artificial intelligence (AI) to learn the behavior patterns of the elderly and detect anomalies have also emerged [7]. Remote communication tools such as video calls and chat allow elderly individuals to connect with their families and caregivers [8]. Additionally, services that utilize mobile robots and assistive robots to support and monitor the daily lives of the elderly are now available [9].

We have developed monitoring support services for elderly individuals living at home by utilizing image-based human pose detection. We have proposed technologies for the quantification and measurement of physical activity quality within the home [10], as well as local sensing, using pre-trained pose detection models based on machine learning. Recently, we incorporated the high-precision pose detection model MoveNet [11] to predict and detect irregular behaviors of elderly individuals at home. Figure 1 shows the overview of human pose detection technology. Additionally, we have utilized pre-trained, image-based models that can be executed locally, such as facial expression emotion recognition models, hand movement detection models, and in-home object detection models.

This study aims to explore the potential for enhancing pose detection performance through the preprocessing of acquired images. The key idea involves the rotation of images using Python and OpenCV. Specifically, we apply preprocessing techniques such as rotation, flipping, resizing, and noise reduction to the images and analyze how these techniques affect the accuracy and efficiency of the pose detection algorithms. By improving the image quality, false detections and missed detections are reduced. Additionally, we examine various combinations of preprocessing methods to establish an optimal preprocessing pipeline. Ultimately, we evaluate the effectiveness of these preprocessing techniques in practical applications and contribute to the practical implementation of pose detection technology.

2. Preliminaries

2.1. Technical Challenges

Technological challenges include posture, viewpoint, image resolution, and occlusion which can significantly affect the accuracy of pose detection. Consequently, the accuracy of pose detection varies between the standing and lying humans due to these differing conditions (Figure 2).

• Differences in Posture: For standing humans, the positions of the joints are identifiable, resulting in higher accuracy of pose detection. In contrast, for lying humans, the positions of the joints may be obscured, making pose detection more challenging.
• Differences in Viewpoint: Standing humans are typically captured from the front, allowing for clear identification of the joint positions. Lying humans, however, are often captured from the side, making it more difficult to discern joint positions.
• Image Resolution: Standing humans appear larger within the image, leading to higher resolution. Lying humans appear smaller, resulting in lower resolution. Lower resolution decreases the accuracy of pose detection.
• Effect of Occlusion: Standing humans are less likely to be affected by occlusion, whereas lying humans may have parts of their body concealed by obstructions, thereby reducing the accuracy of pose detection.

2.2. Image Rotation with OpenCV

OpenCV is an open-source software library for computer vision and machine learning, providing a wide range of functionalities. Its newest version is 4.11.0. This library supports a variety of tasks, including image processing, video analysis, object detection, face recognition, and training of machine learning models, making it a powerful tool for researchers and developers. Furthermore, OpenCV is accessible through multiple programming languages, such as C++, Python, Java, and MATLAB, catering to a broad user base. Notably, Python is a preferred language among many researchers and engineers due to its high readability and low learning curve. As an example of OpenCV implementation in Python, Listing 1 demonstrates a sample code for rotating an image. This code implements the functionality of rotating an image by a specified angle, which is highly useful for image preprocessing tasks.

Listing 1. A sample code for rotating an image using Python.

import cv2 import numpy  as  np img = cv2 . imread ( ‘ image . j p g ’ ) rows , c o l s = img . shape [ : 2 ] c e n t e r = ( c o l s  /  2 ,  rows  /  2 ) a n g l e = 45 s c a l e = 1 . 0 r o t a t i o n m a t r i x  = cv2 . g e t R o t a t i o n M a t r i x 2 D ( c e n t e r ,  angle ,  s c a l e ) r o t a t e d i m g = cv2 . w a r p A f f i n e ( img ,  r o t a t i o n m a t r i x , ( c o l s , rows ) ) cv2 . imshow ( ‘ R o t a t e d Image ’ , r o t a t e d i m g ) cv2 . wait Key ( 0 ) cv2 . destroy All Windows ( )

3. Proposed Method

3.1. Goal and Key Idea

The purpose of this study is to investigate the potential for improving human pose detection performance through the preprocessing of acquired images. The key idea involves the use of image transformations with OpenCV. Specifically, we apply preprocessing techniques such as image rotation, flipping, resizing, and noise reduction, analyzing in detail how these affect the accuracy and efficiency of human pose detection algorithms. By enhancing image quality, we expect to reduce false detection and missed detection. Additionally, this study explores various combinations of preprocessing techniques to construct an optimal preprocessing pipeline. Ultimately, we aim to evaluate the effectiveness of these preprocessing methods in practical applications and contribute to the practical implementation of human pose detection technology.

3.2. Overall Architecture

Figure 3 illustrates the overall architecture of the proposed method, which consists of the following five steps: (Step 1) Input the image into the pose detection model. (Step 2) Determine the rotation parts and angles based on the detection results. (Step 3) Rotate the input image according to the determined rotation angles. (Step 4) Input the rotated image into the pose detection model. (Step 5) Compare and analyze the output results before and after the image rotation.

3.3. Method Description of Key Steps

Listing 2 presents three methods for calculating slopes using Python, while Listing 3 demonstrates one method for converting slopes to angles using Python. Furthermore, Listing 4 illustrates a method for rotating images using Python. By appropriately rotating the images based on the original output results and re-detecting them, it is expected that the accuracy of the results will improve, thereby standardizing the human postures.

Listing 2. Three ways to calculate slope using Python.

# Method 1 from scipy.stats import linregress x_ = [keypoints_xlist[5], keypoints_xlist[6]] y_ = [keypoints_ylist[5], keypoints_ylist[6]] slope, intercept, r_value, p_value, std_err = linregress(x_, y_) print(slope) # Method 2 x1_ = keypoints_xlist[5] y1_ = keypoints_ylist[5] x2_ = keypoints_xlist[6] y2_ = keypoints_ylist[6] def slopee(x1, y1, x2, y2): x = (y1 - y2) / (x1 - x2) return x print(slopee(x1_, y1_, x2_, y2_)) # Method 3 import numpy as np slope_, intercept_ = np.polyfit(x_, y_, 1) print(slope_)

Listing 3. One way to convert ratio to angle using Python.

import math angle = math.degrees(math.atan(slope)) print(angle)

Listing 4. One way to rotate an image using Python.

from PIL import Image im = Image.open(file_path) im_rotate = im.rotate(angle, expand=True) plt.figure(figsize=(16, 12), dpi=20) plt.imshow(im_rotate) plt.gca().set_axis_off() plt.subplots_adjust(top = 1, bottom = 0, right = 1, left = 0, hspace = 0, wspace = 0) plt.margins(0,0) plt.savefig(file_path + "_rotate.jpg")

4. Case Study and Discussion

We use a human pose detection model and input an image of an elderly person lying on their left side into the proposed method. The image is rotated to an upright position before performing pose detection, as shown in Figure 4. Specifically, the detection accuracy for body parts such as the left shoulder, right elbow, left wrist, and left ankle decreases, while the accuracy for the nose, left eye, left ear, right shoulder, and left knee significantly improves.

Image rotation as a preprocessing method improves the pose detection accuracy for joints while decreasing it for others. In light of these findings, we consider the following four points:

• Reconsideration of Preprocessing Methods: Since image rotation is effective for some joints but not for others, it is necessary to explore preprocessing methods beyond rotation, such as interpolation using augmented reality technology or image completion from other viewpoints.
• Model Improvement: Current pose detection models, including the MoveNet model, may be optimized for specific postures. To maintain high accuracy for lying positions as well, it is necessary to improve the model and introduce additional training data.
• Utilization of Multi-view Data: Integrating data from multiple viewpoints, rather than relying solely on images from a single viewpoint, has the potential to improve overall detection accuracy. Especially for lying postures, combining information from different angles can mitigate the effects of occlusion.
• Challenges Toward Practical Application: To realize a monitoring system for the elderly, a flexible detection system that can accommodate various postures and environments is required. Based on the current results, it is necessary to conduct further validation in a wider range of environments and postures to clarify the challenges toward practical application.

5. Conclusions

In this study, we enhance pose detection performance through image preprocessing. Specifically, we apply preprocessing techniques such as rotation, flipping, resizing, and noise reduction using OpenCV and analyze in detail how these techniques affect the accuracy and efficiency of pose detection algorithms. As a result, we observe a significant improvement in pose accuracy for body parts other than 5, 8, 9, and 15. This indicates that improving image quality can reduce false detection and missed detection. Furthermore, by examining combinations of different preprocessing methods, we suggest the potential for further improving the accuracy and efficiency of pose detection technology by constructing an optimal preprocessing pipeline.

It is still necessary to address the following challenges:

• Validation with Diverse Datasets: This study was conducted under specific conditions, so it is necessary to examine the generalizability of the preprocessing methods using more diverse datasets.
• Real-Time Processing: It is crucial to evaluate whether the implementation of preprocessing methods is suitable for real-time processing and to ensure practical applicability.
• Optimization of Preprocessing Methods: There is a need to optimize combinations of different preprocessing methods to develop a more accurate preprocessing pipeline.
• Comparison with Other Pose Detection Models: It is important to compare the proposed method with other pose detection models besides MoveNet to verify its generality and effectiveness. Addressing these challenges is expected to lead to the development of more accurate pose detection technologies and represent a significant step toward practical implementation.