Multi-Step Action Recognition for Long-Term Care Using Temporal Convolutional Network–Dynamic Time Warping–Finite State Machine and MediaPipe ^†

Abstract

An intelligent multi-step action recognition system was designed for long-term caregiver training and assessment. Leveraging MediaPipe for precise and real-time human pose estimation, the system extracts detailed spatiotemporal body and hand keypoints. Temporal convolutional networks are employed to effectively capture temporal dependencies and complex features from sequential motion data. Dynamic time warping provides robust sequence alignment, allowing flexible comparison between performed actions and standard templates despite temporal variations in execution speed or style. A finite state machine imposes logical constraints by modeling expected action step sequences, enabling accurate detection of sequence anomalies or deviations. This hybrid architecture supports comprehensive evaluation and real-time feedback, facilitating improved caregiver skill acquisition, process adherence, and quality control within long-term care settings. The system aims to advance digital transformation in healthcare education by providing a scalable, precise, and adaptive training solution.

1. Introduction

The growing demand for high-quality, long-term care, coupled with the increasing complexity of nursing procedures, highlights the need for effective and systematic caregiver training [1]. Traditional skill-training methods often rely on subjective instructor assessments, lack real-time feedback, and provide limited objective assessments or ways to track learners’ skill progress, especially in multi-step tasks. These drawbacks can lead to inconsistent training quality, reduced safety, and difficulties in standardizing best practices. Recent advancements in AI, such as pose estimation and sequence modeling, offer innovative tools for objectively capturing, analyzing, and evaluating complex human movements [2]. For example, MediaPipe [3,4] tracks human body and hand movements in real-time and with high accuracy, while the time convolutional network (TCN) [5], combined with dynamic time warping (DTW) [6] and finite state machine (FSM), provides a robust framework for identifying and evaluating sequential nursing actions, accommodating execution variations and enforcing logical step sequences. Integrating these technologies is expected to facilitate continuous, personalized, and accurate training, effectively addressing the major limitations of existing approaches. Therefore, we utilized these AI technologies to develop an intelligent, data-driven nursing skills education system as an auxiliary tool for future long-term care staff training and assessment, aiming to enhance adaptability in long-term care environments.

2. Related Works

Human activity recognition (HAR) and action sequence prediction are gaining attention in computer vision, ubiquitous computing, and smart healthcare. Traditional methods using handcrafted features from sensors or video often struggle with complex activities. Recent deep learning advances, especially spatial–temporal architectures, have greatly improved HAR robustness and generalizability.

One widely adopted framework is the convolutional neural network–long short-term memory (CNN-LSTM) hybrid model. CNNs are highly effective at extracting spatial representations from individual frames or skeletal keypoints, while LSTMs specialize in modeling temporal dependencies across sequences. This combination has been shown to achieve state-of-the-art performance on benchmark datasets such as the Mobile Health Dataset and the Human Activity Recognition Using Smartphones Dataset, often reporting accuracies above 98%. Compared with single-model baselines, CNN-LSTM architectures demonstrate superior ability to recognize repetitive movements and multi-step activities, which are particularly relevant in healthcare scenarios such as rehabilitation exercises, fall detection, and continuous patient monitoring [7,8,9].

CNN-LSTM models are widely applied in healthcare for monitoring care activities, detecting abnormal movements (e.g., falls, bed exits), and supporting elderly or rehabilitation care in clinical and home settings. They flexibly handle heterogeneous inputs such as sensor data, video, and pose-estimation outputs (e.g., OpenPose, MediaPipe). To enhance performance and adaptability across populations, advanced variants incorporate attention mechanisms, bidirectional LSTMs, and data augmentation strategies [9,10,11].

In Taiwan and other East Asian regions, researchers have applied CNN-LSTM and pose-based deep learning systems to long-term care monitoring, including tasks such as violence detection in institutions and bed-exit/fall alerts for elderly residents. These implementations highlight the clinical value of such models in improving safety and enabling timely caregiver intervention [12,13,14].

Recent studies [5,6,15] have highlighted the potential of combining TCN, DTW, FSM, and MediaPipe pose estimation in healthcare applications. This integrated approach enables automated analysis of caregiver actions, clinical workflows, and patient behavior patterns. Looking ahead, the fusion of TCN’s ability to capture long-term temporal dependencies, DTW’s flexible sequence alignment, and FSM’s logical step management with MediaPipe’s real-time pose tracking is expected to greatly improve system efficiency, adaptability, and real-time responsiveness, making these techniques fundamental for next-generation intelligent healthcare monitoring and training systems.

2.1. MediaPipe Description

MediaPipe is an open-source machine learning framework developed by Google, designed for building pipelines to process time-series data such as video and audio. It is widely used in applications, including human pose estimation. MediaPipe is a cross-platform framework that supports a range of environments, including desktop and server systems, Android, iOS, and embedded devices like Raspberry Pi and Jetson Nano. This broad platform compatibility allows developers to build applications that perform consistently across various device ecosystems—desktop, mobile, web, and edge devices. Its pose estimation module enables advanced capabilities such as posture analysis, movement tracking, and activity classification [3,4].

2.2. TCN-DTW Framework

TCNs are advanced deep learning models designed to capture long-range temporal dependencies in sequential data. Utilizing causal and dilated convolutions, TCNs efficiently process time series by ensuring no future information leakage and expanding the receptive field exponentially. This enables effective modeling of complex temporal patterns with high computational efficiency and parallelization. TCNs excel in large-scale sequence classification tasks where abundant labeled data is available, automatically extracting hierarchical features that represent temporal dynamics [5].

DTW, on the other hand, is a classical non-parametric alignment technique that computes an optimal match between two time series sequences, accommodating differences in length and speed by elastically warping the time axis. It serves as a robust similarity measure capable of aligning sequences that exhibit temporal distortions, making it particularly effective for anomaly detection and template matching in scenarios with limited training data. However, DTW’s computational cost scales quadratically with sequence length and lacks the capacity to learn high-level features autonomously [6].

Integrating TCN with DTW unites deep feature learning and flexible temporal alignment. TCNs capture rich temporal patterns, while DTW handles variations in sequence speed and duration. This synergy improves robustness and accuracy in multi-step action recognition and anomaly detection, especially in healthcare settings with scarce and variable data. By combining learned representations with template matching, the approach enables efficient sequence classification adaptable to real-world temporal dynamics [15].

3. Proposed Methodology

The integration of TCN, DTW, FSM, and MediaPipe is well-suited for applications with limited data. TCNs offer efficient temporal feature extraction with fewer samples compared to complex deep learning models. DTW provides flexible, non-parametric sequence alignment without large training datasets. FSMs use rule-based state management for interpretable and logical process evaluation, minimizing reliance on extensive data. MediaPipe delivers real-time, precise skeletal keypoint extraction, ensuring reliable inputs even with small datasets. This method balances deep learning and rule-based methods, making it ideal for multi-step action recognition and assessment in clinical, educational, or smart caregiving environments where data is scarce and annotations costly. The proposed workflow is shown in Figure 1.

Figure 1. The system workflow.

3.1. Data Extraction with MediaPipe

Using MediaPipe, 33 key body landmarks are extracted in real time from video input, capturing the participant’s movements. Each landmark is represented by its three-dimensional coordinates (x, y, z). To ensure spatial consistency across different camera views and subjects, the raw landmark coordinates are preprocessed using relative positioning. Specifically, landmark coordinates are normalized relative to the pelvic center, and scaled by the participant’s shoulder width to account for individual anatomical differences.

3.2. MediaPipe Data: Relative Positioning and Normalization

To normalize for spatial variability caused by camera position and participant placement, we adopt a relative positioning method based on both pelvic and shoulder landmarks. Specifically, we selected the pelvic center (MediaPipe landmark index 0) as the primary reference point for translational alignment. Additionally, the horizontal distance between the left and right shoulder points (landmark indices 11 and 12) is used to define a scale factor reflecting the participant’s shoulder width. This scale is employed to normalize the size of the skeleton to reduce inter-subject body size variation. The relative and normalized coordinates ${(x_i}^{\prime },{y_i}^{\prime },{z_i}^{\prime })$ for each landmark are computed as follows.

$$\left(x_i^{\prime },y_i^{\prime },z_i^{\prime }\right)={\displaystyle \frac{\left(x_i-x_{pelvic},y_i-y_{pelvic},z_i-z_{pelvic}\right)}{d_{shoulders}}}$$

(1)

where $d_{shoulders}$ is the Euclidean distance between left and right shoulder landmarks.

The processed relative and normalized landmark data serve as stable and standardized input features for downstream analysis. TCN extracts deep temporal features from these sequences, enabling precise classification, while DTW aligns variable-length sequences flexibly, facilitating robust assessment of healthcare-related movements.

3.3. TCN+DTW Framework

The process begins with data preprocessing, where input sequences are extracted from MediaPipe skeleton-based data. To ensure consistency, the sequence length is normalized using padding or sliding windows, with a default length of 30 frames. These preprocessed sequences are then fed into TCN, which learns temporal-invariant representations and captures motion dynamics over time. The output of TCN is a feature embedding, represented either as a sequence of feature vectors or as a pooled representation.

Next, the extracted feature sequence is compared against stored standard templates using DTW. This method aligns sequences along the temporal dimension, compensating for differences in execution speed, and the alignment cost serves as a measure of similarity between the two actions. For each input sequence, DTW distances are computed against all action templates, and the action label corresponding to the minimum distance is selected. If the mean distance exceeds a predefined threshold, the action is classified as the rejected.

The TCN-DTW framework offers robust temporal feature extraction, ensuring invariance to variations in execution speed and delivering reliable recognition even with limited training data. In Figure 2, the four sub-action visual templates trained using the TCN model are shown in this experiment. To enhance clarity, only the main skeletal features are presented in these images. To address the differences in movement speeds between caregivers, as driven by the design of streaming DTW [16,17], we employ frame buffer management and utilize computational power to implement online inference, overcoming the limitations of the TCN model in real-time sub-action alignment. This integration supports continuous, real-time comparison and matching of ongoing motion sequences, enabling precise and adaptive assessment in healthcare applications. By effectively managing the frame buffer, the system performs inference at 0.5 s intervals, thereby enhancing both real-time responsiveness and processing efficiency.

Figure 2. Visual standard templates of main joint sequence for 4 sub-actions: (a) Standard template for Hands Up action; (b) Standard template for Wave action; (c) Standard template for Squat Down action; (d) Standard template for Bend Over action.

3.4. FSM Controllor

In this system, the FSM controller manages the flow of multi-step actions using predefined standard models derived from expert caregiver demonstrations. Temporal and spatial features form reference patterns, and sub-actions are marked complete only when the performed data satisfactorily match these models, enabling process evaluation, anomaly detection, and instructional feedback. To regulate the action sequence, FSM controls the workflow with states representing each sub-action step, as follows.

[S₀: Start → S₁: Lift → S₂: Turn → S₃: Place → End.]

(2)

Transitions between states occur only after the correct completion of the preceding sub-action, ensuring strict adherence to the standardized care procedure and enabling precise monitoring and quality assurance throughout the process.

3.5. Training and Optimization

Figure 3 shows the TCN-DTW-FSM motion recognition system prototype, highlighting its functions: collecting action data, training the TCN model, creating templates, starting inference, and monitoring compound motions. The model will use multi-step ground truth sub-action sequences for supervised learning, with data augmentation and regularization to enhance generalization and robustness.

Figure 3. The TCN-DTW-FSM Motion Recognition System Executions: (a) Refusal to adjudicate if DTW distance exceeds the pretrained threshold; (b) Inference results: Confidence: 1.00 DTW: 0.3 (wave action).

4. Conclusions

This integrated TCN-DTW-FSM framework offers a scalable and interpretable solution for multi-step action recognition in healthcare training and monitoring, particularly focusing on long-term care personnel training. It supports both real-time training via live camera feeds and offline processing through annotation and extraction from prerecorded videos. By capturing fine-grained sequential motions with temporal dependencies, the system facilitates accurate skill assessment, error detection, and real-time feedback in clinical and remote care settings. To handle caregiver variability and ensure timely inference, we implemented online matching inference by efficient buffer management.

By combining deep learning with rule-based methods, the framework balances efficiency and robustness, making it ideal for multi-step action recognition in clinical, educational, and caregiving environments, especially where data scarcity and high annotation costs are challenges. Currently in the prototype stage, the system has not been trained or evaluated on real-world datasets. Future development will refine dataset preparation, learning configurations, data augmentation, training protocols, and evaluation metrics to improve model performance and system reliability, while also adding multi-person tracking for interactive or group scenarios.