Video-Based Quantitative Assessment of Upper Limb Impairments in Patients with Brain Lesions During Resistance Exercises

Abstract

This study proposes a video-based approach for quantitatively evaluating upper-limb joint abnormalities in individuals with brain lesions during resistance exercises. While the Fugl–Meyer Assessment (FMA) is a reliable clinical tool, its use is limited by the need for expert involvement and repeated assessments. To address this issue, skeletal joint data were extracted from RGB exercise videos using OpenPose, and joint abnormalities were identified by learning normal movement patterns from non-disabled participants. A total of 26 non-disabled individuals and 12 individuals with brain lesions performed chest press, shoulder press, and arm curl exercises. Joint movement patterns were analyzed using correlation analysis and a long short-term memory (LSTM) autoencoder. Only joints relevant to each exercise were evaluated, and joint-level results were integrated to compute arm-level abnormality rates. The correlation-based abnormality rate showed a significant negative correlation with FMA scores (r = −0.7789, p = 2.83 × 10⁻³), while the LSTM autoencoder-based abnormality rate exhibited a stronger correlation(r = −0.8454, p = 5.33 × 10⁻⁴). In addition, affected-side classification accuracy reached 78.0% and 83.3% for correlation analysis and the LSTM autoencoder, respectively. These results indicate that the proposed method is consistent with clinical assessments and can serve as a non-invasive, cost-effective tool for video-based rehabilitation evaluation.

1. Introduction

Individuals with brain lesions often experience difficulties in performing activities of daily living due to upper-limb muscle weakness, reduced joint range of motion, and asymmetric movement patterns [1,2,3,4,5,6,7]. Previous studies have consistently reported that upper-limb motor impairment is highly prevalent in this population, with approximately 70–85% of stroke patients exhibiting some degree of upper-limb dysfunction during the acute stage [8]. Although partial recovery of hand function has been observed in some patients, only about 38% regain limited hand function and merely 11.6% achieve complete recovery within six months after a stroke, indicating that more than half of patients fail to fully recover hand function in the chronic phase [1]. Such impairments in upper-limb function not only diminish an individual’s quality of life [9,10] but also necessitate long-term rehabilitation and continuous functional assessment [11,12].

In clinical practice, standardized assessment tools such as the Fugl–Meyer Assessment (FMA) are widely used to qualitatively evaluate upper-limb function [13,14]. However, these assessments rely heavily on direct observation and intervention by experienced clinicians and are limited by the time and cost associated with repeated evaluations.

In recent years, advances in video-based human motion analysis and deep learning techniques have enabled the quantitative analysis of joint movements and the automated assessment of movement abnormalities without expert intervention. In particular, RGB camera-based pose estimation methods, such as OpenPose (version.1.7.0) and MediaPipe, allow joint position information to be obtained without the need for wearable sensors [15,16,17,18], highlighting their potential as non-invasive rehabilitation assessment tools suitable for home-based environments [19,20,21,22,23]. Among various pose estimation frameworks, OpenPose was selected in this study due to its open-source availability, robustness in multi-joint tracking, and extensive validation in upper-limb motion analysis. Importantly, OpenPose enables markerless skeletal extraction using only RGB video data, which is particularly advantageous in rehabilitation settings where patients often experience discomfort or reluctance toward wearable sensors. This non-contact approach improves participant compliance and enhances the feasibility of clinical deployment during repetitive resistance exercises. These technological developments extend beyond a conventional large language model (LLM) [24] toward the concept of a large action model (LAM), which focuses on learning, understanding, and executing human actions [25]. Consequently, their applicability in rehabilitation and healthcare domains is increasingly gaining attention.

Previous studies have investigated human movement analysis using vision-based pose estimation techniques, primarily focusing either on normal movement patterns in non-disabled individuals or on movement characterization in individuals with motor impairments. For example, Lee et al. utilized OpenPose-based joint extraction combined with temporal–spatial feature analysis and support vector machines to quantitatively analyze postural patterns in non-disabled young adults [26]. Roggio et al. presented a comprehensive narrative review of machine learning-based pose estimation models and their applications in human movement and posture analysis, highlighting the development and capabilities of models such as OpenPose, PoseNet, DeepLabCut, HRNet, and MediaPipe, and emphasizing their potential for accessible, non-invasive motion analysis across clinical, sports, and ergonomic context [27]. In contrast, Ma et al. employed a deep learning-based approach using Kinect v2 data to assess upper-limb function in individuals with motor impairments, focusing on functional score estimation rather than joint-wise abnormality detection through comparison with normative movement patterns [28].

Despite these advances, recent studies and systematic reviews have emphasized that automated upper-limb movement assessment remains an open challenge in rehabilitation contexts. In particular, a recent systematic review by Molle et al. highlighted several unresolved issues in AI-based upper-limb rehabilitation, including the lack of standardized assessment frameworks, limited interpretability of AI-derived metrics, and insufficient linkage between algorithmic outputs and clinically meaningful indicators [29]. The review further pointed out that many existing approaches focus on global performance scores or classification accuracy, while providing limited insight into joint-specific functional deterioration or compensatory movement patterns, which are critical for clinical decision-making.

However, in practical rehabilitation settings, there is a growing need for analytical approaches that go beyond technical analysis of individual movements to quantitatively assess joint function deterioration through comparisons with normal movement patterns and to link these findings with clinical indicators. In particular, objectively identifying which joints exhibit functional impairment during exercise is a critical factor in formulating effective rehabilitation strategies.

In this paper, we propose a method for quantitatively analyzing joint movements and assessing joint abnormalities during resistance exercises based on joint data extracted from video recordings. As illustrated in Figure 1, participants performed three resistance exercises targeting the upper limb, namely shoulder press, chest press, and arm curl, while their movements were recorded using a single RGB camera in a controlled environment. The recorded videos were processed to extract skeletal joint trajectories, which were subsequently organized as multivariate time series data for analysis. To further contextualize the proposed approach, waveform similarity-based correlation analysis and the LSTM autoencoder were applied to the same joint position data, allowing a comparative evaluation of joint abnormality detection performance. The resulting joint-specific abnormality measures were then examined in relation to clinical assessment outcomes.

Joint position data are obtained using OpenPose, after which normal joint movement patterns are learned from non-disabled participants and subsequently applied to exercise data from individuals with brain lesions to detect joint function deterioration. Because the extracted joint trajectories inherently form multivariate time-series data, an LSTM-based autoencoder was employed to model temporal continuity and learn normal movement dynamics in an unsupervised manner. LSTM networks are well suited for capturing sequential dependencies in joint kinematics and can effectively model continuous movement patterns with limited training data. Compared to conventional machine learning methods that rely on handcrafted features, the LSTM autoencoder directly learns temporal dependencies from raw joint trajectories, enabling more robust modeling of complex and variable movement patterns. In addition, unlike supervised classification models, the autoencoder framework does not require labeled abnormal data, which are difficult to obtain in clinical populations.

This characteristic makes the LSTM autoencoder particularly suitable for rehabilitation scenarios, where normal movement data from non-disabled individuals can be leveraged to identify deviations in patients with brain lesions through reconstruction error analysis. To this end, waveform similarity-based correlation analysis [30,31,32,33] and a long short-term memory (LSTM) autoencoder that learns the temporal continuity of joint movements [34,35,36,37,38,39,40,41] are employed for joint abnormality detection, and the characteristics and outcomes of the two approaches are comparatively analyzed. This framework enables objective identification of joint-level abnormalities during exercise and facilitates comparison with clinical assessment outcomes.

Notably, by restricting the evaluation targets to only those joints actively involved in each exercise according to predefined exercise definitions, joint abnormality judgments unrelated to the exercise characteristics are excluded, thereby minimizing false positives. In addition, abnormality rates are computed for each exercise and joint and are compared with the clinical assessment metric, the Fugl–Meyer Assessment (FMA), to validate the clinical relevance of the proposed indicators [42]. Furthermore, exercises exhibiting higher abnormality rates are prioritized, and rehabilitation exercise recommendations are generated accordingly, exploring the feasibility of personalized rehabilitation strategy development.

Overall, this approach presents an objective and quantitative method for evaluating upper-limb function in individuals with brain lesions through video-based joint movement analysis and demonstrates its potential for extension to non-invasive, cost-effective remote rehabilitation assessment and monitoring systems.

2. Materials and Methods

2.1. Subjects

The sample size of this study was determined in advance through a power analysis to ensure sufficient statistical power for correlation analysis. Based on previous studies, a large correlation for the alternative hypothesis of $\mathsf{\rho }=0.7$, a significance level of $\mathsf{\alpha }=0.05$, and a statistical power of $1-\mathsf{\beta }=0.8$ were assumed, resulting in a minimum required sample size of approximately 13 participants.

A total of 45 participants were recruited, including 33 non-disabled individuals and 12 individuals with brain lesions. Patient information is provided in

Table A1. Patient Data Information Table.

ID	Sex	Full FMA (66)	Affected Side
1	Female	53	Left
2	Male	20	Right
3	Female	11	Right
4	Male	12	Right
5	Male	53	Right
6	Male	7	Right
7	Male	57	Left
8	Female	36	Left
9	Female	32	Right
10	Female	66	Right
11	Female	66	Left
12	Male	66	Right

. Accordingly, the total sample size exceeded the minimum requirement, indicating that sufficient statistical power was secured for conducting correlation analyses. The non-disabled group included a relatively larger number of participants to obtain reference data representing normal movement patterns and to estimate average characteristics more reliably.

This difference in group size was intentionally addressed in the analysis, as data from non-disabled participants were used to train normal joint movement patterns and to establish abnormality thresholds, whereas data from participants with brain lesions were used only for evaluation.

In contrast, recruitment of participants with brain lesions was subject to practical constraints due to the nature of the target population. Although the number of participants with brain lesions (n = 12) was slightly below the minimum sample size suggested by an a priori power analysis, this cohort size is comparable to that of previous exploratory studies involving clinical populations. Furthermore, the correlation coefficients observed in the experimental results were consistently higher than the assumed effect size used in the power analysis, supporting the validity of the subsequent analyses despite the limited clinical sample size.

All participants with brain lesions were in the chronic stage, having passed the acute phase, such that their motor impairments were considered relatively stable at the time of data collection. This criterion was adopted to minimize variability associated with spontaneous neurological recovery. To further enhance the functional homogeneity of the disabled group, individuals with severe bilateral upper-limb impairments who were unable to perform the predefined exercise tasks were excluded. As a result, all included participants with brain lesions retained sufficient voluntary upper-limb movement to complete the experimental protocol, allowing meaningful joint movement analysis.

Taken together, the group composition and analysis strategy adopted in this study are considered appropriate for learning stable normal movement patterns and for evaluating deviations in joint movement characteristics in individuals with brain lesions. Nevertheless, the relatively small sample size of the brain lesion group is acknowledged as a limitation, and future studies should include a larger cohort of participants with brain lesions for further validation.

This study was conducted in accordance with the Declaration of Helsinki and was approved by the Institutional Review Board of Handong Global University (protocol code: 2025-HGUA026, approval date: 7 November 2025).

2.2. Experimental Setup

A smart mirror system was used to assess participants’ exercise performance. The smart mirror was installed at a distance of approximately 280 cm from the participant and consisted of an integrated RGB camera (Microsoft LifeCam Studio, Microsoft, Redmond, WA, USA) and a monitor. Exercise movements were recorded at a resolution of 1280 × 720 pixels and a frame rate of 30 fps. A pre-recorded exercise instruction video was displayed on the monitor, and participants performed the exercises while seated, following the visual guidance. During exercise execution, a low-intensity yellow elastic resistance band (THERABAND, The Hygienic Corporation, Akron, OH, USA) was used. All participants performed the exercises under identical conditions and synchronized their movements with the pace of the instructional video presented on the smart mirror. An overview of the experimental setup is illustrated in Figure 2a.

2.3. Experimental Procedure

The experiment included three resistance exercises performed in the order of chest press, shoulder press, and arm curl. Each exercise was performed for 20 repetitions using an elastic resistance band, with participants matching their movements to the speed of the instructional video displayed on the smart mirror. The resistance exercises applied in this study were selected through consultation with physicians and clinical experts in the Department of Rehabilitation Medicine at Pohang Stroke and Spine Hospital. The selection criteria comprehensively considered safety for individuals with brain lesions to perform the exercises in a seated position, clinical effectiveness for improving upper-limb joint mobility and muscle strength, and practical applicability in rehabilitation settings. Based on these considerations, chest press, shoulder press, and arm curl exercises were adopted as the evaluation tasks in this study. The exercise videos used in this study were created by the research team. All exercise sessions were recorded, and skeletal models were extracted from each video frame using OpenPose.

2.4. Data Analysis

2.4.1. Data Processing

The recorded videos were processed using the OpenPose software (v1.7.0, CMU Perceptual Computing Lab, Pittsburgh, PA, USA) to estimate joint positions. Participants’ movement data were represented as joint coordinate data using the Body 25 model. Missing values generated during the estimation process were corrected using interpolation based on the preceding and subsequent frames, and noise was removed using a second-order Butterworth low-pass filter. As resistance exercises do not involve rapid movements, the cutoff frequency was set to 1.5 Hz.

To compensate for inter-participant differences in body dimensions, joint coordinates were normalized based on each participant’s shoulder width. The joints and corresponding joint indices used in the OpenPose Body 25 model are presented in

Table 1. Body 25 Data Specification.

Data Number	Joint Name	Data Number	Joint Name
0	Nose	13	L Knee
1	Neck	14	L Ankle
2	R Shoulder	15	R Eye
3	R Elbow	16	L Eye
4	R Wrist	17	R Ear
5	L Shoulder	18	L Ear
6	L Elbow	19	L Big Toe
7	L Wrist	20	L Small Toe
8	Mid Hip	21	L Heel
9	R Hip	22	R Big Toe
10	R Knee	23	R Small Toe
11	R Ankle	24	R Heel
12	L Hip

.

Shoulder width was calculated as the distance between OpenPose joint indices 2 (right shoulder) and 5 (left shoulder), as defined in Equation (1). The joint coordinates were then normalized using Equation (2), and a relative coordinate system was applied with OpenPose joint index 8 (mid-hip) as the reference point to ensure consistency of the coordinate origin. Joint angles were calculated using Equation (3), and the joint angles used in this study are presented in

Table 2. Joint Angles.

Angle	Joint	Angle	Joint
${\mathsf{\theta }}_1$	R Elbow	${\mathsf{\theta }}_3$	L Elbow
${\mathsf{\theta }}_2$	R Shoulder	${\mathsf{\theta }}_4$	L Shoulder

.

For each resistance exercise, 15 repetitions were selected for analysis by excluding the first three repetitions corresponding to the initial adaptation phase and the last two repetitions potentially affected by fatigue from the total of 20 repetitions. At this stage, during point-to-point movements, the segment in which the wrist velocity profile most closely resembled a bell-shaped curve was selected for analysis using Equation (4) [43,44,45].

Among the collected data, recordings from 7 non-disabled participants who exhibited inappropriate postures during exercise were identified as outliers and excluded. Consequently, data from 26 non-disabled individuals and 12 individuals with brain lesions were included in the final analysis.

$$\mathrm{S}\mathrm{h}\mathrm{o}\mathrm{u}\mathrm{l}\mathrm{d}\mathrm{e}\mathrm{r} \mathrm{l}\mathrm{e}\mathrm{n}\mathrm{g}\mathrm{t}\mathrm{h}\left(\mathrm{p}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{l}\right)=\sqrt{{\left(x_5-x_2\right)}^2+{\left(y_5-y_2\right)}^2 }$$

(1)

where $x_2,{ y}_2$ and $x_5,y_5$ denote the two-dimensional pixel coordinates of the left and right shoulder joints, respectively, extracted using OpenPose. Shoulder length is measured in pixels and is used as a normalization reference to reduce inter-subject scale variability.

$$\mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{z}\mathrm{e}\mathrm{d} \mathrm{j}\mathrm{o}\mathrm{i}\mathrm{n}\mathrm{t} \mathrm{c}\mathrm{o}\mathrm{o}\mathrm{r}\mathrm{d}\mathrm{i}\mathrm{n}\mathrm{a}\mathrm{t}\mathrm{e}\mathrm{s}={\displaystyle \frac{\mathrm{J}\mathrm{o}\mathrm{i}\mathrm{n}\mathrm{t} \mathrm{c}\mathrm{o}\mathrm{o}\mathrm{r}\mathrm{d}\mathrm{i}\mathrm{n}\mathrm{a}\mathrm{t}\mathrm{e}\mathrm{s}}{\mathrm{S}\mathrm{h}\mathrm{o}\mathrm{u}\mathrm{l}\mathrm{d}\mathrm{e}\mathrm{r} \mathrm{l}\mathrm{e}\mathrm{n}\mathrm{g}\mathrm{t}\mathrm{h}}}$$

(2)

where the joint coordinates represent the raw two-dimensional pixel coordinates $(\mathrm{x},\mathrm{y})$ of each joint, and shoulder length is defined in Equation (1). This normalization ensures scale invariance across participants.

$${\theta }_n\left(\mathrm{d}\mathrm{e}\mathrm{g}\mathrm{r}\mathrm{e}\mathrm{e}\right)={\cos }^{-1}\left({\displaystyle \frac{\overrightarrow{A}·\overrightarrow{B}}{ ‖\overrightarrow{A}‖ ‖\overrightarrow{B}‖}}\right)$$

(3)

where $\overrightarrow{A}$ and $\overrightarrow{B}$ are vectors formed by adjacent joints defining the joint angle of interest, $\overrightarrow{A}\cdot \overrightarrow{B}$ denotes the dot product, and $\parallel \overrightarrow{A}\parallel$ and $\parallel \overrightarrow{B}\parallel$ represent the Euclidean norms of the vectors. The resulting angle ${\theta }_n$ is expressed in degrees.

$$\mathrm{r}\left(\mathrm{t}\right)=\left(\mathrm{x}\left(\mathrm{t}\right),\mathrm{y}\left(\mathrm{t}\right)\right), \mathrm{v}\mathrm{e}\mathrm{l}\mathrm{o}\mathrm{c}\mathrm{i}\mathrm{t}\mathrm{y}\left(\mathrm{t}\right)=\left|{\displaystyle \frac{\mathrm{d}\mathrm{r}\left(\mathrm{t}\right)}{\mathrm{d}\mathrm{t}}}\right| [\mathrm{pixel}/\sec ]$$

(4)

where $\mathrm{r}\left(t\right)$ represents the two-dimensional joint position at time $t$, $x\left(t\right)$ and $y\left(t\right)$ are pixel coordinates, and $t$ denotes time in seconds. Velocity is computed as the magnitude of the first derivative of the joint position with respect to time and is expressed in pixels per second.

2.4.2. Correlation

$${\mathsf{\rho }}_{\mathrm{x}\mathrm{y}}\left(\tau \right)={\displaystyle \frac{{\sum }_t(x\left(t\right)-\overline{x})(y\left(t+\tau \right)-\overline{y})}{\sqrt{{\sum }_t{\left(x\left(t\right)-\overline{x}\right)}^2{\sum }_t{\left(y\left(t+\tau \right)-\overline{y}\right)}^2 }}}$$

(5)

where $x\left(t\right)$ and $y\left(t\right)$ denote time-series signals of joint features, $\overline{x}$ and $\overline{y}$ represent their mean values, and $\tau$ is the time lag. This normalized cross-correlation coefficient quantifies waveform similarity between two joint feature sequences.

For each resistance exercise, the skeletal joint feature time series extracted from the videos were resampled to a normalized length, and joint abnormalities were evaluated based on similarity to the average movement pattern of the non-disabled group. To reduce the influence of global body position changes, the joint coordinate time series for each arm were aligned to a common origin using the shoulder coordinates in the initial frame. The same translation defined by the shoulder displacement was applied to all remaining joint coordinates. The wrist joint was evaluated using wrist joint coordinates and wrist velocity, the elbow joint was evaluated using elbow joint coordinates and elbow joint angle, and the shoulder joint was evaluated using shoulder joint coordinates and shoulder joint angle as features to assess joint abnormalities.

The underlying assumption of this approach is that non-disabled participants are capable of performing the prescribed resistance exercises with coordinated movements of both arms, thereby providing a representative baseline of physiologically normal joint kinematics.

To construct the normal reference and ensure an unbiased evaluation, the non-disabled dataset was randomly divided into three disjoint subsets. Specifically, 55% of the non-disabled data were used to generate the normal reference patterns by averaging the joint feature time series for each exercise. For each participant, cross-correlation was computed between the individual joint feature time series and the corresponding normal reference pattern to quantify temporal movement similarity using Equation (5). At this stage, the correlation values obtained from multiple related features for each joint were averaged and used as a representative indicator to determine whether the joint exhibited abnormal behavior.

Because pattern similarity alone does not sufficiently reflect abnormalities in movement magnitude, amplitude features were additionally calculated for selected joint angles and positional features. Using an independent validation set comprising 40% of the non-disabled data, 99% confidence intervals (CI) were defined for both cross-correlation coefficients and amplitude features based on their empirical distributions [39,46,47,48]. This approach allows for inter-individual anatomical differences and natural movement variability while enabling effective detection of joint movements outside the normal range. Importantly, the validation subset consisted exclusively of non-disabled participants to verify that the derived normal reference and thresholds could generalize to unseen but physiologically normal movement patterns.

A joint feature was classified as normal when both the cross-correlation and amplitude values were within the defined normal range. If either condition was not satisfied, the joint feature was classified as abnormal. Joints not involved in the exercise were represented by gray circles, joints for which both the correlation results and amplitude were within the 99% CI were shown as green circles, and joints for which either the correlation results or the amplitude fell outside the 99% CI were shown as red circles, as illustrated in Figure 3. The remaining 5% of the non-disabled data, which were not used for reference construction or threshold determination, together with the entire brain lesion dataset, were used exclusively for testing. By applying the model to both unseen non-disabled participants and participants with brain lesions, this evaluation strategy enables assessment of whether deviations detected in patient data reflect abnormal movement patterns relative to a well-defined normal baseline, rather than overfitting to the training data. The abnormality rate for each arm and exercise was then computed using Equation (6).

$$\mathrm{J}\mathrm{o}\mathrm{i}\mathrm{n}\mathrm{t} \mathrm{A}\mathrm{b}\mathrm{n}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{t}\mathrm{y} \mathrm{R}\mathrm{a}\mathrm{t}\mathrm{e}\left[\%\right]={\displaystyle \frac{\mathrm{N}\mathrm{u}\mathrm{m}\mathrm{b}\mathrm{e}\mathrm{r} \mathrm{o}\mathrm{f} \mathrm{a}\mathrm{b}\mathrm{n}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{l} \mathrm{j}\mathrm{o}\mathrm{i}\mathrm{n}\mathrm{t}\mathrm{s} (\mathrm{p}\mathrm{e}\mathrm{r} \mathrm{a}\mathrm{r}\mathrm{m})}{\mathrm{T}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l} \mathrm{j}\mathrm{o}\mathrm{i}\mathrm{n}\mathrm{t}\mathrm{s} \mathrm{u}\mathrm{s}\mathrm{e}\mathrm{d} \mathrm{i}\mathrm{n} \mathrm{t}\mathrm{h}\mathrm{e} \mathrm{e}\mathrm{x}\mathrm{e}\mathrm{r}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{e} (\mathrm{p}\mathrm{e}\mathrm{r} \mathrm{a}\mathrm{r}\mathrm{m})}}\times 100$$

(6)

where abnormal joints are defined as joints whose correlation coefficients fall below the predefined threshold or whose reconstruction error or amplitude values exceed the predefined abnormality thresholds.

2.4.3. LSTM Autoencoder

To model temporal patterns of joint movements, a bidirectional long short-term memory (BiLSTM)-based autoencoder was employed, as illustrated in Figure 4. The input data consisted of multivariate time series with a frame length $T$ and $F$ joint features, represented in the form $T\times F$. To handle sequences of varying lengths, a masking layer was applied at the input stage. Joint coordinate time series were aligned to a common origin using the shoulder coordinates in the initial frame, consistent with the preprocessing used in the correlation analysis.

The encoder comprised three BiLSTM layers with 256, 128, and 64 units. The first two layers output the full sequence and applied batch normalization to improve training stability. The final BiLSTM layer compressed the sequence into a fixed-length latent representation, which was then passed through a 256-dimensional fully connected layer with ReLU activation to generate the latent vector. The decoder was designed symmetrically to the encoder. The latent vector was repeated along the temporal dimension and processed through BiLSTM layers with 64, 128, and 256 units to reconstruct the input time series. The final output was generated through a TimeDistributed fully connected layer that reconstructed joint features at each frame.

The model was trained in an unsupervised manner using mean squared error as the loss function. Training employed the Adam optimizer with a learning rate of $1\times {10}^{-4}$ and gradient clipping.

Of the non-disabled dataset, 55%, comprising 14 participants, was used for training, 40%, comprising 10 participants, for validation, and the remaining 5%, comprising 2 participants, along with all data from participants with brain lesions, comprising 12 participants, were used for testing. The same data partitioning strategy was consistently applied to both the correlation-based analysis and the LSTM Autoencoder-based analysis to ensure fair comparison between the two methods. Only non-disabled data were used during training and validation to ensure that the autoencoder learned a compact representation of normal bilateral joint coordination patterns without being influenced by pathological or compensatory movements. Thresholds for joint abnormality detection were defined as the 99% CI of joint-wise reconstruction error distributions from the validation data. Joint movement amplitudes were also computed from the raw time series, and values exceeding the 99% CI of the non-disabled group were classified as abnormal. This design reflects the assumption that movements performed by participants with brain lesions, including unilateral movement, reduced range of motion, and compensatory postures, represent deviations from normal motor control and should therefore be detected as anomalies relative to the learned normal model. For the correlation-based analysis, abnormality thresholds were likewise derived from the same non-disabled validation subset, and the held-out test data were evaluated identically in both approaches. The wrist joint was evaluated using wrist joint coordinates and wrist velocity, the elbow joint was evaluated using elbow joint coordinates and elbow joint angle, and the shoulder joint was evaluated using shoulder joint coordinates and shoulder joint angle as features to assess joint abnormalities. For each joint, the MSE values obtained from multiple related features were averaged and used as a representative metric to determine joint abnormality.

Joints not involved in the exercise were indicated by gray circles, joints with both MSE and amplitude within the 99% CI were shown as green circles, and joints in which either the MSE or the amplitude exceeded the 99% CI were shown as red circles, as illustrated in Figure 3.

Finally, a joint was classified as normal only when both the reconstruction error and amplitude conditions were satisfied. If either condition was violated, the joint was classified as abnormal. Arm-level abnormality rates for each exercise were then computed using Equation (6).

2.4.4. Fugl–Meyer Assessment (FMA)

In this study, the FMA was used to clinically evaluate upper-limb motor function in patients with brain lesions. The FMA is a standardized clinical assessment tool widely used to assess motor recovery in individuals with stroke and other central nervous system injuries and has been reported to have high reliability and validity.

The upper-limb section of the FMA has a maximum score of 66 points and provides a comprehensive evaluation of motor function and coordination of the upper limb including the shoulder, elbow, forearm, wrist, and hand. Each item is scored on a three-point ordinal scale with 0 indicating inability to perform, 1 indicating partial performance, and 2 indicating normal performance. Higher scores indicate better upper-limb function.

In this study, only the total upper-limb FMA score out of 66 points was used for analysis, and individual subdomain scores were not analyzed separately. The FMA evaluation was conducted prior to the experimental session by an experienced clinician following standardized procedures and served as a reference measure for validating the clinical relevance of the proposed video-based joint abnormality indices.

3. Results

3.1. Correlation Analysis

Figure 5 presents a comparison of the temporal changes in right wrist–related features between non-disabled participants and individuals with brain lesions for each exercise. The non-disabled participants generally exhibited repetitive and consistent movement patterns, whereas the participants with brain lesions showed movement patterns that differed from those of the non-disabled group. In particular, some participants with brain lesions demonstrated minimal or nearly flat feature variations due to little or no movement of one arm, which can be interpreted as reflecting restricted functional use of the affected limb.

In addition, for the wrist velocity feature, participants who performed the exercises normally, including both non-disabled individuals and some participants with brain lesions, exhibited velocity profiles close to a bell-shaped curve, indicating preserved point-to-point movement characteristics. The blue mean trajectory of the non-disabled group was used as a baseline reference for subsequent correlation analyses applied to both non-disabled and brain lesion groups.

Based on the correlation analysis, no joint abnormalities were detected in two non-disabled participants across all resistance exercises, including shoulder press, chest press, and arm curl. This result indicates that normal joint movement patterns were stably maintained in these participants.

For the arm curl exercise, the shoulder joint was excluded from the evaluation because it is not a primary joint involved, according to the exercise definition. This exclusion minimized false positives caused by abnormality judgments in joints unrelated to the exercise characteristics.

Figure 6 illustrates representative examples of the correlation-based joint abnormality assessment results. Figure 6a shows the results for a non-disabled participant selected from the held-out test set that was not included in the training or validation phases, serving as a reference example of normal joint movement patterns. Figure 6b presents the analysis results for a participant with a brain lesion whose affected side was the right side. The affected arm exhibited little to no voluntary movement during exercise, and therefore, all evaluated joints were classified as abnormal. In contrast, for the non-affected left arm, joint abnormality detection varied depending on the type of exercise. In particular, joint abnormalities were detected during the chest press exercise, unlike in the other exercises, and even within the same exercise, the abnormality judgment for the wrist joint differed across movement phases. These results indicate that joint abnormalities may be identified even in a moving arm when the exercise posture is inappropriate, as the joint-related feature waveforms deviate from the normal mean patterns. Due to space limitations, only one representative participant from each group is visualized, although the same analysis was consistently applied to all participants in the test set.

Figure 6c shows the joint trajectory changes observed during the actual exercise performance of the participants presented in Figure 6a,b. The blue curves represent the joint trajectories of the non-disabled participant, while the red curves represent those of the participant with a brain lesion. The non-disabled participant exhibited consistent joint trajectories without detectable abnormalities across all joints. For the participant with a brain lesion, whose affected side was the right side, little to no movement was observed in the joint trajectories of the right arm, which is consistent with the joint abnormalities detected on the affected side in Figure 6b.

For the shoulder press and arm curl exercises, the joints on the non-affected side exhibited trajectory patterns similar to those of the non-disabled participant, and no joint abnormalities were detected. In contrast, during the chest press exercise, joint trajectories that differed from the normal mean pattern were observed not only in the affected side but also in some joints on the non-affected side, leading to the detection of joint abnormalities. Notably, for the participant with a brain lesion, greater wrist displacement was observed in the 2D plane during the backward phase of the movement compared to the forward-reaching phase. As a result, even within the same exercise, the wrist joint abnormality assessment differed depending on the movement direction. These findings indicate that joint abnormalities may be detected even when movement is present, particularly when exercise posture or inter-joint coordination deviates from the normal pattern.

For each arm, the joint abnormality rates calculated for individual exercises were averaged to obtain an overall arm-level abnormality rate. For each arm, the joint abnormality rates calculated for individual exercises were averaged to obtain an overall arm-level abnormality rate. The results for each group and arm are summarized in

Table 3. Abnormality rates of the left and right arms for each group were obtained using the correlation-based method and the LSTM autoencoder (mean ± standard deviation).

Method	Non-Disabled Group (Left Arm) [%]	Non-Disabled Group (Right Arm) [%]	Brain Lesion Group (Left Arm) [%]	Brain Lesion Group (Right Arm) [%]
Correlation (mean $\pm$ std)	$0 \pm 0$	$0 \pm 0$	$32.4 \pm 15.1$	$50.0 \pm 24.1$
LSTM Autoencoder (mean $\pm$ std)	$0 \pm 0$	$0 \pm 0$	$33.8 \pm 13.1$	$50.5 \pm 33.4$

. In the non-disabled group, the abnormality rate was $0 \pm 0$% for both arms, indicating that all evaluated joint movements remained within the predefined normal range across exercises.

In contrast, the group with brain lesions exhibited markedly higher abnormality rates, with 50.0 ± 24.1% for the right arm and 32.4 ± 15.1% for the left arm. This increase reflects the fact that participants with brain lesions often showed unilateral movement, reduced range of motion, or movement patterns that deviated from those observed in the non-disabled reference group. Consequently, even when voluntary arm movement was present, deviations in temporal coordination or joint usage led to higher abnormality rates compared with the non-disabled group.

The arm with the higher abnormality rate was identified as the affected side. As a result, the affected side identified by the correlation-based analysis matched the clinically reported affected side in 75.0% of all participants. Figure 7a presents the confusion matrix for affected side classification. These findings suggest that the correlation-based approach provides a reasonable level of validity for affected-side classification.

The correlation-based approach was included as an interpretable baseline method to assess whether the joint abnormality patterns identified by the LSTM autoencoder were clinically meaningful. While correlation analysis evaluates linear relationships between joint trajectories, the LSTM autoencoder models temporal continuity and inter-joint interactions in a data-driven manner.

3.2. LSTM Autoencoder Analysis

Using the LSTM autoencoder-based model, joint abnormality status was analyzed for each arm. In non-disabled participants, reconstruction errors for all evaluated joints remained within the 99% CI and were therefore classified as normal. This result indicates that the proposed model reliably learned the temporal patterns of normal joint movements. Figure 8a illustrates representative results from one non-disabled participant selected from the held-out test subset, which was not included in either the training or validation phases. All joints remained within the normal reference range, confirming the generalizability of the learned normal movement patterns to unseen non-disabled data.

In contrast, participants with brain lesions showed increased reconstruction errors exceeding the 99% CI in specific joints during certain exercises and movement phases. These joints largely corresponded to regions identified as functionally impaired by the correlation-based analysis. In particular, Figure 6a,b, together with Figure 8a,b, present the results obtained from the same representative participant, and demonstrate that both analysis methods consistently identified the same joints as impaired, supporting the reliability of the proposed approach. Figure 6c further visualizes the actual joint coordinate trajectories of this participant during exercise execution, providing a direct correspondence between the quantified abnormality results and the underlying movement patterns.

Figure 8b presents representative results from one participant with a brain lesion selected from the patient group, whose affected side was the right arm. The affected right arm exhibited minimal movement during exercise, resulting in reconstruction errors exceeding the normal reference range for most joints, and was therefore classified as having overall joint dysfunction. In contrast, the unaffected left arm showed varying joint abnormality results depending on the type of exercise. In particular, joint abnormalities were detected during the chest press exercise, unlike in other exercises, where increased reconstruction errors were observed in certain joints. Moreover, even within the same exercise, the abnormality assessment of the wrist joint differed across movement phases. These findings indicate that, even when arm movement is present, improper posture alignment or joint coordination during exercise can cause the temporal patterns of joint-related features to deviate from those learned by the LSTM Autoencoder from normal data, leading to the classification of joint abnormalities.

When joint movement was severely restricted, not only did reconstruction errors increase, but motion-based metrics also indicated movements below the defined thresholds. This finding suggests that the proposed framework can distinguish between general joint abnormalities and abnormalities accompanied by actual movement reduction. Consistent with the correlation-based approach, arm-level joint abnormality rates were computed, and the arm with the higher abnormality rate was classified as the affected side. In the non-disabled group, the arm-level joint abnormality rate obtained from the LSTM Autoencoder was 0 ± 0% for both arms, indicating consistently normal movement patterns. In contrast, participants with brain lesions exhibited substantially higher abnormality rates, with the clinically affected right arm showing an average abnormality rate of 50.5 ± 33.4%, and the contralateral left arm showing 33.8 ± 13.1%. This method correctly identified the clinically reported affected side in 83.3% of all participants. Figure 7b presents the confusion matrix for affected side classification. Due to space limitations, only one representative participant from each group is visualized in Figure 8a,b, although the same analysis procedure was applied to all test-set non-disabled participants and to all participants with brain lesions.

Additionally, non-parametric statistical analyses were conducted to quantitatively evaluate differences in joint abnormality rates between the test-set non-disabled group (n = 2) and the brain lesion group (n = 12). Group comparisons were performed using the Mann–Whitney U test. The results showed that, in the correlation-based approach, joint abnormality rates in the brain lesion group were significantly higher than those in the non-disabled group for both the right arm ($p=0.0354$) and the left arm ($p=0.0330$). Similarly, in the LSTM autoencoder-based approach, significant group differences were observed for both the right arm ($p=0.0341$) and the left arm ($p=0.0343$). Effect size analysis revealed Cliff’s delta values of 1.000 for all comparisons.

To assess the consistency between the two analysis methods, joint abnormality rates obtained from the correlation-based and LSTM autoencoder-based approaches were compared for each arm. The Wilcoxon signed-rank test indicated no significant differences between the two methods for either the right arm ($p=0.9697$) or the left arm ($p=0.5936$). In contrast, Spearman correlation analysis revealed a strong positive correlation between the two methods for the right arm ($\rho =0.910$, $p<0.001$) and a significant positive correlation for the left arm ($\rho =0.637$, $p=0.0260$).

Furthermore, abnormality rates were compared across the three resistance exercises to identify and prioritize exercises in which joint abnormalities occurred more frequently. A higher abnormality rate indicates greater difficulty in joint control or muscle coordination during the corresponding exercise, thereby highlighting functional domains that require targeted intervention.

Based on this prioritization, rehabilitation exercise recommendations were formulated using exercise- and joint-specific evidence. Resistance and strength training have been shown to significantly improve upper-limb strength and functional performance in individuals with stroke [49,50]. Accordingly, exercises were selected to strengthen the muscle groups primarily involved in movements with higher abnormality rates. For example, when a high abnormality rate was observed during the shoulder press, exercises focusing on shoulder flexion and abduction strength, as well as neuromuscular control of the shoulder, such as assisted shoulder elevation and circular movements, were recommended [51]. Similarly, abnormalities during the chest press led to the recommendation of exercises targeting shoulder flexors and elbow extensors, while abnormalities during the arm curl prompted exercises aimed at elbow flexion and biceps activation [52].

3.3. Compare Fugl Meyer Assessment

Joint abnormality rates derived from the correlation analysis and the LSTM autoencoder were compared with the clinical assessment metric FMA. The FMA is an upper-limb motor function scale with a maximum score of 66, where higher scores indicate less functional impairment.

The correlation coefficient between the joint abnormality rate obtained from the correlation-based analysis and the FMA score was $-0.7789$, indicating a strong negative relationship between the two variables. The p-value was $2.83\times {10}^{-3}$, demonstrating statistical significance, as illustrated in Figure 9a. This result suggests that lower FMA scores are associated with higher joint abnormality rates.

In contrast, the joint abnormality rate derived from the LSTM autoencoder showed a stronger negative correlation with the FMA score, with a correlation coefficient of $-0.8454$. The p-value was $5.33\times {10}^{-4}$, reflecting high statistical significance, as illustrated in Figure 9b. These findings indicate that the LSTM autoencoder-based approach exhibits better agreement with clinical assessment metrics than simple time-series correlation analysis.

Overall, a consistent trend was observed in which higher FMA scores corresponded to lower joint abnormality rates. This trend reflects that individuals with less upper-limb functional impairment maintain more normal temporal patterns of joint movement. In particular, the LSTM autoencoder effectively captured the nonlinear and temporal characteristics of joint motion, resulting in a higher level of concordance with clinical evaluation outcomes.

4. Discussion

4.1. Results of Correlation and LSTM Autoencoder

In this study, joint abnormalities during resistance exercises were evaluated using correlation analysis and an LSTM autoencoder-based anomaly detection model. The correlation-based analysis detected no joint abnormalities in non-disabled participants across all resistance exercises, indicating stable joint coordination and normal movement patterns. In contrast, participants with brain lesions showed increased joint abnormality rates during specific exercises and movement phases. These abnormalities were generally consistent with clinically reported regions of joint functional impairment.

A similar trend was observed in the LSTM autoencoder-based analysis. The model trained on normal data reliably reconstructed joint movements of non-disabled participants, with reconstruction errors remaining within the 99% CI. In participants with brain lesions, however, reconstruction errors for certain joints exceeded the CI during specific exercises and phases, and these joints largely corresponded to those identified as impaired by the correlation-based analysis. These results suggest that the unsupervised LSTM autoencoder effectively learns temporal patterns of normal joint movements and sensitively detects abnormal motion.

In some participants with brain lesions, joint range of motion was relatively preserved, yet abnormalities were detected in specific joints due to improper posture during exercise execution. This finding indicates that joint abnormality assessment is influenced not only by movement magnitude but also by coordination and postural alignment during task performance. Additionally, discrepancies between correlation-based and LSTM autoencoder-based results were observed for certain joints. These differences likely occurred when metric values were near the decision threshold, leading to different normal or abnormal classifications. Such issues may be mitigated by introducing a margin around the threshold or by expanding the normal dataset to establish more stable reference distributions.

The results of the non-parametric statistical analyses quantitatively support that joint abnormality rates derived from both analysis methods can clearly discriminate between the non-disabled group and the brain lesion group. In particular, the very large effect size (Cliff’s delta = 1.000) indicates that, despite the limited sample size, the distributions of joint abnormality rates are distinctly separated between the two groups. This finding suggests that the normal reference defined in this study sensitively captures abnormal movement patterns exhibited by participants with brain lesions.

In addition, no significant differences were observed between the correlation-based approach and the LSTM autoencoder-based approach, while joint abnormality rates obtained from the two methods showed strong positive correlations. This indicates that, although the two approaches are based on different analytical principles, they consistently capture patterns of joint functional impairment, thereby supporting the consistency and reliability of the proposed framework. The classification discrepancies observed in some joints are likely attributable to differences in sensitivity near the decision threshold, which may be mitigated in future work by expanding the normal dataset or introducing a margin around the threshold.

4.2. Clinical Relevance Through Comparison with FMA

In this study, joint abnormality rates derived from video-based joint motion analysis were obtained using both correlation analysis and an LSTM autoencoder and were compared with the clinical assessment metric FMA. The results showed that both methods exhibited strong negative correlations with FMA scores. This finding is consistent with clinical interpretations indicating that individuals with less upper-limb functional impairment tend to maintain more normal joint movement patterns.

Notably, the abnormality rate derived from the LSTM autoencoder showed higher correlation coefficients and larger effect sizes than those obtained from the correlation-based analysis. This difference can be attributed to the distinct analytical approaches of the two methods. While correlation analysis primarily evaluates linear relationships between individual joints or specific features, the LSTM autoencoder simultaneously considers inter-joint interactions and temporal continuity, enabling modeling of complex movement patterns across the entire exercise sequence. As a result, the LSTM autoencoder can more sensitively capture abnormal movement characteristics that are difficult to detect at the level of individual joints or single time points, leading to improved agreement with clinical assessment metrics.

The large effect sizes observed in this study further indicate that the proposed joint abnormality rate reflects clinically meaningful differences rather than merely statistical significance. This suggests that quantitative metrics derived from video-based analysis have the potential to effectively explain variations in clinical functional status.

In addition, joint abnormality rates were compared across exercises to prioritize movements with higher abnormality levels, and these priorities were used to recommend rehabilitation exercises. This demonstrates that the proposed framework can be extended beyond anomaly detection to support personalized rehabilitation strategies based on individual functional levels.

From a practical perspective, the proposed approach has potential applications in home-based and remote rehabilitation scenarios, particularly for individuals in the chronic stage of brain lesions who require continuous functional monitoring. By enabling objective assessment of joint movement quality using only exercise videos captured with a standard camera, the framework may serve as a non-invasive and cost-effective tool to support remote rehabilitation monitoring, self-assessment, and preliminary screening without direct expert supervision.

4.3. Limitation and Future Work

This study has several limitations that should be considered when interpreting the results.

First, for the chest press and arm curl exercises, motion data were acquired using a two-dimensional RGB camera, which does not provide depth information. Consequently, when upper-limb joints were aligned with the camera viewpoint, errors in joint angle estimation could occur, with elbow angles occasionally being misestimated as 0° or 180° despite the presence of actual joint flexion. To mitigate this limitation, joint abnormality detection was performed relative to motion patterns learned from non-disabled participants, incorporating both joint angle trajectories and joint coordinate movements. This design helped minimize the impact of 2D projection errors on abnormality detection. Nevertheless, the lack of depth information contributed to the expansion of anomaly detection thresholds for certain joints and led to occasional discrepancies between correlation-based and LSTM autoencoder-based results in specific exercise phases.

Second, the normal reference dataset consisted of 26 non-disabled participants who exhibited variability in physical characteristics and exercise execution styles. This inter-individual variability influenced threshold determination and may have contributed to discrepancies between the outcomes of the correlation-based analysis and the LSTM autoencoder for certain joints. Although the imbalance between the non-disabled and brain lesion groups was justified through power analysis, the relatively small number of participants with brain lesions limits the ability to fully capture the diversity of impairment severity and movement patterns observed in real-world clinical populations. Given the wide heterogeneity of motor dysfunction among individuals with brain lesions, this constraint remains a notable limitation of the present study. Nevertheless, collecting large-scale patient data in rehabilitation research is inherently challenging due to clinical and logistical constraints. Future work will therefore focus on expanding the dataset of participants with brain lesions to better represent varying levels of motor impairment and to improve the robustness and generalizability of the proposed joint abnormality assessment framework.

Third, the proposed method was evaluated using only three resistance exercises (chest press, shoulder press, and arm curl), which were intentionally selected due to their simplicity and safety. The ultimate goal of this study is to support home-based rehabilitation for individuals with brain lesions using minimal equipment, enabling safe, accessible, and sustainable exercise without the need for frequent visits to rehabilitation centers. To accommodate individuals with limited lower-limb mobility and to reduce injury risk, exercises were designed to be performed in a seated position and to be easily followed by both non-disabled and disabled participants. While this design choice facilitated reliable baseline acquisition and safe participation, it limits the generalizability of the proposed framework to more complex or multi-joint movements. Future work will extend the proposed approach to a broader range of exercises that remain easy to follow while engaging multiple muscle groups and functional movement patterns.

Future work will focus on acquiring three-dimensional joint data that incorporate depth information by using RGB-D cameras, thereby improving the accuracy of joint angle estimation. At the same time, the normal dataset will be expanded to establish more robust reference distributions, which is expected to reduce variability due to individual differences and enhance the robustness of the proposed model. Through these improvements, the proposed approach can be extended toward a more reliable joint abnormality assessment and rehabilitation support system.

5. Conclusions

This study proposed a method for quantitatively evaluating upper-limb joint abnormalities during resistance exercises using video-based skeletal data. To this end, waveform similarity-based correlation analysis and an unsupervised LSTM autoencoder model that learns temporal patterns of joint motion were applied. Joint abnormality rates derived from both methods were then compared with the clinical assessment metric FMA.

The results showed that non-disabled participants maintained normal joint movement patterns across all resistance exercises, and no joint abnormalities were detected by either method. In contrast, participants with brain lesions exhibited increased joint abnormality rates during specific exercises and movement phases, which were highly consistent with existing clinical observations. In particular, the LSTM autoencoder-based analysis, which accounts for inter-joint interactions and temporal continuity, demonstrated higher correlation coefficients and larger effect sizes with clinical metrics than the correlation-based approach, indicating its ability to more precisely capture complex abnormal movement patterns.

Furthermore, a clear negative correlation was observed between joint abnormality rates and FMA scores, with higher FMA scores corresponding to lower abnormality rates. This finding suggests that the proposed video-based analysis provides a quantitative indicator that reflects actual clinical functional levels rather than merely statistical differences. In addition, rehabilitation priorities were derived based on exercise-specific abnormality rates, and personalized rehabilitation exercise recommendations were generated, demonstrating the potential of the proposed framework to support individualized rehabilitation strategies.

Because the proposed approach enables assessment of joint function using only videos recorded in home environments without direct expert intervention, it offers strong potential as a non-invasive and cost-effective rehabilitation evaluation tool. With future integration of three-dimensional joint analysis using RGB-D cameras and expansion of normal reference data to enhance model robustness, the proposed method is expected to be extendable to remote rehabilitation monitoring systems and self-assessment tools.