Arm Swing Asymmetry Measurement from 2D Gait Videos

Abstract

Arm swing during gait has been positively related to gait stability and gait efficiency, particularly in the presence of neurological disorders that affect locomotion. However, most gait studies have focused on lower extremities, while arm swing usually remains ignored. In addition, these studies are mostly based on costly, highly-specialized vision systems or on wearable devices which, despite their popularity among researchers and specialists, are still relatively uncommon for the general population. This work proposes a way of estimating arm swing asymmetry from a single 2D gait video. First, two silhouette-based representations that separately capture motion data from both arms were built. Second, a measure to quantify arm swing energy from such a representation was introduced, producing two side-dependent motion measurements. Third, an arm swing asymmetry index was obtained. The method was validated on two public datasets, one with 68 healthy subjects walking normally and one with 10 healthy subjects simulating different styles of arm swing asymmetry. The validity of the asymmetry index at capturing different arm swing patterns was assessed by two non-parametric tests: the Mann–Whitney U test and the Wilcoxon signed-rank test. The so-called physiological asymmetry was observed on the normal gait sequences of both datasets in a statistically similar way. The asymmetry index was able to fairly characterize the different levels of asymmetry simulated in the second set. Results show that it is possible to estimate the arm swing asymmetry from a single 2D gait video, with enough sensitivity to discriminate anomalous patterns from normality. This opens the door to low-cost easy-to-use mobile applications to assist clinicians in monitoring gait condition in primary care (e.g., in the elderly), when more accurate and specialized technologies are often not available.

1. Introduction

Arm swing (AS) has been positively related to the recovery of gait stability after a perturbation [1], to gait efficiency [2,3] and, in toddlers and children with cerebral palsy, to gait balance [4,5]. The value of the arm movements as an outward sign of health condition has also been well established [6,7,8,9], while rehabilitation studies [10,11] have suggested that exercises aimed at normalizing AS in patients with neurological disorders contribute to improve the inter-limb coordination and the locomotion pattern. However, in spite of such evidence, most recent gait studies still ignore upper limbs [8,11].

Much of the efforts aimed at measuring AS have been focused on asymmetry, a term that denotes the amount of discrepancy between the left and the right side during gait. AS amplitude has proven to vary significantly between the left and right side both in patients with neurodegenerative diseases [6,9,12,13] and, to a lesser extent, in healthy people [14,15]. As observed in [6], AS adapts to walking velocity both in patients of Parkinson’s disease (PD) and in control subjects, while arm swing asymmetry remains relatively unaffected by changes in walking conditions. In addition, significant differences in AS asymmetry were found between early PD and the control group, concluding that arm swing asymmetry can be more reliable in the diagnosis and monitoring of PD than arm swing amplitude. The value of AS asymmetry to characterize the unilateral dynamic of upper limb in early PD was also observed in [9], resulting significantly higher in patients than in the control group. They also provide evidence in favor of the robustness of AS asymmetry to changes in walking speed.

In the context of healthy people, [14] stated that “a certain degree of asymmetry is physiological”, and it is not related to handedness. Reproducible asymmetries between the left and right AS were measured in normal subjects. These findings were later confirmed by [15] when measuring AS asymmetry in overground walking. Motion data consisted of the range of motion (in degrees) measured by IMUs attached to wrists. They concluded that left-dominant AS is typical in healthy gait, regardless of walking condition or handedness.

In most previous studies, kinematic data were collected by tracking reflective markers placed at anatomical locations by a multi-camera 3D motion capture system such as Vicon [6,16] and Qualisys [9,14], which are very accurate solutions, but also costly and highly specialized. In the case of wearable gadgets (e.g., markers, IMU, etc.), despite their simplicity, precision, and growing popularity among researchers and clinical specialists, they are generally scarce in non-specialized healthcare centers as compared to, for example, smartphones, which have become virtually ubiquitous.

Along with these approaches, simple vision devices such as RGB or RGB-D cameras have been gaining momentum in rehabilitation [17,18], gait analysis [13,19,20,21] and parkinsonian gait assessment [22]. Main reasons include their low cost, user-friendliness, ubiquity, and ability for remote sensing. Recently, [23] showed that the Kinect error can be lower than $5^{\circ }$ as compared to the Vicon system, supporting its feasibility for certain clinical applications. Besides, since gait studies are conducted in a cooperative setting, factors that might alter gait perception such as subject appearance (e.g., clothing) and recording conditions (e.g., scene, lighting) can be easily kept under control. As it is well known, the ability of an RGB camera sensor to capture or perceive the actual characteristics of a subject’s gait pattern (here referred to as gait perception) could be severely limited by wearing loose clothing, by carrying backpacks, by poor lighting conditions, by the presence of objects in the scene between the optical sensor and the subject, by a changing background, by the sensor viewpoint, etc. Fortunately, a fully cooperative setting can be assumed in a vision-based analysis of gait disorders in healthcare environments, where all these factors can be avoided or kept within certain established limits. For example, we can expect simple indoor scenarios, fixed backgrounds, stable illumination conditions, tight clothes, no accessories, smooth floor, etc.

Previous efforts in estimating gait symmetry from a markerless 2D video can be found in [20,21]. In [20], measurements were focused on the legs, while in [21], a symmetry index was obtained between amounts of movement calculated separately from half-cycle representations of the human gait. Such representations comprise information from all four limbs of the body together, making it difficult to attribute a measured anomaly to a particular limb or a body half (according to the sagittal plane). Unlike the aforesaid efforts, this work proposes the computation of a measure of asymmetry between representations that condense the energy of each arm separately obtained throughout the entire gait cycle. In this way, the measurements can potentially discriminate anomalies present in the swing of a particular arm.

More formally, this paper introduces a method for measuring AS asymmetry from only a 2D gait video. Contributions include (1) a side-dependent silhouette-based gait representation, and (2) a method for quantifying AS energy from such a representation. Arm swing asymmetry is then computed between both side-dependent energy measurements. The method is independent of recording parameters, making it valid for cross-platform analysis. Experiments were performed on two gait video databases, one with healthy subjects walking normally and one with healthy subjects simulating different styles of AS asymmetry. Results were validated by two non-parametric statistical tests. Since the method requires only a 2D video, it brings transfer opportunities for low-cost mobile systems to assist clinicians in monitoring gait parameters when more accurate and specialized technologies or trained staffs are not available.

2. Materials and Methods

2.1. Datasets

This study involves two public databases, the OU-ISIR Treadmill Dataset B [24] and the INIT Gait Database [20]. The former is a general-purpose gait video database that comprises indoor recordings of 68 healthy subjects, wearing up to 32 clothing styles. Sequences that combine regular pants and full shirt were chosen because of their neutral outline. This database is intended to establish an asymmetry benchmark in healthy walking from the new measure’s perspective. The second database is also composed of indoor sequences of high-quality binary silhouettes of 10 healthy volunteers walking normally ($nm$ style) and simulating several abnormal gait styles. Four of them reproduce AS asymmetry by combining the two arms and two reduced ranges of motion:

$r_0:$

The left arm swings normally, while the right arm is volitionally held still.

$l_0:$

The right arm swings normally, while the left arm is volitionally held still.

$r_{0.5}:$

The left arm swings normally, while the right arm swing is incomplete.

$l_{0.5}:$

The right arm swings normally, while the left arm swing is incomplete.

Instructions for $r_0$ and $l_0$ styles were to hold the affected arm relaxed, just next to the trunk, without swinging, while for $r_{0.5}$ and $l_{0.5}$, participants were asked to perform half swing with the affected arm. Although exactly half swing can not be assumed, it can be reasonably accepted as a working hypothesis an incomplete trajectory of the affected arm. Figure 1 illustrates styles $r_0$ and $r_{0.5}$ as opposed to $nm$.

2.2. Arm Swing Asymmetry Measurement

The proposed method consists of 3 steps:

• Two side-based Gait Energy Images (GEI) [25] are built, so that each one comprises the motion data of each arm separately. A region of interest (ROI) containing arm motion is automatically extracted from each side-based representation.
• A Perceptible Motion Index (PMI) is introduced to quantify the amount of perceivable arm motion from each side-based ROI.
• A modified Robinson index [15,26] is used to measure AS asymmetry.

2.2.1. Side-Based Gait Energy Images

The Gait Energy Image (GEI) is a well-known silhouette-based gait representation that summarizes a subject’s dynamic and appearance. Given a sequence of aligned, size-normalized silhouettes, the GEI is computed by averaging all silhouettes. In this work, silhouettes were preprocessed as in [20]. Figure 2a shows a GEI example, where three types of GEI pixels can be identified:

White pixels.

They capture body regions with zero or little relative motion with respect to image borders, e.g., head and torso. They encode appearance.

Gray pixels.

They capture most of the gait energy mainly caused by the cyclic movement of limbs. A gray value means that the pixel has sometimes been background (black in some gait frames) and sometimes silhouette (white in other frames). The more intermediate the gray value, the greater the balance between background and silhouette and, therefore, the greater the movement recorded.

Black pixels.

They capture background regions common to all silhouettes.

Typically, a GEI is obtained from full gait cycles, summarizing the whole gait energy in a single representation. In order to isolate the motion of each arm, a side-based GEI is built as follows:

• A sequence of silhouettes is split into segments delimited by midstance/midswing poses (Figure 2a). Each resulting segment covers half a cycle.
• Segment-based GEIs are computed (Figure 2b). The way of splitting the sequence leads to segment-based GEIs in which each arm is captured either in the front side or in the back side (according to the coronal/frontal plane). This dynamic alternates along the segments: odd segments reflect an arm arrangement; the even ones, the opposite.
• Odd and even segment-based GEI are averaged separately (see Figure 2c). The two resulting GEIs are referred to as GEI${}_A$ and GEI${}_B$, respectively. Due to the gait cyclic nature, the arm captured in the front side of GEI${}_A$ coincides with the one in the back side of GEI${}_B$. Similarly, the motion in the back side of GEI${}_A$ and in the front side of GEI${}_B$ correspond to the other arm. Legs are also segregated.
• Two side-based GEIs are built by binding together the front side of GEI${}_A$ and the back side of GEI${}_B$ and, contrarily, the back of GEI${}_A$ and the front of GEI${}_B$ (Figure 2d). Let us denote the resulted side-based representations as GEI${}_{AB}$ and GEI${}_{BA}$. Note that each of these representations condenses the movement of a single arm. For the purpose of estimating AS asymmetry, the correspondence between GEI${}_{AB}$/GEI${}_{BA}$ and the left/right arms is irrelevant.

Two ROIs enclosing arm motion are then extracted from GEI${}_{AB}$ and GEI${}_{BA}$, following a popular body proportion canon: the head to body ratio is one to eight for an adult. The ROI is established from 2.5 to 4.5 heads (Figure 2e). This heuristic rule proved very effective in isolating the energy information captured from the movement of the arms.

2.2.2. Perceptible Motion Index

The Perceptible Motion Index (PMI) is intended to quantify the motion information accumulated in a ROI, which is mostly due to the related arm. The term perceptible considers that there is a portion of AS that keeps hidden by the trunk silhouette which cannot be measured. A compensation factor is introduced to weight every bit of perceptible motion directly proportional to its distance to the trunk. PMI is defined below:

$$PMI=\sum _{xy}w\left(x\right)e(x,y)=\sum _{x}w\left(x\right)\sum _{y}e(x,y),$$

(1)

where $w\left(x\right)$ is a weighting function that grows with the distance to the trunk (depends only on the x-coordinate), and $e(x,y)$ accounts for the energy of the arm motion at pixel $(x,y)$, and it is computed as $e(x,y)=1-\frac{\left|g(x,y)-127.5\right|}{127.5}$ with $g(x,y)$ being the gray level at that ROI pixel [20]. The maximum of $e(x,y)=1$ is reached at $g(x,y)=127.5$, when the pixel $(x,y)$ has been half times background and the other half silhouette; its minimum $e(x,y)=0$ occurs when $(x,y)$ has only been either background (0) or silhouette (255).

The proposed weighting function $w\left(x\right)$ is shown in Figure 3. Note that $w\left(x\right)$ is zero in the x-range imputed to the trunk, and it grows as it moves away from the center. That is, $w\left(x\right)$ cancels any contribution coming from the trunk region (where arm motion remains undetected), and keeps motion information found beyond the trunk. It is formally defined as:

$$w\left(x\right)=max\{s\left(k·(x-x_{back})\right),1-s\left(k·(x-x_{front})\right)\},$$

(2)

where $s\left(t\right)=1/(1+e^{-t})$ is the sigmoid function, $k>0$ introduces a horizontal stretching ($0<k<1$) or shrinking ($k>1$) of $s\left(t\right)$, and $x_{back}$ and $x_{front}$ cause horizontal shifts of the inflection points at which $w\left(x\right)$ accelerates. This notation assumes the subject walks from right to left in the scene. Finally, $x_{back}$ and $x_{front}$ are estimated as the minimum and the maximum of the first derivative of the probability density function of foreground data along the x-axis. This process is illustrated in Figure 3 and formally stated below:

$$\begin{array}{cc}\hfill & x_{back}= arg\; min{f}^{\prime }\left(x\right)\hfill \\ \hfill & x_{front}= arg\; max{f}^{\prime }\left(x\right)\hfill \\ \hfill & f^{\prime }\left(x\right)=f(x+\Delta x)-f\left(x\right)\hfill \\ \hfill & f\left(x\right)=\frac{{\sum }_{\mathrm{y}}g(x,\mathrm{y})}{{\sum }_{\mathrm{x}}{\sum }_{\mathrm{y}}g(\mathrm{x},\mathrm{y})},\mathrm{with}g(\mathrm{x},\mathrm{y})\mathrm{being}\mathrm{the}\mathrm{gray}\mathrm{map}\mathrm{of}\mathrm{the}\mathrm{ROI}.\hfill \end{array}$$

(3)

Experiments were designed in terms of $k=1$ (standard sigmoid function).

2.2.3. Modified Robinson Index

Let $m_{AB}$ and $m_{BA}$ be the PMI values computed on ROIs extracted from GEI${}_{AB}$ and GEI${}_{BA}$, respectively. The following version of the Robinson index is used to measure AS asymmetry (ASA):

$$ASA=\frac{\left|m_{AB}-m_{BA}\right|}{max\{m_{AB},m_{BA}\}}·100.$$

(4)

According to [27], ASA measures the disagreement between two measurements made from discrete time events. However, unlike conventional discrete approaches, Equation (4) summarizes a spatio-temporal disagreement over the entire gait cycle.

Because the method adapts to the subject anatomy and relies on the normalized functions (3) and (4), it can be considered reasonably independent of acquisition conditions, walking speed, and spatio-temporal resolution.

3. Results

Figure 4 shows the ASA distributions from OU-ISIR and from the five INIT AS styles. Similar distribution patterns can be found in the two independent groups of normal gait, OU-ISIR and $nm$, with medians $25.5\%$ and $19.6\%$, respectively. The distribution shapes of $l_{0.5}$ and $r_{0.5}$ styles (one arm partially swings) were also similar with medians $44.9\%$ and $56.9\%$, respectively, as well as the shapes of $l_0$ and $r_0$ styles (one arm remains motionless) with medians $70.5\%$ and $75.8\%$, respectively. Despite the big differences between the three levels of motion, one might expect $l_0$ and $r_0$ results closer to 1. However, marginal movement registered in the trunk contour affects the perceived asymmetry.

Considering the presence of outliers and the large differences in variances, the non-parametric Mann–Whitney U test [28] was performed to determine whether two groups of ASA measurements can be considered the result of the same data generating process (null hypothesis $H_0$). Besides, to reduce the impact of subject variability, the Wilcoxon signed-rank test [29] was also applied to paired mean performances of subjects in the INIT database. Both tests were carried out using the implementations provided by the module scipy.stats from the Python library SciPy.

Table 1. Results of non-parametric tests between the column and the row groups in terms of Mann–Whitney p-value/Wilcoxon p-value. Bold p-values denote $H_0$ rejection at $\alpha =0.05$ (differences are significant). Symbol ‘–’ denotes a meaningless comparison.

	OU-ISIR	INIT Database
		$\mathit{nm}$	${\mathit{l}}_{\mathbf{0}.\mathbf{5}}$	${\mathit{l}}_{\mathbf{0}}$	${\mathit{r}}_{\mathbf{0}.\mathbf{5}}$
$nm$	0.15/−
$l_{0.5}$	0.006/−	0.006/0.047
$l_0$	$\mathbf{6}\times {\mathbf{10}}^{-\mathbf{9}}$/−	$\mathbf{5}\times {\mathbf{10}}^{-\mathbf{7}}$/0.005	0.006/0.012
$r_{0.5}$	$\mathbf{2}\times {\mathbf{10}}^{-\mathbf{5}}$/−	$\mathbf{1}\times {\mathbf{10}}^{-\mathbf{4}}$/0.012	0.310/0.200	−
$r_0$	$\mathbf{5}\times {\mathbf{10}}^{-\mathbf{11}}$/−	$\mathbf{7}\times {\mathbf{10}}^{-\mathbf{8}}$/0.005	−	0.013/0.012	$\mathbf{5}\times {\mathbf{10}}^{-\mathbf{5}}$/0.009

shows p-values of both tests, stressing $H_0$ rejections at a significance level of $\alpha =0.05$.

The main findings can be summarized as follows: (1) differences between the two independent groups of normal gait (OU-ISIR and $nm$) were not significant; (2) differences between each pair of styles with different motion patterns (OU-ISIR/$nm$ vs. $l_{\ast }$, OU-ISIR/$nm$ vs. $r_{\ast }$, $l_{0.5}$ vs. $l_0$, $r_{0.5}$ vs. $r_0$) were statistically significant in both tests; (3) differences between $l_{0.5}$ and $r_{0.5}$ styles were not significant in either of the two tests; (4) differences between $l_0$ and $r_0$ styles were significant in both tests. The first three results agree with expectations. That is, the ASA measure was able to establish statistical similarity between independent groups of normal gait, as well as to statistically differentiate the three levels of AS asymmetry, in most cases with very small p-values. The fourth result could be due to the left arm preference and the greater visibility of the left arm in this experiment. Both arguments are discussed below.

4. Discussion

As a first general observation, the method was able to statistically differentiate each AS asymmetry pattern from the rest, while found statistical equivalence between asymmetry measurements of the two normal gait databases acquired under very different conditions. As stated in [9,14,15], our ASA measure on healthy gait also showed a small but clear degree of asymmetry. This should be consider when assessing gait disorders.

Although a direct comparison with previous works is meaningless (they involve different data and acquisition technologies), some insight could be drawn from contrasting means and standard deviations when ASA is measured, as in (4). For example, refs. [14,15] reported mean asymmetries of $25.9\pm 24.0$ and $39.5\pm 21.8$ from healthy young and older people, respectively. These results are highly consistent with $28.3\pm 18.01$ and $21.9\pm 15.9$ obtained from the OU-ISIR and the INIT $nm$ styles, respectively.

Results on the INIT database also showed a little more asymmetry when the left arm moved freely and the right one had motion restrictions ($r_0$, $r_{0.5}$) than in their opposite settings ($l_0$, $l_{0.5}$). On the one hand, this result is consistent with the left arm preference found in [14,15] on healthy subjects who were mostly right-handed. On the other hand, the walking direction in the INIT gait sequences, from right to left, kept the left arm closer to the camera and, potentially, more visible. This may have introduced some bias in the results. However, statistically significant differences were proved for both arms. In a real scenario, this could be addressed by asking the patient to walk both from right to left and from left to right, and by combining the two measurements.

Considering the previous discussion, the value of the proposed method can be reasonably stated as a low-cost effective solution to quantify ASA from a single 2D gait video under a cooperative setting. Simple mobile applications can then be easily conceived to assist primary care professionals who usually lack specialized technologies and training.

5. Conclusions

Measuring arm swing asymmetry (ASA) can provide insight into health and quality of life, anticipate risks, and suggest appropriate therapies. Despite this, most gait studies still disregard arm swing. A cross-platform method for measuring ASA based on only a single 2D gait video was introduced. It is quite independent of acquisition conditions and spatio-temporal resolution. The method involves a novel silhouette-based gait representation and an algorithm to quantify arm motion. The method was validated on two gait video datasets, one with healthy subjects walking normally and one with healthy subjects simulating different levels of ASA. Experiments supported by two non-parametric statistical tests showed consistent results on two independent groups of normal gait sequences, and significant differences between groups with distinct asymmetry patterns. Mean ASA measurements were highly consistent with results reported in other works from different datasets and sensing technologies. Future work could explore more accurate functions of measuring arm motion, as well as possibilities of deploying this low-cost approach in real environments.