Movement Trajectory Recognition of Sign Language Based on Optimized Dynamic Time Warping

Abstract

Movement trajectory recognition is the key link of sign language (SL) translation research, which directly affects the accuracy of SL translation results. A new method is proposed for the accurate recognition of movement trajectory. First, the gesture motion information collected should be converted into a fixed coordinate system by the coordinate transformation. The SL movement trajectory is reconstructed using the adaptive Simpson algorithm to maintain the originality and integrity of the trajectory. The algorithm is then extended to multidimensional time series by using Mahalanobis distance (MD). The activation function of generalized linear regression (GLR) is modified to optimize the dynamic time warping (DTW) algorithm, which ensures that the local shape characteristics are considered for the global amplitude characteristics and avoids the problem of abnormal matching in the process of trajectory recognition. Finally, the similarity measure method is used to calculate the distance between two warped trajectories, to judge whether they are classified to the same category. Experimental results show that this method is effective for the recognition of SL movement trajectory, and the accuracy of trajectory recognition is 86.25%. The difference ratio between the inter-class features and intra-class features of the movement trajectory is 20, and the generalization ability of the algorithm can be effectively improved.

1. Introduction

Sign language (SL) is a complex dynamic mode accompanied by various gestures [1,2] and is the main form of communication for the deaf community. Automatically recognizing and converting SL into an easily understandable form for non-deaf people [3,4] will help the integration of the deaf community into society, which will bring great benefits to many deaf communities around the world. At present, SL translation mainly focuses on gesture recognition [5,6,7], whereas trajectory recognition is often simplified or ignored by many scholars. The same gesture matching with different movement trajectory often expresses two different meanings, that is, representing different SL. Therefore, the trajectory recognition is directly related to the accuracy of SL translation.

Machine vision and deep algorithms have been widely used in the recognition and detection of trajectory. In Reference [8], tracked devices and depth sensors are used to capture three-dimensional (3D) trajectory, which achieves good results in short trajectory and multi-finger gestures recognition, but has great limitations in long trajectory recognition, such as SL translation. In References [9,10], a 3D camera is used to track 3D trajectory. The machine vision cannot meet the wearable and low-cost requirements of the SL translation system; hence, research methods based on combined electromyography (EMG) and motion information are gradually favored [11,12]. In gesture motion information, acceleration (ACC) information is often directly encoded to the classifier for processing. In Reference [13], ACC sensors are used to collect data in three-axis direction and encoded to the pattern classifier. In Reference [12], ACC and EMG information are encoded to support vector machine (SVM) for gesture classification. ACC can only represent object displacement in a certain direction. When the attitude angle changes during motion, ACC cannot accurately reflect the trajectory of the object. Therefore, how to restore the SL movement trajectory without distortion is the first condition for accurate trajectory classification. Liu [14] restored the air handwriting trajectory, and the accuracy was 85%, but the stability needs to be further improved. In Reference [15], hidden Markov model (HMM) is used to recognize five kinds of dynamic gesture trajectories, and the average recognition rate is 84%. Five kinds of trajectories are difficult to meet the requirements of complex dynamic patterns of SL, and a recognition method with more types and higher accuracy is urgently needed.

The SL movement trajectory deviates to some extent due to the user’s action speed and action arm length; even for the same user, physiological or psychological changes lead to inconsistent execution speed of SL movement [16]. Time-scale normalization can simultaneously process the length of two movement sequences but may lead to a sharp decrease in the similarity of movement sequences. Therefore, an effective measurement method for the similarity of movement sequences with different length is needed.

Dynamic time warping (DTW) [17,18,19,20,21] can effectively solve the expansion and offset of the time series by warping the time axis, which is suitable for local speed. However, DTW focuses on the amplitude characteristics of time series but ignores its local shape characteristics, thus leading to the risk of abnormal matching. Weighted DTW [22] uses a nonlinear weight function to optimize sequence matching but ignores to the local shape characteristics of the sequence. Accurately matching the key features representing the shape characteristics of time series, such as local peak value and valley value, is necessary to ensure the accuracy of similarity measurement between sequences. Generalized linear regression (GLR) [23,24,25] is an accurate, efficient, and robust classification method with great advantages in the analysis and extraction of signal extremum features. In this paper, a new algorithm based on DTW for GLR model optimization (GLR-DTW) is proposed. The regression coefficient of the distance matrix network is constructed by modifying the activation function of GLR to optimize the algorithm. The similarity measure is used to identify different types of SL movement trajectory. The results show that this method effectively recognizes SL movement trajectory.

2. Materials and Methods

2.1. Overview

Figure 1 shows the steps of the trajectory recognition method proposed in this paper. Firstly, the Trigno wireless acquisition system (Delsys Ltd.) is used to collect the ACC and angular velocity (AV) signals when the SL is acted, and the ACC signals after the coordinate transformation are reconstructed to the SL movement trajectory by the adaptive Simpson algorithm. Then, the reconstructed trajectory compares with the trajectory of the template library one-by-one and are encoded to the GLR-DTW algorithm together. Finally, the similarity measure is used to judge whether the two warped trajectories are classified to the same category.

2.2. Trajectory Category

According to the analysis and induction of more than 100 kinds of high-frequency words [26] in Chinese SL, many words have great similarities in movement trajectory and can be divided into eight categories, as shown in Figure 2, after repeated induction. The description of trajectory category and corresponding vocabularies are shown in

Table 1. Description of eight kinds of trajectory and corresponding vocabulary.

Number	Name of Trajectory	Abbreviation	Chinese SL Vocabulary 1	Example
1	Horizontal straight-line movement	HLM	Da 2; Chang 3	Figure 2a
2	Vertical straight-line movement	VLM	Wen 4; Man 5	Figure 2b
3	Horizontal wave-shape movement	HWM	Jiang 6; Ge 7	Figure 2c
4	Vertical wave-shape movement	VWM	Baxi 8; Yidali 9	Figure 2d
5	Horizontal shaking	HS	Renshi 10; Jianmian 11	Figure 2e
6	Vertical shaking	VS	Jintian 12; Gaoxing 13	Figure 2f
7	Horizontal circular arc movement	HCM	Yun 14; Renmin 15	Figure 2g
8	Vertical circular arc movement	VCM	Yiqie 16; Dou 17	Figure 2h

¹ The Pinyin of Chinese characters will be explained here. ² Big. ³ Long. ⁴ Warm. ⁵ Full. ⁶ River. ⁷ Song. ⁸ Brazil. ⁹ Italy. ¹⁰ Understanding. ¹¹ Meeting. ¹² Today. ¹³ Happy. ¹⁴ Cloud. ¹⁵ People. ¹⁶ Everything. ¹⁷ All.

.

2.3. Data Acquisition

In this paper, Delsys-Trigno is used to collect the motion information when the SL is acted. The Trigno sensor is equipped with a three-axis accelerometer and a three-axis gyroscope, which can collect ACC and AV signals of SL action in real time. At the same time, the system is equipped with signal acquisition and transmission software, which greatly facilitates the acquisition experiment of motion information. The sensor is attached to the wrist of the subject, as shown in Figure 3.

2.4. Modeling Method of Trajectory

In SL movement trajectory modeling, ACC data cannot be directly integrated to obtain the exact displacement curve. Shaking or rotation of the hand changes the spatial direction of the wearing sensor and consequently, the coordinate system of the ACC sensor [14]. Transforming the measured ACC data into a fixed coordinate system, such as a geographic coordinate system, is necessary for an accurate trajectory detection.

The geographical coordinate system of the moving object is assumed to be $OXYZ$, and the carrier coordinate system is $O{X}_0{Y}_0{Z}_0$, as shown in Figure 4a. The transformation between coordinate systems can be realized by continuous rotation, as shown in Figure 4b. The specific transformation process can be referred to in References [14,27]. $O{X}_0{Y}_0{Z}_0$ transformation to $OXYZ$ can be obtained as follows:

$$\begin{array}{c}\left[\begin{array}{c}x\\ y\\ z\end{array}\right]=\left[\begin{array}{ccc}1& 0& 0\\ 0& \cos \alpha & \sin \alpha \\ 0& -\sin \beta & \cos \alpha \end{array}\right]\left[\begin{array}{ccc}\cos \beta & 0& -\sin \beta \\ 0& 1& 0\\ \sin \beta & 0& \cos \beta \end{array}\right]\left[\begin{array}{ccc}\cos \gamma & \sin \gamma & 0\\ -\sin \gamma & \cos \gamma & 0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}{x}_0\\ y_0\\ z_0\end{array}\right]\\ =\left[\begin{array}{ccc}\cos \gamma \cos \beta & \sin \gamma \cos \alpha +\cos \gamma \sin \beta \sin \alpha & \sin \gamma \sin \alpha -\cos \gamma \sin \beta \cos \alpha \\ -\sin \gamma \cos \beta & \cos \gamma \cos \alpha +\cos \gamma \sin \beta \sin \alpha & \cos \gamma \sin \alpha -\sin \gamma \sin \beta \cos \alpha \\ \sin \beta & -\cos \beta \sin \alpha & \cos \beta \cos \alpha \end{array}\right]\left[\begin{array}{c}{x}_0\\ y_0\\ z_0\end{array}\right]={\mathrm{C}}_{\mathrm{e}}^{\mathrm{b}}\left[\begin{array}{c}{x}_0\\ y_0\\ z_0\end{array}\right]\end{array}$$

(1)

where ${\left[x,y,z\right]}^T$ and ${\left[x_0,y_0,z_0\right]}^T$ are the ACC vectors of the geographical coordinate system and the sensor carrier coordinate system respectively, and ${\mathrm{C}}_e^b$ is the rotation matrix indicating the transformation relationship between two coordinate systems. $\alpha$, $\beta$, and $\gamma$ are angle values that can be calculated by integrating the collected AV signals.

After the coordinate transformation, the ACC vector will not be affected by the change of the sensor carrier direction. The trajectory curve can be obtained after the integration operation. Simpson integral is an approximate numerical integration method that uses conic approximation instead of trapezoid or rectangular integration to obtain the integral value.

For a given conic $g(x)=\alpha x^2+\beta x+\gamma$, the definite integral for $g(x)$ is as follows:

$$G(x)={\displaystyle {\int }_0^{x}g(x)dx}=\frac{\alpha x^3}{3}+\frac{\beta x^2}{2}+\gamma x+d$$

(2)

where $d$ is constant.

In the plane rectangular coordinate system, by fitting $f(x)$ with parabola determined by $\left(x_0,g(x_0)\right)$, $\left(x_1,g(x_1)\right)$, and $\left(x_2,g(x_2)\right)$ (where $x_1=\frac{x_0+x_2}{2}$), the approximate integral value on the interval $\left[x_0,x_2\right]$ is obtained as follows:

$${\displaystyle {\int }_{x_0}^{x_2}f(x)dx}\approx G(x_2)-G(x_0)=\frac{\alpha x_2^3-\alpha x_0^3}{3}+\frac{\beta x_2^2-\beta x_0^2}{2}+\gamma x_2-\gamma x_0$$

(3)

After Formula (3) is simplified, the Simpson integral formula of $f(x)$ on interval $\left[a,b\right]$ can be obtained as:

$${\displaystyle {\int }_a^{b}f(x)dx}\approx \frac{b-a}{6}\left[f(a)+4f\left(\frac{a+b}{2}\right)+f(b)\right]$$

(4)

The low sampling rate system will generate large error to the Simpson integral’s three-point fitting method. Many points will affect the calculation speed. An adaptive method is used to solve the trajectory for computation and error precision.

The integral interval is divided, and $c$ is set as the midpoint of interval $\left[a,b\right]$. If the following formula holds, then the recursive operation is completed:

$$\left|S(a,c)+S(c,b)-S(a,b)\right|<15\epsilon$$

(5)

where $S$ is Simpson’s formula and $\epsilon$ is error precision. If the above formula does not hold, then the interval $\left[a,c\right]$ and $\left[c,b\right]$ need to be further divided and iterated until the requirements of Formula (5) are satisfied. After the double integration of ACC data by the adaptive Simpson method, the vivid and intuitive 3D trajectory curve of SL movement is constructed and lays a foundation for the following trajectory recognition.

2.5. GLR-DTW Algorithm

DTW, which measures the similarity of two different length time series to effectively solve the expansion and migration of the series on the time axis, was proposed by Itakura and has been widely used in speech recognition and data mining [28,29,30]. The speech signal has great randomness, different people have varying pronunciations and speech speeds and drag sound sometimes appears. The SL movement trajectory features are similar to this speech feature. Hence, DTW can be applied to classify the SL movement trajectory.

The traditional DTW algorithm aims to measure a single dimension time series and uses Euclidean distance as the distance measure [9,31]. If the three-axis motion signals are warped separately, then the length of the warped signal sequence of each axis will differ, resulting in the serious distortion of the 3D trajectory. Holt et al. [32] extended the DTW algorithm to measure multidimensional time series but ignored the correlation between different dimensions when calculating the two sequences’ distance. Mahalanobis distance (MD) [33,34] is a method of calculating the similarity of two sequences, as proposed by Mahalanobis. In this paper, the GLR-DTW algorithm uses MD to expand the dimension of time series and modifies the activation function of GLR to construct the regression coefficient of the MD matrix network, which can achieve the accurate matching of local shape characteristic points while taking into account the correlation of three-axis motion signals.

For a given multidimensional time series $X=\{X_i,i=1,2,\cdots n\}$ and $Y=\{Y_j,j=1,2,\cdots m\}$, where $\{\begin{array}{c}{X}_i={\{x_{ki},k=1,2,\cdots q\}}^T\\ Y_j={\{y_{kj},k=1,2,\cdots q\}}^T\end{array}$.

In the formula, $X$ is the template sequence, $Y$ is the test sequence, $n$ and $m$ are the lengths of $X$ and $Y$ respectively, and $q$ is the dimension of $X$ and $Y$. With emphasis on the 3D signal sequence, the value of $q$ is set to 3.

The distance between any two points in the $X$ and $Y$ sequence can be expressed as:

$$d(X_i,Y_j)=\sqrt{{(X_i-Y_j)}^T{\sum }^{-1}(X_i-Y_j)}$$

(6)

where $\sum$ is the covariance matrix of the template sequence $X$ [23]. The matrix network of MD is constructed as follows:

$$\left[\begin{array}{cccc}d(X_1,Y_1)& d(X_2,Y_1)& \cdots & d(X_n,Y_1)\\ d(X_1,Y_2)& d(X_2,Y_2)& \cdots & d(X_n,Y_2)\\ & & & \\ d(X_1,Y_m)& d(X_2,Y_m)& \cdots & d(X_n,Y_m)\end{array}\right]$$

(7)

When searching the best path in the matrix network, the following constraints must be met.

The boundedness constraint should be followed, that is, starting from the starting point $d(X_1,Y_1)$ to the ending point $d(X_n,Y_m)$. The monotonicity and continuity constraint should be followed, that is, if the current node is $d(X_i,Y_j)$, then the next node must be selected between $d(X_{i+1},Y_j)$, $d(X_i,Y_{j+1})$, and $d(X_{i+1},Y_{j+1})$, and the path must be the shortest.

Therefore, the recursive algorithm of GLR-DTW is as follows:

$$\{\begin{array}{l}{d}_{i,j}={\lambda }_{ij}\cdot d{(X_i,Y_j)}^2={\lambda }_{ij}\cdot {(X_i-Y_j)}^T{\sum }^{-1}(X_i-Y_j)\\ D(i,j)=d_{i,j}+\min \left\{D(i-1,j),D(i,j-1),D(i-1,j-1)\right\}\\ D(1,1)=d_{1,1}\end{array}$$

(8)

where $D(i,j)$ is the cumulative distance from $d_{1,1}$ to $d_{i,j}$, which is the criterion of similarity measure. ${\lambda }_{ij}$ is the regression coefficient of the distance network optimized by the GLR model.

The sigmoid activation function of the GLR model is:

$$S(z)=\frac{1}{1+e^{-z}}$$

(9)

Through the extreme value detection of time series, the activation function $S(z)$ is modified according to the extreme value characteristics, and the regression coefficient of the matrix network is constructed. Accurate matching of local shape characteristic points is achieved in the sequence warping. The construction process is as follows:

$U_X=\{{\mu }_{ki},i=1,2,\cdots n,k=1,2,\cdots q\}$ and $U_Y=\{{\mu }_{kj},j=1,2,\cdots m,k=1,2,\cdots q\}$, representing the characteristic points of $X$ and $Y$, are obtained through extremum detection.

Among them, the rules for ${\mu }_{ki}$ and ${\mu }_{kj}$ values are as follows:

(1)

The value is 1 when the detection is a local maximum point.

(2)

The value is –1 when the detection is a local minimum point.

(3)

The value is 0 when the detection is a non-extreme point.

Therefore, the value of $\left|{\mu }_{ki}{\mu }_{kj}-1\right|$ can be calculated as:

(1)

The value is 0 when ${\mu }_{ki}=1,{\mu }_{kj}=1$ or ${\mu }_{ki}=-1,{\mu }_{kj}=-1$; that is, the maximum point matches the maximum point, or the minimum point matches the minimum point.

(2)

The value is 1 when at least one of ${\mu }_{ki}$ and ${\mu }_{kj}$ is 0; that is, the non-extreme point matches the other points.

(3)

The value is 2 when ${\mu }_{ki}=-1,{\mu }_{kj}=1$ or ${\mu }_{ki}=1,{\mu }_{kj}=-1$; that is, the maximum point matches the minimum point.

The above values are combined to modify the activation function, and the regression coefficient of the distance matrix network is obtained as follows:

$${\lambda }_{ij}=\frac{C}{1+\exp \left(-g{\displaystyle \sum _{k=1}^q\left|{\mu }_{ki}{\mu }_{kj}-1\right|}\right)}$$

(10)

where $q$ is the dimension of time series, $C$ is the maximum regression coefficient, which is a constant set as 2 in the simulation experiment, $g$ is the non-linear curvature, and the curve $\lambda (z)=\frac{C}{1+\exp \left(-gz\right)}$ with different $g$ value is shown in Figure 5.

The simulation results show that when $g\in \left[0.02,0.1\right]$, ${\lambda }_{ij}$ has good linearity, and when $g\in \left[0.1,0.2\right]$, ${\lambda }_{ij}$ has good optimization effect, and the value is 0.2 in the following experiment. When $g=1$, ${\lambda }_{ij}$ changes into standard sigmoid function.

Analysis results revealed that the value range of ${\lambda }_{ij}$ is [C/2, C]. If “maximum to maximum point” or “minimum to minimum point” matching occurs in all dimensions of the sequence, then $\sum \left|{\mu }_{ki}{\mu }_{kj}-1\right|=0$, and ${\lambda }_{ij}$ is always equal to the minimum C/2. The regression coefficient is the smallest, the distance measurement of the two sequences is reduced, and the similarity increases. Otherwise, the distance of the two sequences increases. DTW optimized by the regression coefficient realizes the calculation of amplitude distance and considers the shape characteristics of time series to avoid abnormal matching.

In path warping, the index numbers of the sequence are recorded. ${\overline{w}}_{\mathrm{x}}(k)$ and ${\overline{w}}_{\mathrm{y}}(k)$ are the index numbers of $X$ and $Y$, respectively. The optimal path can then be expressed as follows:

$$\overline{W}=\left(\begin{array}{c}{\overline{w}}_x(k)\\ {\overline{w}}_y(k)\end{array}\right),k=1,2,3,\cdots p$$

(11)

where $p\in \left[\max (n,m),n+m\right]$, which is the length of the new sequence.

The optimal warping path, $\overline{W}$, is obtained, and the sequences $\overline{X}(k)$ and $\overline{Y}(k)$ are extended from $X_i$ and $Y_j$ by $\overline{W}$. The sequences after dynamic warping are expressed as follows:

$$\{\begin{array}{c}\overline{X}(k)=X({\overline{w}}_x(k))\\ \overline{Y}(k)=Y({\overline{w}}_y(k))\end{array},k=1,2,3,\cdots p$$

(12)

3. Results and Discussion

3.1. Template Trajectory Library

In the experiment of recognition of the SL movement trajectory, eight kinds of standard SL movement trajectory libraries should be established as the template trajectory of algorithm input, as shown in Figure 6. A SL teacher from a deaf school in Hangzhou was invited as the SL instructor. After wearing the Trigno sensor, the eight kinds of movements in Figure 2 were executed successively, and the ACC and AV signals of corresponding movements were collected as the motion information of the template trajectory library.

Another 10 healthy volunteers (5 males and 5 females) were recruited. After repeated training of eight kinds of movements, wearing the Trigno sensor, each volunteer performed the eight movements at regular speed (about 300 ms each movement, equivalent to the speed of template trajectory) and fast speed (about 200 ms each movement), recording the corresponding ACC and AV signals.

3.2. Experiment of Movement Trajectory Modeling

In order to facilitate the observation and analysis of the experimental results, the “vertical wave-shape movement” was selected as an example to show the experimental process of trajectory classification. Figure 7a,c presents the collected three-axis ACC and AV signals of vertical wave-shape movement (VWM) in the template library, and Figure 7b,d shows the collected three-axis ACC and AV signals of the faster VWM (fVWM), respectively. In these figures, ACCx, ACCy and ACCz represent the ACC signals of the x-, y-, and z-axis respectively, and AVx, AVy and AVz represent the AV signals of the x-, y-, and z-axis, respectively. The dashed boxes in the two figures are the effective signal intervals to represent these two groups of movements. VWM takes about 300 ms, while fVWM takes 200 ms.

Observation and analysis of Figure 7 showed that even with the same wave-shaped movement curve, the speed of movement directly leads to differences in ACC and AV signal characteristics, thus complicating the movement recognition. In addition, the six-dimensional (6D) data input of ACC and AV signals increases the training burden of the pattern classifier.

Figure 8a displays the waveform of VWM ACC signals after coordinate conversion, and Figure 8b shows the waveform of fVWM ACC signals after coordinate conversion. TSx, TSy and TSz represent the ACC signals of the x-, y-, and z-axis respectively, in the geographic coordinate system. The two figures only include the waveform of the effective movement interval for comparison and observation. Comparative analysis indicated that the ACC curves have shown a certain similar trend after the transformation of the two groups.

Figure 9 shows a 3D trajectory obtained by using the adaptive Simpson integral for the signals of Figure 8. In the figure, the red line is the VWM trajectory, and the blue line is the fVWM trajectory. The wave-shape characteristics of the two curves are visible in this figure. The modeling of movement trajectory greatly reduces the dimension of the eigenvector and greatly decreases the design complexity of the pattern classifier.

3.3. Experiment of GLR-DIW

The 3D trajectories of VWM and fVWM were restored and modeled. In order to verify the effectiveness of the GLR-DTW algorithm, the above VWM and fVWM trajectories were encoded to the DTW and GLR-DTW algorithms respectively, and the new trajectories, after warping, were compared, as shown in Figure 10. In the figure, the red line is the warped VWM trajectory, and the blue line is the warped fVWM trajectory. The algorithm cannot directly find the differences because it warps on the time axis, which is not reflected by the 3D coordinate map. Therefore, the expansions of the two output trajectory sequences on the time axis were analyzed and compared, as shown in Figure 11.

In Figure 11a, the TPLx, TPLy and TPLz are the x-, y-, and z-axis data of the template sequence, i.e., VWM trajectory, respectively, and TSTx, TSTy and TSTz are the x-, y-, and z-axis data of the test sequence, i.e., fVWM trajectory, respectively. The three solid lines and dashed lines in Figure 11b are two sets of new sequence output by the DTW algorithm. Given that the Euclidean distance is used in the DTW algorithm, the correlation among varying dimensional data is ignored in the multidimensional time series calculation. In Figure 11b, DTW only considers the sequence characteristic of the largest amplitude dimension, leading to the abnormal matching of other dimension sequences. The matching of local peak point and valley point appears at symbol 1.

The three solid lines and dashed lines in Figure 11c are two sets of new sequence output by the GLR-DTW algorithm. Given that GLR-DTW first uses MD as the distance measure of the algorithm, the sequences can achieve synchronous warping under the premise of considering the correlation of 3D data. In addition, the regression coefficient of the algorithm is optimized to achieve the accurate matching of local extreme value characteristics, thereby solving the abnormal matching problem of DTW, as shown in symbol 2 of Figure 11c. Observation and analysis revealed that the interpolation point at symbol 3 in Figure 11c is the acceleration area of the fVWM trajectory with fast speed and short time. When the sequence is matched, sequence stretching is achieved. After constant warping, the lengths of the two sets of sequences are equalized.

In summary, GLR-DTW is better than traditional DTW in the recognition of movement trajectory and solves the drawback of abnormal matching. It is helpful to improve the accuracy of movement trajectory classification.

3.4. Similarity Measurement and Classification

The calculation process of the GLR-DTW algorithm is actually searching the best path—the shortest cumulative distance between two sets of sequences. After the search is completed, the distance between sequences was also calculated. When measuring the similarity of two trajectory sequences, one group of trajectories of a volunteer (subject A) and trajectories of the template library were selected and encoded in turn to the GLR-DTW algorithm designed in this paper. The distance between the two sequences was obtained as the basis for similarity measurement. Specific data are listed in

Table 2. Distance measurement of the two trajectory sequences.

	A’s HLM	A’s HLM	A’s HWM	A’s VWM	A’s HS	A’s VS	A’s HCM	A’s VCM
Template HLM	12.1902	3434.37	1554.33	1773.51	2494.58	2534.45	5705.32	5680.11
Template VLM	3197.84	16.9422	1802.92	1443.56	2534.47	2434.90	5669.26	5757.40
Template HVM	1500.87	1834.00	38.0277	4234.38	1034.20	1234.36	5098.20	5811.52
Template VWM	1744.70	1434.90	4434.78	41.3289	1206.50	982.11	5985.74	4919.94
Template HS	2344.23	2601.31	1022.39	1324.58	26.3975	3344.39	4080.32	3891.13
Template VS	2543.78	2450.85	1234.76	1014.70	3534.43	29.0043	4191.20	4000.12
Template HCM	5971.25	5008.24	5191.70	5888.94	4091.33	4229.32	40.1974	2998.78
Template VCM	5592.40	5729.09	5891.15	5191.53	3990.82	4077.04	2833.78	45.2332

.

The eight sequences all intercept 30 points because the distance between every two trajectory sequences is also related to the number of sequence points. Table 2 shows that the distance of the same movement trajectory between two subjects is between 10 and 50, and the distance of a different movement trajectory is basically more than 1000. The difference ratio between different and similar trajectory features is 1000/50 = 20, indicating large variation. Therefore, this distance measurement method is feasible as a recognition criterion for different SL movement trajectories.

The volunteer’s test trajectory and the eight template trajectories are warped one-by-one and the distance is calculated. If a group with the smallest distance among them is found, and the difference ratio between other groups and this group are all greater than 20, then this group is classified into the same category. Volunteers’ SL act at the regular speed and faster speed, i.e., each movement had 20 trajectory data, which encoded to HMM, DTW, and GLR-DTW for comparison, respectively. The correct recognition number comparison of HMM, DTW, and GLR-DTW are shown in

Table 3. The comparison of correct recognition number.

-	Number of Samples	HMM [32]	DTW	GLR-DTW
HLM trajectory	20	19	19	20
VLM trajectory	20	18	19	19
HWM trajectory	20	17	18	18
VWM trajectory	20	18	15	17
HS trajectory	20	17	16	17
VS trajectory	20	16	18	18
HCM trajectory	20	15	13	14
VCM trajectory	20	14	15	15
Average accuracy	-	83.75%	83.13%	86.25%

. Among the three methods, the average accuracy of GLR-DTW proposed in this paper is the highest, reaching 86.25%, which is 3.12% better than the traditional DTW.

Time complexity is the time spent from the end of trajectory signals’ acquisition to the conclusion of recognition, which is mainly a measure of the real-time performance of the algorithm. The HMM algorithm strongly depends on the training samples, and the parameter model can be obtained through repeated calculation, which has a large amount of calculation and takes a long time. Therefore, HMM has the longest calculation time among the three methods. Due to the addition of the covariance matrix and regression coefficient calculation, the response time of GLR-DTW is slightly longer than that of DTW, and the average response time is about 140 ms, as shown in

Table 4. The comparison of time complexity (ms).

	HMM [32]	DTW	GLR-DTW
HLM trajectory	207.3	104.7	123.0
VLM trajectory	198.4	102.1	122.7
HWM trajectory	221.4	131.4	148.1
VWM trajectory	208.9	123.2	153.2
HS trajectory	204.6	120.8	139.3
VS trajectory	193.0	117.0	134.8
HCM trajectory	211.1	131.2	154.4
VCM trajectory	216.8	138.5	150.6
Average time	207.7	121.1	140.8

. However, with a response time of 140 ms, users can hardly detect the delay of the recognition process, and the real-time performance is good. Therefore, the real-time performance of the GLR-DTW algorithm meets the requirements of the real-time system.

Figure 12 shows the recognition of the GLR-DTW algorithm under the conditions of regular speed and fast speed. According to the observation in this figure, the classification accuracy of eight kinds of trajectories is basically the same under different speed conditions. The action arm length of SL was different between male and female volunteers, but it had no effect on recognition results. Therefore, the trajectory classification based on GLR-DTW solves the recognition problem of SL movement trajectory deviation under different speed conditions and different arm conditions.

4. Conclusions

SL movement trajectory recognition is an important research direction for SL translation. This paper deeply discussed a method of SL movement trajectory recognition based on GLR-DTW. First, the algorithm collected ACC and AV information during SL execution, transformed 6D original data into 3D feature data through a trajectory modeling algorithm, and output a 3D trajectory curve with many intuitive features. The DTW algorithm optimized by modifying GLR activation function was then used to warp movement trajectories. Finally, similarity measures were employed to identify different types of SL movement. Experimental results show that this method effectively recognizes SL movement trajectory. A large difference was observed between the inter- and intra-class features of movement trajectory; hence, the generalization ability of the algorithm can be effectively improved. In addition, the method reduced the dimension of the input eigenvector, greatly simplified the complexity of the pattern classifier, and has a certain reference value for the processing of similar trajectories.

This paper also discussed the problem of trajectory classification under different speed and different arm conditions in detail. This method can reconstruct the original 3D trajectory curve through the motion information and has good robustness to the trajectory classification under different conditions. Compared with the traditional DTW method, the method proposed in this paper has a great improvement in the problem of abnormal matching of trajectory classification. In the future work, the method proposed in this paper will be coupled with the existing hand shape and gesture recognition systems and try to be transplanted to a real-time system. However, it is worth noting that this study only examines healthy people. In the next step, we will cooperate with the rehabilitation institution to do further experiments and research, and test the method proposed in this paper on a group of deaf patients under physical therapy.