Skeleton-Based Dynamic Hand Gesture Recognition Using an Enhanced Network with One-Shot Learning

Abstract

Dynamic hand gesture recognition based on one-shot learning requires full assimilation of the motion features from a few annotated data. However, how to effectively extract the spatio-temporal features of the hand gestures remains a challenging issue. This paper proposes a skeleton-based dynamic hand gesture recognition using an enhanced network (GREN) based on one-shot learning by improving the memory-augmented neural network, which can rapidly assimilate the motion features of dynamic hand gestures. Besides, the network effectively combines and stores the shared features between dissimilar classes, which lowers the prediction error caused by the unnecessary hyper-parameters updating, and improves the recognition accuracy with the increase of categories. In this paper, the public dynamic hand gesture database (DHGD) is used for the experimental comparison of the state-of-the-art performance of the GREN network, and although only 30% of the dataset was used for training, the accuracy of skeleton-based dynamic hand gesture recognition reached 82.29% based on one-shot learning. Experiments with the Microsoft Research Asia (MSRA) hand gesture dataset verified the robustness of the GREN network. The experimental results demonstrate that the GREN network is feasible for skeleton-based dynamic hand gesture recognition based on one-shot learning.

1. Introduction

With the rapid development of Kinect, Leap Motion, and other sensors in recent years, hand motion capture is getting much more efficient. By estimating the posture of the hand gesture, the position information of each joint can be detected from video or image sequences. Recent research [1,2,3,4,5] has tried various ways for dynamic hand gesture recognition based on 3D skeleton data characterized as strong correlations, temporal continuity, and co-occurrence relationships. Besides, the skeleton-based algorithm has fewer parameters, which is easier to calculate and more suitable for analyzing dynamic hand gestures. However, it is still challenging because hands are non-rigid objects, which can express a variety of different semantics [6]. With the gesture recognition technology being applied in more fields such as gaming and industry training, it is often necessary to make different customized annotation samples in large sizes. However, it is worth noting that the existing hand gesture database could not meet the needs of gesture interaction in various fields. The cost of large-scale gesture sample extraction artificially in each field is so high that it would limit the application of gesture recognition [7,8]. Meanwhile, the traditional gradient-based networks also require extensive iterative training to complete the model optimization. When encountering the new data, the models need to relearn their hyper-parameters to adequately incorporate the new information without catastrophic interference [9], which is inefficient. The existing networks fail to complete the optimization of the model with small size training samples, while one-shot learning could infer results as expected [10]. Therefore, the method of one-shot learning can be used to solve the problem that the model could not be optimized by the insufficient samples of skeleton-based dynamic hand gestures.

However, if the current algorithm of “one-shot learning” is directly applied to the hand gesture recognition, there will be three gradient-based optimization problems. Firstly, due to the small amount of data, many advanced and mature algorithms, such as Momentum [11] and Adagrad [12], cannot be optimized in limited iterations; especially when encountering non-convex problems, many hyper-parameters cannot achieve convergence. Secondly, for different tasks, the parameters of the network need to be initialized randomly. If the amount of data is too small, the final model cannot achieve convergence. This can be alleviated by conducting transferring learning methods, such as fine-tuning [13,14]. Finally, for the traditional neural network, its memory storage is limited. Additionally, the process of learning a new set of patterns will suddenly and completely erase a network’s knowledge of what it had already learned, which is referred to as catastrophic interference [15]. Therefore, we need to find a memory module that can be used for large-scale storage and can also be accessed for relevant information. The large capacity enhanced memory neural networks, such as a neural Turing machine (NTM) [16], provides a feasible method for one-shot learning combined with hand gesture recognition. The NTM provides the capability to quickly encode and retrieve new information by limiting the changes in the output of the network before and after the network update [15,17]. In addition, it can also eliminate gradient-based optimization problems. On this basis, Santoro [9] introduced a new and pure content-based method for accessing an external memory, which is different from previous methods, additionally using a memory location-based focusing mechanism. The method can rapidly bind never-before-seen information to the external memory after a single presentation and combines the gradient descent to slowly learn an abstract method for obtaining useful representations of raw data. As a result, it can accurately identify the categories of data that have occurred only once.

This paper focuses on the architecture of enhanced neural networks based on skeleton-based algorithms and one-shot learning. Based on the memory-augmented neural network (MANN) [9], we propose skeleton-based dynamic hand gesture recognition using an enhanced network (GREN). The long short-term memory (LSTM) network is selected as the controller of the GREN network to enhance the recognition and memory ability of the network. Compared with the MANN network, which was originally applied to image recognition, the proposed GREN network classifies hand gestures by identifying skeletal sequences. Through the recognition of the GREN network, we conduct experiments on a dynamic hand gesture dataset (DHGD) [18] to show the effectiveness of our method. Then, we implement our method on the Microsoft Research Asia (MSRA) hand gesture dataset [19] to verify its contributions.

The rest of this paper is organized as follows:

• Section 2 details the related work of skeleton-based dynamic hand gesture recognition and one-shot learning.
• The GREN network is introduced in Section 3.
• The experiments of skeleton-based dynamic hand gesture recognition are explained in detail in Section 4.
• In Section 5, results and discussion are presented.
• The conclusions are given in Section 6.

2. Related Work

2.1. Skeleton-Based Dynamic Hand Gesture Recognition

Much research has been focused on skeleton-based dynamic hand gesture recognition [20,21,22,23,24,25,26,27,28,29]. Chen X. et al. [30] proposed a skeleton-based dynamic hand gesture recognition algorithm that has also been suggested to surpass depth-based methods in the aspect of performance. Chin-Shyurng et al. [31] created a skeleton-based model by capturing the palm position, and the dynamic time-warping algorithm was applied to the recognition of disparate conducting gestures at various conducting speeds, which achieves real-time dynamic musical conducting gesture recognition. Ding, Ing-Jr et al. [32] designed an adaptive hidden Markov model (HMM)-based gesture recognition method with user adaptation (UA) to simplify large-scale video processing to realize the natural user interface (NUI) of a humanoid robot device. Similarly, Kumar, Pradeep et al. [33] used the HMM to identify occluded gestures in line with a robust position invariant sign language recognition (SLR) framework.

Additionally, some studies have employed deep learning methods to conduct skeleton-based dynamic hand gesture recognition. Mazhar, Osama et al. [34] proposed that humans need neither to wear any specific clothing (motion capture clothes or inertial sensors) nor to carry a special remote control or learn complex teaching instructions in gesture recognition. As a result, they developed a real-time, robust, and background-independent gesture detection module in the light of convolutional neural network (CNN) transmission learning. Chen, XH et al. [29] exploited motion features of traits and global movements to augment features of recurrent neural networks (RNNs) for gesture recognition and improve the classification performance. Lin, C et al. [35] proposed a novel refined fused model in combination with the masked Res-C3D network and skeleton LSTM for abnormal gesture recognition in RGB-D videos, which learns discriminative representations of gesture sequences in particular abnormal gesture samples by fusing multiple characteristics from different models. Based on a combination of a CNN network and an LSTM network, Nunez, JC et al. [36] proposed a deep learning-based approach for temporal 3D pose recognition problems, and the proposed network architecture does not need to be adapted to the type of activity or the gesture to be recognized, as well as the geometry of the 3D sequence data as input. So far, there is no available deep learning network that can be directly used for skeleton-based dynamic hand gesture recognition based on small size samples.

2.2. One-Shot Learning

The implementations of one-shot learning can be divided into statistics-based, weight-based matching, and meta-learning. For the statistics-based, Lake [37] adopted the Bayesian framework realized one-shot learning of handwritten character pictures based on the statistical point of view and the way humans learn things, triggering the new wave of one-shot learning.

Besides the above statistics, there are also many methods on the basis of weighted matching for one-shot learning, which performs certain criteria modeling on known samples and then determines the class according to the distance of samples. The most typical method is the k-nearest neighbor (KNN), which is a nonparametric estimation method that can directly employ distance to determine the category without prior training. Another method is to learn an end-to-end nearest neighbor classifier, which can not only quickly learn new samples but also have a great generalization of known samples. Snell et al. [38] carried out classification by calculating the distance from prototype representations of each class, which turns into the nearest neighbor classification in the metric space. While Koch et al. [39] performed efficacious feature extraction on new samples by limiting input methods, then used supervised metric learning based on twin networks to train and finally reused features extracted by that network for small or no sample learning. Similarly, Oriol Vinyals et al. [40] also utilized metric learning based on deep neuro features, which uses external memory to enhance the neural network that maps a small labeled support set and an unlabeled example to its label, obviating the need for fine-tuning to adapt to new class types.

Meta-learning, also known as “learning to learn”, aims to train a model on a variety of learning tasks, such that it can solve new learning tasks using only a small number of training samples [41]. A neural network with memory can implement meta-learning, but its memory storage is limited. A large number of new features may exceed the memory storage capacity so that the network cannot learn new tasks. The NTM network can solve this problem, as it is capable of both long-term storage via slow updates of its weights and short-term storage via its external memory module [16]. Based on the NTM network, Santoro et al. [9] introduced a memory access module that emphasizes accurate encoding of relevant (recent) information and pure content-based retrieval to implement meta-learning. Besides, Ravi et al. [42] proposed an LSTM-based meta-learner model, whose parameterization allows it to learn appropriate parameter updates specifically for the scenario where a set amount of updates will be made, while also learning a general initialization of another learner (classifier) network that allows for quick convergence of training.

In general, the current one-shot learning-based methods are in a booming period. However, there is still no appropriate method for one-shot learning with skeleton-based hand gesture recognition. Therefore, this paper will study the current advanced achievements and propose a suitable algorithm to realize hand gesture recognition in line with one-shot learning.

3. Dynamic Hand Gesture Recognition with the GREN Network

By improving a MANN network, this paper implements the GREN network based on one-shot learning, which is a variant of the NTM network from Santoro et al. [9]. Compared with the MANN network originally applied to image recognition, the proposed GREN network classifies hand gestures by recognizing skeletal sequences. The structure of the GREN network is shown in Figure 1.

The GREN network consists of three components: a controller, read and write heads, and an external memory. The controller, neuron $C_{lstm}$, employed in our model is an LSTM network, which receives the current input and controls the read- and write-heads to interact with the external memory, respectively. Memory encoding and retrieval in an external memory are rapid, with vector representations being placed into or taken out of memory potentially every time-step [16], which makes it a perfect candidate for one-shot prediction. Additionally, it can be stored either for long-term storage by slowly updating the weights or for short-term storage by an external memory. Thus, when the model learns the type of representation of a gesture sequence, it will be placed into memory, and later these representations will be used to make predictions of data that it has only seen once. Besides, according to the difference of classification methods between the input of images and sequences, the average pooling layer (avgpool) is introduced to further focus on characteristics of sequence and improve the calculation efficiency in the network. For one-shot learning, the output distribution is categorical, which is implemented as a softmax function.

At the beginning, the initialized state of the GREN network is represented by $init\_state$. The external memory is initialized, which does not store any data representations. Also, the memory $r_0$ retrieved from the external memory is empty. In addition, the cell state of the initialized controller, neuron $C_{lstm}$, is represented by $c_0$. Given the input sequence $X_t$, the controller receives the memory $r_{t-1}$ and cell state $c_{t-1}$ provided by the previous state $prev\_state$, then produces a query key vector $k_t$ used to retrieve a particular memory. When encountering sequences of the already-seen class, the particular memory vector row could be retrieved by read heads, which is addressed using the cosine similarity measure:

$$\mathrm{K}\left(k_t,M_t\left(i\right)\right)=\frac{k_t·M_t\left(i\right)}{\Vert k_t\Vert ·\Vert M_t\left(i\right)\Vert }$$

(1)

where $M_t$ is the memory matrix at time-step $t$ and $M_t\left(i\right)$ is the $i^{th}$ row in this matrix. The row of $M_t\left(i\right)$ serve as memory “slots”, with the row vectors themselves constituting individual memories.

After then, a read-weight vector $w_t^r$ is produced by these similarity measures according to the softmax function:

$$w_t^r\left(i\right)\leftarrow \frac{\exp \left(\mathrm{K}\left(k_t,M_t\left(i\right)\right)\right)}{{\sum }_j\exp \left(\mathrm{K}\left(k_t,M_t\left(j\right)\right)\right)}$$

(2)

where the read heads can amplify or attenuate the precision of the focus by the read weights.

Those read weights $w_t^r$ and corresponding memory $M_t\left(i\right)$ are used to retrieve the memory $r_t$:

$$r_t\leftarrow {\displaystyle \sum }_i{w}_t^r\left(i\right)·M_t\left(i\right)$$

(3)

where the memory $r_t$ is used by the controller as both an input to a classifier, namely, a softmax layer for class prediction and as an additional input for the next input sequence.

To achieve the combined learning in disparate classes and implement the one-shot learning, the least recently used access module (LRUA) proposed by Adam Santoro [9] is adopted, which is a pure content-based memory write head that writes memories to either the least used memory location or the most recently used one, and focusing on the accurate encoding of the most relevant information. In terms of a new sequence, it is written to a rarely-used location with the recently encoded information preserved or to the last used location, which can be used for updating with newer or possibly more relevant information:

$$w_t^u\leftarrow \gamma ·w_{t-1}^u+w_t^r+w_t^w$$

(4)

$$w_t^{lu}\left(i\right)=1if{w}_t^u\le m\left(w_t^u,n\right)else0$$

(5)

$$w_t^w\leftarrow \sigma \left(\alpha \right)·w_{t-1}^r+\left(1-\sigma \left(\alpha \right)\right)·w_{t-1}^{lu}$$

(6)

$$M_t\left(i\right)\leftarrow M_{t-1}\left(i\right)+w_t^w\left(i\right)·k_t·\forall i$$

(7)

where $w_t^u$ is the usage weight updated at each time-step to keep track of locations most recently read from or written to; $\gamma$ is the decay parameter; $w_t^{lu}$ is the least-used weight computed using $w_t^u$ for a given time-step; the notation $m\left(v,n\right)$ is introduced to denote the $n^{th}$ smallest element of the vector $v$; $n$ is set to equal the number of the writer to memory; $w_t^w$ is the written weight computed by the sigmoid function $\sigma (.)$, which combines the previous read weights $w_{t-1}^r$ and previous least-used weights $w_{t-1}^{lu}$; $\alpha$ is a dynamic scalar gate parameter to interpolate between weights. Before writing to memory, the least used memory location is computed from $w_{t-1}^u$ and set it to zero, then the memory $M_t$ is written by the computed vector of written weights $w_t^w$. Thus, $M_t\left(i\right)$ can be written into the zeroed memory location or the previously used memory location; if it is the latter, then $w_t^{lu}$ will simply get erased.

With the above analysis, we propose the following GREN algorithm, as shown in Algorithm 1.

Algorithm 1: GREN
	Input: Given N samples $\left\{X_1,X_2,\dots ,X_N\right\}$ belonging to C classes with    Sample-classes $y_t\in Y=\left\{1,\dots ,C\right\}, for t=1,\dots ,N$;
	Output: A softmax layer for class prediction;
1	Initialization:
2	$prev\_state\leftarrow init\_state\left(N\right)\{$
3	$c_0\leftarrow C_{lstm}\left(N\right);$
4	$r_0\leftarrow 0_{N\times \left(head\_num*memory\_size\right)};$
5	$w_0^r\leftarrow one\_hot\_weigh\_vector\left(N,head\_num,memory\_slots\right);$
6	$w_0^u\leftarrow one\_hot\_weigh\_vector\left(N,memory\_slots\right);$
7	$M_0\leftarrow {\epsilon }_{N\times memory\_slots\times memory\_size};$
8	$return \left\{c_0,r_0,w_0^r,w_0^u,M_0\right\};$
9	$\};$
10	$o=[ ];$
11	$for t\leftarrow 1 to N do$
12	$h_t,c_t\leftarrow C_{lstm}\left(\left(X_t,y_t\right),prev\_state\right);$
13	$for i\leftarrow 0 to X_t.length do$
14	$output,curr\_state\leftarrow gren\left(\left(X_t\left(i\right),x\_labl{e}_t\left(i\right)\right),prev\_state\right)\{$
15	Memory Retrieval:
16	$K\left(k_t,M_t\left(i\right)\right)\leftarrow cosine\_similarity\left(k_t,M_t\left(i\right)\right);$
17	$w_t^r\left(i\right)\leftarrow softmax\left(K\left(k_t,M_t\left(i\right)\right)\right);$
18	$r_t+=w_t^r\left(i\right)·M_t\left(i\right)$;
19	Memory Encoding (LRUA):
20	$w_t^u\leftarrow \gamma ·w_{t-1}^u+w_t^r+w_t^w;$
21	$if w_t^u\le m\left(w_t^u,n\right)then{w}_t^{lu}\left(i\right)=1 else w_t^{lu}\left(i\right)=0;$
22	$w_t^w\leftarrow sigmoid\left(\alpha \right)·w_{t-1}^r+\left(1-sigmoid\left(\alpha \right)\right)·w_{t-1}^{lu};$
23	$M_t\left(i\right)\leftarrow M_{t-1}\left(i\right)+w_t^w\left(i\right)·k_t;$
24	$return \left\{h_t,r_t\right\},\left\{c_t,r_t,w_t^r,w_t^u,M_t\right\};$
25	$\}$;
26	$prev\_state=curr\_state;$
27	$if i==0 then$
28	$o2o\_w\leftarrow \left(output.length,M_{class}\right),rand\_unif\_init\left(minv,maxv\right);$
29	$o2o\_b\leftarrow \left(M_{class}\right),rand\_unif\_init\left(minv,maxv\right);$
30	$end if$;
31	$output=output·o2o\_w+o2o\_b;$
32	$output=softmax\left(output\right);$
33	$o.append\left(output\right);$
34	$end$
35	$learning\_loss=-cross\_entropy\_cost\left(y_t,o\right);$
36	$optimizer=AdamOptimizer\left(learning\_rate\right);$
37	$train\_op=optimizer.minimize\left(learning\_loss\right);$
38	end

In the algorithm, the $one\_hot\_weigh\_vector\left(a,b,c\right)$ function generates a tensor of shape $a\times b\times c$ with $\left[:,:, 0\right]$ set to $one$ (or $\left[:, 0\right]$, if the $one\_hot\_weigh\_vector\left(a,b\right)$ function generates a tensor of shape $a\times b$); $\left\{\left(a,b\right),rand\_unif\_init\left(minv,maxv\right)\right\}$ generates a tensor of shape $a\times b$ (or $\left\{\left(a\right),rand\_unif\_init\left(minv,maxv\right)\right\}$ generates a tensor of shape $a\times 1$) with a uniform distribution, and the value of all elements is set between $minv$ and $maxv$.

In general, for the current time-step $t$, the sample data $X_t$ and the corresponding sample-class $y_t$ will be received by the controller $C_{lstm}$. The current state of the GREN network $curr\_state$ is used by the controller as an additional input for the next time-step. According to each sequence of the sample, the GREN algorithm randomly generates the class label $x\_labe{l}_t$. If the sample date $X_t$ comes from a never-before-seen class, it will be bound to the appropriate sample-class $y_t$ and stored by the write heads in the external memory, which is presented in the subsequent time-step (see Figure 1). Later, once a sample from an already-seen class is presented, the controller will retrieve the bound sample-class information by the read heads from the external memory for class prediction. A softmax layer, $softmax(·)$, is selected to output the standardized probability distribution of the model prediction, and combined with the cross-entropy cost function, $cross\_entropy\_cost(·)$, to measure the loss between the predicted value and correct class label. Then, the adaptive moment estimation (Adam) [43], $AdamOptimizer(·)$, is adopted to minimize the loss, and the back-propagated error signal from the current prediction updates those previous weights, which is followed by the updating of the external memory. Those processes would be repeated until the model converges.

4. Experiments

In this section, two hand gesture datasets named dynamic hand gesture database (DHGD) and MSRA are used for the experiments. Details about the experimental setup of the GREN network are introduced in the later part of this section.

4.1. Datasets

4.1.1. DHGD Hand Gesture Dataset

The public DHGD hand gesture dataset [18] contains sequences for 14 right-hand gestures performed in two ways: using one finger and the whole hand. Each class of gestures is performed 1 to 10 times by 28 participants in both of the above two ways, resulting in 2800 sequences, and the length of the gestures varies from 20 to 50 frames. Each frame contains the coordinates of the 22 joints in the 2D depth image space and 3D world space, and those joints are shown in Figure 2.

Some gestures (such as swipe and shake), which are defined by the movement of the hand, called the coarse gesture, while others are defined by the shape of the gesture, called the fine gesture.

Table 1. List of 14 gestures in the dynamic hand gesture database (DHGD).

Name of the Gesture	Type of the Gesture	Type of the Gesture
1	Grab	Fine
2	Tap	Coarse
3	Expand	Fine
4	Pinch	Fine
5	Rotation Clockwise	Fine
6	Rotation Counter Clockwise	Fine
7	Swipe Right	Coarse
8	Swipe Left	Coarse
9	Swipe Up	Coarse
10	Swipe Down	Coarse
11	Swipe X	Coarse
12	Swipe +	Coarse
13	Swipe V	Coarse
14	Shake	Coarse

shows the different classes of gestures in DHGD:

4.1.2. MSRA Hand Gesture Dataset

The public MSRA [19] hand gesture dataset, which contains skeleton-based sequence data of 17 right-hand gestures performed by 28 participants, is chosen to verify the robustness of the GREN network. The 17 right-hand gestures are manually chosen and are mostly from American Sign Language, to span the space of finger articulation as much as possible. Additionally, the length of each gesture varies from 490 to 500 frames. Each of these frames contains the coordinates of the 21 joints in the 2D depth image space and 3D world space, and those joints are shown in Figure 3.

4.2. Experimental Setup

4.2.1. Data Pre-Process

The skeleton-based hand gesture datasets should be preprocessed as the input of our network. The whole framework of the data preprocessing is shown in Figure 4, in which the $k^{th}$ class gesture is processed by our method as an example.

First of all, the nested interval unscented Kalman filter (UKF) [44] is used to eliminate the possible noise in the hand gesture datasets. Moreover, due to some hand gesture datasets may contain unequal sequences from different participants, the short and long sequences should be changed into a standard sequence. The length of the standard sequence is set to a fixed value $n$ based on both the average length of the sequence of each gesture. For short sequences, the length of them is increased by linear interpolation. For long sequences, we will eliminate the first few frames and the last few frames of the sequence because there are usually many pause actions at the beginning and the end, and they are not important to the whole gesture. The joint $P_{i,k}\left(t\right)$, a full hand skeleton $H_k\left(t\right)$ and the $k^{th}$ class gesture $G_k$ are shown as follows:

$$P_{i,k}\left(t\right)=\left[x_{i,k}\left(t\right),y_{i,k}\left(t\right),z_{i,k}\left(t\right)\right]$$

(8)

$$H_k\left(t\right)={\displaystyle \sum }_{i=1}^m{P}_{i,k}\left(t\right)$$

(9)

$$G_k={\displaystyle \sum }_{t=0}^n{H}_k\left(t\right)$$

(10)

where $n$ is the scale of the $k^{th}$ class gesture sequences; all of the joints $i$ in one hand are combined into a full hand skeleton $H_k\left(t\right)$ when the time scale of the $k^{th}$ class gesture is at $t$; $m$ represents the maximum number of joints in a full hand skeleton; the shape of the $k^{th}$ class gesture $G_k$ is processed into $n\times \left(am\right)$; the feature scale is $am$, and $a$ is the spatial scale.

The shape of the standard sequence is split into $n_1\times n_2\times \left(am\right)$ through the segmentation gestures (SG), where the $k^{th}$ class gesture forms $n_1$ sets of sequences and the time scale of each set is $n_2$.

Then, the skeleton-based hand gesture sequences can be mapped to the same specific interval by normalizing the changing hand joints, which is effective to improve the convergence rate of our network:

$${\mu }_B\leftarrow \frac{1}{m}{H}_k\left(t\right)$$

(11)

$${\sigma }_B^2\leftarrow \frac{1}{m}{\displaystyle \sum }_{i=1}^m{\left(P_{i,k}\left(t\right)-{\mu }_B\right)}^2$$

(12)

$${\widehat{P}}_{i,k}\left(t\right)\leftarrow \frac{P_{i,k}\left(t\right)-{\mu }_B}{\sqrt{{\sigma }_B^2+\epsilon }}$$

(13)

where ${\mu }_B$ is the mean of the sample and ${\sigma }_B^2$ is the sample variance; The linear transformation is added to these sequences and normalizes them to obtain ${\widehat{P}}_{i,k}\left(t\right)$, which limits the distribution of them and makes the network more stable during training; $\epsilon$ is the role of the minimum number, which avoids zero in the denominator in the expression.

The network may lose its original feature representation capabilities by the normalization. A pair of learnable parameters γ and β are set for each normalization to eliminate hidden dangers, which is used to restore the original distribution to obtain $Q_{i,k}\left(t\right)$.

$$Q_{i,k}\left(t\right)\leftarrow \gamma ·{\widehat{P}}_{i,k}\left(t\right)+\beta \equiv B{N}_{\gamma ,\beta }\left(P_{i,k}\left(t\right)\right)$$

(14)

In the formula, $B{N}_{\gamma ,\beta }\left(P_{i,k}\left(t\right)\right)$ is represented as a complete batch normalization (BN).

Additionally, the joint coordinates of the hand skeleton-based sequences are limited by the neighborhood, which increases the variance of the estimate and is not conducive to enhancing network learning. The average pooling layer (avgpool) can solve the above problems, which makes the structure of the skeleton-based sequence simpler and more stable, improves the calculation efficiency of the network, and avoids over-fitting during training. Here $Q_{i,k}\left(t\right)$ is introduced to represent the changes in the same joints of the adjacent multiple frames after the avgpool:

$${\widehat{Q}}_{i,k}\left(t\right)=\frac{1}{f^2}{\displaystyle \sum }_{t=t_0}^{t_0+2}\left(Q_{i,k}\left(t\right)\right)$$

(15)

$${\widehat{G}}_k={\displaystyle \sum }_{t=0}^{\widehat{n}}{\displaystyle \sum }_{i=1}^m{\widehat{Q}}_{i,k}\left(t\right)$$

(16)

where $f$ is the size of a filter of the average pooling layer; the size of $\widehat{n}$ is set to the equal of $n_1\ast \left(n_2/f\right)$; the shape of ${\widehat{G}}_k$ is split into $n_1\times (n_2/f)\times \left(am/f\right)$, which contains the features information of the $k^{th}$ class gesture, and as the input sequence of our network.

Finally, for one-shot learning, only a small part of the hand gesture datasets was taken as the training samples for subsequent experiments.

4.2.2. Implementation

The whole process of dynamic hand gesture recognition based on one-shot learning is shown in Figure 5.

Firstly, the M different classes are randomly selected from the N classes already contained in the dataset, which prevents the network from simply mapping class labels to the output. From the episode to the next episode, those classes presented in the current episode with the associated labels and specific samples will be shuffled. Later, the sample sequences are equally singled out from each of the M classes, which are supposed to be of the same size. Each group from the randomly re-labeled M classes extracts 10 sets of sequences as the training data at random. Of course, it is not enough to take merely 10 sets of the sequence for each training. Additionally, the corresponding batch size is taken by random sampling as the input of training. Then, the model will be validated by the validation set every $k$ epochs, and output the prediction accuracy with corresponding loss. Finally, the above processes are repeated until the model converges.

For the converged network model, the test set will be randomly selected to evaluate its generalization ability. After the test, the model’s ability to recognize those new unrecognized sequences will be the criterion of model selection.

According to the public DHGD hand gesture dataset, the time scale is set to 60 so that the size of the gesture sequences will be at least 100 sets in each class. After the data preprocessing, the shape of ${\widehat{G}}_k$ is split into $60\times 20\times 22$. For “one-shot learning”, 60%, 20%, and 20% of the data are used for the training set, the validation set, and the test set, respectively.

The DHGD dataset contains two different ways of 14-classes gestures: one finger and the whole hand. N is set to 14 as the number of the unique class; M is set to 3 as the number of sample classes; $k$ is set to 100 as the epoch-size in each training. For the 28-classes gestures encompassing the above two ways, N is set to 28 as the number of the unique class, while sizes of M and $k$ remain unchanged in each training. A grid search [45] is performed over a number of hyper-parameters: controller size (200 hidden units for an LSTM), the learning rate ($4\mathrm{e}-5$), the number of read–write heads from memory (4), and training times (80,000). For the 14-classes, the batch size is taken as 8, while it is set to 16 in the case of 28-classes. The model presents the best results over those hyper-parameters configurations.

In this study, another comparison experiment has been conducted based on the MSRA dataset. The time scale is also set to 12. After the data preprocessing, the shape of ${\widehat{G}}_k$ is segmented into $60\times 5\times 21$. Moreover, 50% of the data is used for the training set; 25% of the data utilized for the validation set; 25% of the data applied to the test set. For the MSRA dataset containing hand gestures of 17 classes, N is set to 17 as the number of the unique classes, and sizes of M and $k$ remain unchanged in each training. Compared with the 14-classes and 28-classes, hyper-parameters for the 17-classes are shown: controller size (200 hidden units for an LSTM), the learning rate ($4\mathrm{e}-5$), the number of read–write heads from memory (4), batch size (16), and training times (70,000).

5. Results and Discussion

To visualize the process of the recognition accuracy measured on the validation set, we have separately analyzed two different ways of 14-classes: one finger and the whole hand, and the 28-classes encompassing both the above two ways. In addition, the accuracy curve is shown in Figure 6.

From Figure 6, the 14-classes, (1) represents right-hand gestures performed with one finger, and (2) represents gestures with the whole hand. The curve of the one-finger classified by our method is shown in blue, the curve of the whole-hand is shown with an orange line, and the curve of the 28-classes is shown with a grey line. It is observed that the recognition accuracy of the 14-classes (2) is superior to the 14-classes (1), and the 28-classes is between those two. Compared with the 14-classes (1), the 28-classes has better performance.

To assess the effectiveness of our algorithm for classifying the hand gestures of DHGD into 14-classes and 28-classes, we compare the standard LSTM network with regard to their DHGD recognition accuracy.

Table 2. Comparison results between long short-term memory (LSTM) and gesture recognition using an enhanced network (GREN) networks based on the DHGD dataset.

Type		LSTM (%)	GREN (%)
14-classes	1	75.18	78.65
	2	79.82	85.90
28-classes	both	76.89	82.03

shows the comparison results of skeleton-based hand gesture recognition between LSTM and GREN networks.

From Table 2, the final accuracy of our GREN network reaches 82.29% for the 14-classes classification that is the average of the two ways and 82.03% for the 28-classes classification. The proposed network indicates that recognition accuracy can reach 78.65% for the one-finger and 85.90% for the whole-hand. Thus, compared with the standard LSTM networks, the accuracy of the recognition increased by approximately 5.14%, the accuracy of the one-finger increased by approximately 3.47%, and the whole-hand accuracy increased by 6.08%, which show excellent performance of our method in one-shot learning.

We compare the GREN network with the state-of-the-art algorithm in DHGD, and the results are shown in

Table 3. Result of different method comparison for 14/28-classes gestures on dynamic hand gesture dataset using skeleton-based data.

Learning	Methods	Accuracy 14-Classes Gestures	Accuracy 28-Classes Gestures
Large-samples	HON4D: Histogram of Oriented 4D Normals for Activity Recognition from Depth Sequences [46]	75.53%	74.03%
	3-D Human Action Recognition by Shape Analysis Of Motion Trajectories on Riemannian Manifold [47]	79.61%	62.00%
	Joint Angles Similarities and HOG2 for Action Recognition [48]	80.85%	76.53%
	Key Frames with Convolutional Neural Network [18]	82.90%	71.90%
	Skeleton-Based Dynamic Hand Gesture Recognition [49]	83.07%	79.14%
	NIUKF-LSTM [44]	84.92%	80.44%
	SL-Fusion-Average [36]	85.46%	74.19%
	MFA-Net [29]	85.75%	81.04%
One-shot	GREN	82.29%	82.03%

.

For the different ways of learning, a mature scheme of one-shot learning combined with hand gesture recognition has not been proposed before. Those advanced methods of comparison adopt the way of recognizing large size samples for experiments. While our GREN network uses small size samples in the DHGD dataset and trains based on one-shot learning.

Compared with other advanced algorithms, our method also performs well. For the 14-classes classification, the final accuracy of our GREN network is 82.29%, which is higher than most other algorithms. Additionally, our GREN network presents a higher accuracy in the 28-classes recognition than does that of the other advanced algorithm. A comparison of other advanced algorithms shows that the accuracy of the GREN network will not reduce significantly with the increase of the classes of hand gestures in the 28-classes recognition. Experimental results suggest that the proposed GREN network is an efficient method for hand gesture recognition.

Besides, to verify the robustness of the network, a similar experimental setup has also been performed on the MSRA hand gesture dataset. To more clearly demonstrate our network, we compared the experimental result with the LSTM network based on the MSRA dataset, which is shown in

Table 4. Comparison results between LSTM and GREN networks based on the MSRA dataset.

Type	LSTM (%)	GREN (%)
17-classes	72.92	79.17

.

From Table 4, the final accuracy of our network is 79.17% for the 17-classes classification. Additionally, compared with the LSTM networks, the accuracy of the recognition increased by approximately 6.25%, which shows the better performance of the GREN network. The experiment verifies that this network could be replicated for other similar datasets, even if they are small sample size datasets.

6. Conclusions

This paper proposes the GREN network to recognize dynamic hand gestures based on a small number of skeleton-based sequence samples. According to the MANN network, the ability to store and update sequence data is further enhanced by introducing the average pooling layer (avgpool) and batch normalization (BN), so that we can combine the hand skeleton sequence with the GREN network to achieve dynamic hand gesture recognition based on one-shot learning. Experiments with the DHGD hand gesture dataset demonstrate the state-of-the-art performance of the GREN network for skeleton-based dynamic hand gesture recognition based on one-shot learning. Additionally, the MSRA hand gesture dataset verifies the robustness of our GREN network.