TFS Point-on-Hand Sign Recognition Using Part Affinity Fields

Abstract

Our study investigates an application of a bottom-up design for keypoint regression, Part Affinity Fields (PAFs), for sign language recognition. Automatic sign language recognition could facilitate communication between deaf people and the hearing majority. Sign languages generally employ both semantic and finger-spelling signing. Semantic signing includes acting out to convey meaning, while finger spelling complements signing through the spelling out of proper names. Specifically, this article addresses an automatic recognition framework for the static point-on-hand (PoH) signing of Thai Finger Spelling (TFS)—the finger-spelling part of Thai Sign Language (TSL). From a pattern recognition perspective, PoH signing is quite distinct among signing schemes for requirement of precise localization of key parts on the signing hands. A recent study addressed PoH using an off-the-shelf version of MediaPipe Hands (MPH) and found shortcomings particularly when there was a high degree of hand-to-hand interaction. The top-down design of MPH was hypothesized to be the culprit. Our study investigates a bottom-up design, Part Affinity Fields (PAFs), along with examination of the related factors. The results support the hypothesis of a high-degree of hand-to-hand interaction posited by the MPH study. However, the overall performance of the PAF-based approach is shown to be modestly effective (72% accuracy vs. 58% and 47% of the MPH- and X-Pose-based approaches). In addition, its generalization is shown to be lacking. Thus TFS point-on-hand sign recognition remains a challenge.

1. Introduction

This article addresses automatic recognition of static point-on-hand (PoH) signing in Thai Finger Spelling (TFS), which is the spelling part of modern Thai Sign Language (TSL). PoH signing is quite distinct from signing gestures commonly used in other sign languages. It requires precise keypoint localization and involves a high degree of hand–hand interaction. These characteristics make PoH sign recognition uniquely challenging.

A recent study [1] pioneers a system based on MediaPipe Hands (MPH; [2]) to address a static PoH scheme, but only achieves moderate success. The study hypothesizes incompatibility between the top-down design of MPH and the high degree of hand–hand interaction in PoH signing to be a major cause of its limited success. The top-down design of MPH detects hands first. Then anatomical keypoints are localized in each detected hand. When there is a high degree of hand–hand interaction, the system cannot single out a cropped image of one hand without parts of another. The underlying assumption of performing localization on an image with a single hand is seriously violated. This is supposed to be a cause of significant deterioration in its effectiveness.

To advance the development of automatic sign recognition and further investigate the hypothesis on hand-to-hand interaction, our study examined one of the state-of-the-art methods in keypoint regression, Part Affinity Fields (PAFs; [3]), for application in static PoH sign recognition. The PAF method is based on a bottom-up design, localizing all keypoints and subsequently associating them. It is not based on the assumption of having a single hand in its working frames. Therefore, it was expected to mitigate hand-to-hand interaction issues.

Section 2 provides the context on TSL, TFS, and PoH signs. Section 3 reviews related studies. Section 4 gives a background on PAFs and other selected keypoint detection methods. Section 5 summarizes the proposed method. Section 6 describes our experiments and results. Section 7 concludes our investigation.

2. Sign Language and Thai Finger Spelling

Face-to-face communication is an important channel to convey messages, express feelings, and form personal connections. Deaf and mute people use two types of communication: reading–writing and sign language. Reading–writing is slower and more formal than signing. In addition, Antia et al. [4] report that deaf individuals are often slower than the hearing majority in reading and writing. Jepsen et al. [5] also discuss the negative suppression impact of sign languages on deaf people, including loss of self-esteem and exclusion from self-determination or political participation. Thus, even in our modern digital age, sign language remains germane.

Sign languages use a visual–gestural modality, which can be divided into semantic and spelling signs. Semantic signing uses hand and body gestures along with facial expressions to convey thoughts, meanings, attitudes, and feelings. Spelling signing, or finger spelling, uses finger and hand gestures to spell out a proper name or a word whose meaning has not yet been established in a particular language. Sign language is not a universally common language. Instead, sign languages can naturally emerge as deaf communities form [5]. Thai Sign Language (TSL, or ISO 639-3 tsq [6]) was not emerged naturally, it was developed from American Sign Language (ASL, or ISO 639-3 ase) in 1951 and currently found to be 52% similar. TSL is used in deaf communities throughout Thailand with regional lexical differences [7]. The National Association of the Deaf in Thailand and Ratchasuda College of Mahidol University are two authorities governing TSL. TSL status is categorized as developing by Ethnologue. Developing status (status code EGIDS 5) means “vigorous use, with literature in a standardized form being used by some though this is not yet widespread or sustainable” [6].

TSL was endorsed as the national sign language of Thailand in 1999, is adopted nationwide in deaf education, and is used by approximately 67,000 people [7]. TSL uses handshapes (about 58 shapes [7] (pp. 633–634)), orientations, locations (relative to body parts), movements, and non-manual expressions, including facial expressions to act out or symbolize meanings for semantic signing. TSL has both directional verbs and a semantic classifier system. TSL lexicons have been compiled for the first time in a publication as Thai Sign Language Dictionary: Book One [8]. TSL syntax arranges a simple sentence with subject–object–verb, in contrast to the subject–verb–object order used in the Thai spoken language. Modifiers (including negatives) are signed after the head word they modify. Word order and other grammatical issues in TSL are largely described by Danthanavanich [9]. While TSL semantic signing is quite distinct and very different from the Thai spoken language, its spelling part—Thai Finger Spelling (TFS)—is quite compatible with Thai written language, which has 66 alphabets representing 42 consonants, 32 vowels (there are 18 distinct alphabets representing vowels, and some vowels require a combination of multiple distinct alphabets), 4 intonation marks, and 2 special marks.

From a pattern recognition perspective, we categorize TFS into four signing schemes: static and dynamic single-handed (SH) schemes and static and dynamic point-on-hand (PoH) schemes, as shown in Figure 1, where signs are based on modern practice. (The main differences are in signs for Thai alphabets “E” and “E1”. An older signer may use a different practice, i.e., they may sign AFS “E” (E sign used in American Finger Spelling) followed by AFS “X” to represent Thai alphabet “E” and sign AFS “E” alone to represent Thai alphabet “E1”). Static and dynamic single-handed signing schemes are derived from American Sign Language. Single-handed signs are commonly observed in many sign languages. On the other hand, the static and dynamic PoH schemes are less common in other sign languages and have recently attracted attention in the pattern recognition community [1,10].

Figure 1. All TFS signs are organized into four different schemes. Each cell corresponds to a unique Thai alphabet. An alphabet in Thai script is shown on the top left of each cell, while an index for easy reference is shown on the top right. Dynamic schemes employ a series of (static) postures or movement.

The PoH signing schemes presented in the last two columns of Figure 1 are distinct because they use precise localization of hand keypoints. A static PoH gesture signs by pointing an index finger to a specific position on another hand without movement. The specific location on an outstretched palm determines the alphabet. Figure 2 illustrates the designated areas on the palm and the alphabet assigned by a static PoH scheme. A dynamic PoH scheme signs an alphabet in a similar way, but with a movement. Note that a signer can use either the left or right hand for both static and dynamic SH schemes, as well as switching hands in both PoH schemes.

Figure 2. (Left) PoH signs. (Right) designated areas of TFS signs in the static point-on-hand scheme. An alphabet in Thai script is shown above the signs, while an index is shown underneath.

3. Related Works

Finger-spelling recognition has been an active research area. Pinnington et al. [11] provided a review on the advancements of the recognition of American Sign Language (ASL) finger spelling. They summarize deep learning models, e.g., Convolutional Neural Networks [12] (CNNs), Recurrent Neural Networks [13] (RNNs), and Transformers [14], as the major approaches while emphasizing important factors for practical applications, including real-time performance under varied conditions, as well as quality and diversity of the dataset.

Thai Finger Spelling (TFS) has been adapted from ASL finger spelling, with substantial additions, particularly point-on-hand (PoH) signing. For TFS recognition, early works [15,16,17] address single-handed schemes covering the consonant alphabet. PoH signs covering vowels and intonations have recently been addressed. Two early studies on PoH signs [1,10] employ MediaPipe Hands (MPH) with marginal success. Sanalohit and Katanyukul [1] (addressing only static PoH) found performance deterioration for a case with a high degree of hand interaction. Vijitkunsawat and Racharak [10] (addressing both PoH schemes) found a serious accuracy drop when applying the models to out-of-sample data of static PoH signs. Out-of-samples are “unseen data under different backdrops and sources” [10]. Put simply, they are signs performed by new signers who were not involved in preparing the training samples. This notion emphasizes the signer’s role in the recognition performance.

Note that while Vijitkunsawat and Racharak [10] investigate all 90 TFS signs including both static and dynamic PoH signs, this dramatic drop in accuracy is found only in the static PoH signs.

Static and dynamic PoH schemes have received significantly less research attention than their single-handed counterparts. PoH schemes (Figure 1) are distinct because they (1) use both hands; (2) use the same pointing gesture for every PoH sign, but targeting different locations on the palm of the other hand; and (3) use movement, e.g., reverse finger flickering, where just after the pointing tip touches the finger on the open-palm hand, the touched finger curls in (“VV1” to “VV4” in the last column in Figure 1). These characteristics pose unique challenges not seen in single-handed schemes: (1) a recognition system must handle a high degree of hand-to-hand interaction; (2) it also needs more precise hand keypoint localization; and (3) it must account for movement, particularly when addressing dynamic PoH signing.

A pioneering study on PoH signing [1] investigated an approach that uses an off-the-shelf version of MediaPipe Hands [2]. The MPH-based system was found to be 60.4% accurate. Sanalohit and Katanyukul [1] hypothesized that the high degree of hand-to-hand interaction in PoH and MPH shortcomings in handling the issue are the main cause of the mediocre performance. They therefore concluded that an MPH-based approach is not suitable for the task because of (1) its top-down design involving hand detection followed by the localization of keypoints in each detected hand and (2) its underlying assumption that each localization operates on a cropped image of a single hand.

Later studies [10,18,19] pivot more to practical issues of deployment, particularly taking signing video as an input. Vijitkunsawat and Racharak [10] investigated various methods on a wider range of TFS signs, including static PoH signs. The study achieved noticeably more impressive performance, with an accuracy of 95%, using MEMP [20]. However, when the method was tested on out-of-sample images signed by new signers, not on those from the training dataset, its accuracy dropped sharply to 40%. This reveals a significant factor of signers as well as an issue of generalization. The authors attributed the performance drop to overlapping hand motions. This hypothesis seems to agree with the one discussed by Sanalohit and Katanyukul [1], as both employed MPH.

Emphasizing processing video input, Sungsri et al. [18] used a fusion-based convolutional recurrent neural network to address 42 signs (covering all 42 Thai consonants performed by single-handed signs). The fusion mechanism aims to synergize the spatial information processing of 2D CNNs, the spatiotemporal information processing of 3D CNNs, and the temporal information processing of bi-directional long short-term memory. This symbiosis led to an accuracy of 91.73% on single-handed signs. Chokchaitam et al. [19] used the MediaPipe Holistic Model along with various RNN models to classify semantic signs. They claimed 86% real-time accuracy on TSL semantic signs.

When put in perspective, semantic signs have received considerably less research attention than the finger-spelling counterpart. This may partly be attributed to a lack of an up-to-date authoritative/comprehensive dictionary for TSL semantic signs, which makes it difficult to properly define the scope of a study. Regarding finger-spelling recognition, the comprehensive work by Vijitkunsawat and Racharak [10] has shown a need for improvement particularly for static PoH sign recognition: an out-of-sample accuracy of 40% on static PoH, compared with 71.49% on a single hand with one stroke, 86% on a single hand with two strokes, 100% on a single hand with three strokes, and 81.33% on dynamic PoH.

Our work investigates a bottom-up architecture to address static PoH sign recognition. This study, based on a bottom-up architecture, aims to address static PoH sign recognition as well as answering the hypothesis theorized by Sanalohit and Katanyukul [1] (and somewhat advocated by Vijitkunsawat and Racharak [10]). In addition, our investigation may also provide a greater understanding of the interplay between architectural design and the choreography of the subjects.

4. Keypoint Detection

4.1. Part Affinity Fields

Part interaction and occlusion are not unique to PoH in a broader sense. Multi-person pose estimation also faces similar challenges. Given a color image $\mathbf{X}\in {\mathbb{R}}^{3\times w\times h}$ (presumably containing humans), pose estimation aims to localize body parts and the spatial dependencies among them.

A top-down design would detect persons: $\mathbf{X}\mapsto \{{\mathbf{d}}_1,\dots ,{\mathbf{d}}_N\}$, where detection box ${\mathbf{d}}_i$ presumably contains a person and N denotes the total number of detected persons. Then it localizes body keypoints in each detection box: ${\mathbf{d}}_i\mapsto \{{\mathbf{b}}_1,\dots ,{\mathbf{b}}_J\}$ for $i=1,\dots ,N$, where ${\mathbf{b}}_j$ is the predicted location of the $j^{th}$ part (or keypoint) among all J parts of interest. A top-down design can take advantage of techniques developed for a single-person scenario, but it fails to utilize a global context in capturing spatial dependencies across different people in a multi-person situation.

Conversely, a bottom-up design would first localize all body parts, then associate them with individual persons. Cao et al. [3] advocate a bottom-up design and propose OpenPose, whose key mechanism is Part Affinity Fields (PAFs). The bottom-up design has the advantage of a broader context for localization, but a multi-person scenario poses challenges in associating parts with the correct person.

The PAF mechanism solves the part-association problem by using a spatial representation of vector fields that encode pairwise relationships between body parts. These vector fields represent links between parts, and they provide essential clues for associating different parts. The OpenPose model is trained to provide (1) score maps whose score quantifies the likelihood of its corresponding body parts being in the location and (2) vector-field maps. Each vector-field map links two connecting body parts. The total number of maps is a combination of the number of body parts and the number of their links. Therefore, theoretically, OpenPose can handle a variable number of persons (given enough input resolution), and its runtime is virtually independent of the number of people in an input image, if not counting the runtime for the part-matching process.

Mechanism. Specifically, given a color image as an input $\mathbf{X}\in {\mathbb{R}}^{3\times w\times h}$, the OpenPose model predicts keypoint heatmaps and link vector fields. We denote keypoint heatmaps by $\widehat{\mathbf{S}}=\{{\widehat{\mathbf{S}}}_1,\dots ,{\widehat{\mathbf{S}}}_J\}$ for J keypoint types. Each heatmap ${\widehat{\mathbf{S}}}_j\in {[0,1]}^{w\times h}$ corresponds to anatomical keypoints of the $j^{th}$ type: we denote by ${\widehat{\mathbf{S}}}_j\left(\mathbf{p}\right)\in [0,1]$ a pixel value at location $\mathbf{p}$; a higher pixel value indicates higher confidence in the presence of the keypoint at $\mathbf{p}$. Hence, $\widehat{\mathbf{S}}\in {[0,1]}^{J\times w\times h}$.

Note “Openpose” may be used interchangeably with “PAF”. Original OpenPose declares ${\widehat{\mathbf{S}}}_j\in {\mathbb{R}}^{w\times h}$, but logically, it is perceived and trained to confine to $[0,1]$. Our preliminary investigation tested regulation using a sigmoid function but found no significant effect. OpenPose is intended for addressing a multi-person scenario. Keypoints of the same anatomical type can belong to multiple individuals.

Similarly, we denote link vector fields $\widehat{\mathbf{L}}=\{{\widehat{\mathbf{L}}}_1,\dots ,{\widehat{\mathbf{L}}}_C\}$ for C link types. Each field ${\widehat{\mathbf{L}}}_c\in {\mathbb{R}}^{2\times w\times h}$ corresponds to links of the $c^{th}$ type: we denote by ${\widehat{\mathbf{L}}}_c\left(\mathbf{p}\right)\in {\mathbb{R}}^2$ a link vector at location $\mathbf{p}$; a link vector at $\mathbf{p}$ provides association and direction between the two keypoints. A link between the two keypoints is expected to span multiple locations from one keypoint to another in a designated way. Link vector fields $\widehat{\mathbf{L}}\in {\mathbb{R}}^{C\times 2\times w\times h}$.

Hence, OpenPose frames the task as regression, $\mathbf{X}\mapsto \widehat{\mathbf{S}},\widehat{\mathbf{L}}$, using pre-trained VGG [21] and customized CNNs in a multiple-stage manner. That is,

• (1) Feature $\mathbf{Z}=\mathrm{vgg}\left(\mathbf{X}\right)$, where $\mathrm{vgg}(·)$ is implemented using the first 10 layers of VGG.
• (2) State-1 predictions ${\mathbf{S}}^{\left(1\right)}={\rho }^{\left(1\right)}\left(\mathbf{Z}\right)$ and ${\mathbf{L}}^{\left(1\right)}={\varphi }^{\left(1\right)}\left(\mathbf{Z}\right)$, where functions ${\rho }^{\left(1\right)}$ and ${\varphi }^{\left(1\right)}$ are implemented using customized CNNs. (See Cao et al. [3] for details.)
• (3) Refined predictions. That is, for all $t=2,\dots ,T$, where T is the number of stages, compute ${\mathbf{S}}^{\left(t\right)}={\rho }^{\left(t\right)}(\mathbf{Z},{\mathbf{S}}^{(t-1)},{\mathbf{L}}^{(t-1)})$ and ${\mathbf{L}}^{\left(t\right)}={\varphi }^{\left(t\right)}(\mathbf{Z},{\mathbf{S}}^{(t-1)},{\mathbf{L}}^{(t-1)})$, where functions ${\rho }^{\left(t\right)}$ and ${\varphi }^{\left(t\right)}$ are also implemented using customized CNNs.
• (4) The final outputs are $\widehat{\mathbf{S}}\equiv {\mathbf{S}}^{\left(T\right)}$ and $\widehat{\mathbf{L}}\equiv {\mathbf{L}}^{\left(T\right)}$.

Training. The OpenPose model is fine-tuned by minimizing regression loss $E={\sum }_{t=1}^T\left(E_S^{\left(t\right)}+E_L^{\left(t\right)}\right)$, where part loss $E_S^{\left(t\right)}={\sum }_{j=1}^J{\sum }_{\mathbf{p}}\mathbf{W}\left(\mathbf{p}\right)\left|\right|{\mathbf{S}}_j^{\left(t\right)}\left(\mathbf{p}\right)-{\mathbf{S}}_j^{\ast }\left(\mathbf{p}\right){\left|\right|}_2^2$ and link loss $E_L^{\left(t\right)}={\sum }_{c=1}^C{\sum }_{\mathbf{p}}\mathbf{W}\left(\mathbf{p}\right)\left|\right|{\mathbf{L}}_c^{\left(t\right)}\left(\mathbf{p}\right)-{\mathbf{L}}_c^{\ast }\left(\mathbf{p}\right){\left|\right|}_2^2$. Binary mask $\mathbf{W}$ accounts for a practical issue of ground-truth acquisition: $\mathbf{W}\left(\mathbf{p}\right)=0$ when annotation is missing at $\mathbf{p}$ and $\mathbf{W}\left(\mathbf{p}\right)=1$ when annotation is valid. Ground-truth part confidence maps $\{{\mathbf{S}}_1^{\ast },\dots ,{\mathbf{S}}_J^{\ast }\}$ and ground-truth part affinity vector fields $\{{\mathbf{L}}_1^{\ast },\dots ,{\mathbf{L}}_C^{\ast }\}$ are prepared from the annotated 2D keypoints.

Ground truth. We denote an annotated location of a type-j keypoint belonging to the $k^{th}$ person by ${\mathbf{a}}_{j,k}\in {\mathbb{R}}^2$. Ground-truth part confidence map ${\mathbf{S}}_j^{\ast }$ can be prepared such that

$$\begin{array}{cc}\hfill {\mathbf{S}}_j^{\ast }\left(\mathbf{p}\right)& =\underset{k}{max}exp\left(-{\displaystyle \frac{\left|\right|\mathbf{p}-{\mathbf{a}}_{j,k}{\left|\right|}_2^2}{{\sigma }_s^2}}\right),\text{for} \text{every} \text{location} \mathbf{p}\hfill \end{array}$$

(1)

where ${\sigma }_s$ is a user-specified parameter to account for fuzziness arising in human-made annotation.

Ground-truth vector fields are prepared from the idea that if location $\mathbf{p}$ is on the limb—linking two keypoints ${\mathbf{a}}_{j_1,k}$ and ${\mathbf{a}}_{j_2,k}$ then a vector at the location should be a unit vector pointing from $j_1$ to $j_2$. Vectors at locations not on the limb should be zero vectors. That translates to:

$$\begin{array}{c}\hfill {\mathbf{L}}_{c,k}^{\ast }\left(\mathbf{p}\right)=\left\{\begin{array}{cc}\mathbf{v}\hfill & \text{if} \mathbf{p} \text{on} \text{limb} c \text{of} \text{person} k,\hfill \\ \mathbf{0}\hfill & \text{otherwise}\hfill \end{array}\right.\end{array}$$

(2)

where the unit vector $\mathbf{v}=({\mathbf{a}}_{j_2,k}-{\mathbf{a}}_{j_1,k})/\left|\right|{\mathbf{a}}_{j_2,k}-{\mathbf{a}}_{j_1,k}{\left|\right|}_2$. Whether location $\mathbf{p}$ is on the limb can be tested by checking (1) whether the location lies between the keypoints in the direction of the link, i.e., $0\le \mathbf{v}·(\mathbf{p}-{\mathbf{a}}_{j_1,k})\le \left|\right|{\mathbf{a}}_{j_2,k}-{\mathbf{a}}_{j_1,k}{\left|\right|}_2$, and (2) whether the location lies within the limb width, i.e.,

$$\begin{array}{c}\hfill |{\mathbf{v}}_{\perp }·(\mathbf{p}-{\mathbf{a}}_{j_1,k})|\le {\sigma }_l\end{array}$$

(3)

where ${\mathbf{v}}_{\perp }$ is a unit vector perpendicular to $\mathbf{v}$. A limb width ${\sigma }_l$ is a user-specific parameter. Eventually, a vector field of all persons combines every individual field:

$$\begin{array}{cc}\hfill {\mathbf{L}}_c^{\ast }\left(\mathbf{p}\right)& ={\displaystyle \frac{1}{n_c\left(\mathbf{p}\right)}}\sum _k{\mathbf{L}}_{c,k}^{\ast }\left(\mathbf{p}\right).\hfill \end{array}$$

(4)

The factor $n_c\left(\mathbf{p}\right)$ is the number of non-zero vectors at $\mathbf{p}$ for link c of all persons.

Inference. Given the model outputs $\widehat{\mathbf{S}}$ and $\widehat{\mathbf{L}}$, locations of the keypoints can be deduced from $\widehat{\mathbf{S}}$ using non-maximum suppression. That is, $\{{\beta }_1,\dots ,{\beta }_{\kappa }\}=$$B_j=\mathrm{NonMaxSupp}\left({\widehat{\mathbf{S}}}_j\right)$, where ${\beta }_k\in {\mathbb{R}}^2$ is the kth detected location of keypoint j out of all $\kappa$ detections, $B_j$ is the set of all detections of type j and $\mathrm{NonMaxSupp}$ represents non-maximum suppression.

4.2. Other Keypoint Detection Methods

Keypoint detection or pose estimation is an active area of research [22,23]. Early approaches to keypoint detection rely on CNNs, e.g., MPH and PAFs, as discussed. More recent studies have explored approaches based on the vision transformer [24].

While translation equivariance and locality in CNNs offer inductive bias beneficial to vision tasks, a vision transformer relies primarily on efficient computation of a transformer architecture. Using image recognition as a representative for a vision task, Dosovitskiy et al. [24] show that when trained on a mid-sized dataset, a vision transformer underperforms its CNN-based counterparts. However, when trained on a large dataset (14M-300M images), a vision transformer can be very effective on a vision task with minimal inductive bias, a 2D structure of an image. Their vision transformer employs patch embedding along with a learnable class token and a simple 1D position embedding to convert a 2D-structure image into a sequence of tokens. Then, it processes the sequence with a transformer: a stack of multiple alternating layers between multi-layer-perceptron (MLP) layers and multi-head-self-attention (MSA) layers along with layer norm (LN), and residual connections. While MLP layers provide powerful non-linear projection within each token, MSA layers allow for effective association over short- and long-term relations among tokens. Computations of MSA and MLP are much more efficient than computing a CNN layer, in which matrix re-arrangement must be performed along with matrix multiplication to implement shared local connections.

Extending to keypoint detection, Xu et al. [25] investigated an application of the vision transformer to a pose estimation task and proposed ViTPose. ViTPose achieved the state of the art on the MS COCO Keypoint test-dev dataset at the time of its introduction. ViTPose uses a vision transformer as the encoder and has a decoder to restructure an encoded sequence; then it upsamples the restructured output and regresses the keypoint heatmaps.

In the meantime, as the application of computer vision gets highly diverse, Radford et al. [26] proposed Contrastive Language Image Pre-training (CLIP), a joint vision–language model. CLIP is pre-trained to match images to text captions. To achieve vision–language modalities, text and image encoders are jointly trained to maximize the cosine-similarity measures between the matched image and text embeddings and to minimize the measures between the unmatched pairs. Both objectives are carried out through contrastive learning [27]. This framework provides cross-modality text and image encoders, which together play a crucial role in associating text with image and vice versa in numerous subsequent works, including CLAMP [28] and X-Pose [29].

Yang et al. [29] proposed X-Pose, an end-to-end promptable multi-class keypoint detector. X-Pose allows either a textual or a visual prompt to specify object and keypoint categories to detect. Its structure is complex, but one of its key features is allowing for a multi-modal prompt. This feature is achieved using cross-modal image and text encoders of CLIP. A visual or textual prompt is encoded by the corresponding CLIP encoder to a compatible prompt embedding. The prompt embedding is then used to guide detection through multiple cross-attention operations. The model is trained using a combination of regression losses and alignment losses. Regression losses are used to regulate the precision of predicted locations of objects and keypoints. Alignment losses are formulated as contrastive losses to regulate the agreement between predicted outputs and the specified prompts. During training, the process randomly selects, at 50% probability, either a visual or textual prompt for each iteration. X-Pose has been described as a reliable keypoint detector for its generalization and high performance across various datasets.

As found by Dosovitskiy et al. [24], without inductive bias, a transformer-based vision model requires substantial data for training. Since our primary data source (PoH dataset) is quite small, the PAF method appears more suitable for the situation. The following section discusses the proposed PAF-based static PoH sign recognition strategy.

5. PAF-Based Static PoH Sign Recognition

Given an image of PoH signing $\mathbf{X}\in {\mathbb{R}}^{3\times w\times h}$, relevant hand keypoints are obtained using PAFs. The hand keypoints $\mathbf{K}=\{{\mathbf{k}}_1,\dots ,{\mathbf{k}}_J\}$, where J is the number of specified keypoints. The $i^{th}$ keypoint location ${\mathbf{k}}_i=[x_i,y_i]$, where $x_i$ and $y_i$ are predicted coordinates. Note that our task involves only one signer. Hence, each keypoint has only one instant, unlike a multi-person scenario in the original PAF work. The predicted locations of the keypoints are then processed to predict a PoH posture from the nearest point to the pointing tip. Specifically, the predicted PoH sign $i^{\ast }=arg{min}_{i\in P}\left|\right|{\mathbf{k}}_i-\mathbf{t}\left|\right|$, where $\mathbf{t}$ is a location of a pointing-tip keypoint and P is a set of sign indices corresponding to keypoint locations on the open palm. A sign in static PoH signing can be determined solely by the specific palm location to which the signer is pointing: a designated keypoint corresponds one to one to a static PoH sign. See Figure 2. The entire process is summarized in Table 1, where the functions $\mathrm{scale}$ and $\mathrm{pad}$ represent scaling and padding. The function kp-detect represents the keypoint detector (i.e., PAFs in our study). The function look-up represents a look-up function to map a keypoint index to a proper static PoH sign.

Table 1. Static PoH sign recognition.

1. input:	image $\mathbf{X}\in {\mathbb{R}}^{3\times \mathit{w}\times \mathit{h}}$
2. pad and scale:	$\mathbf{X}=\mathrm{scale}\left(\mathrm{pad}\right(\mathbf{X}\left)\right)$	# make image size uniform
3. keypoint detection:	$\{{\beta }_1,\dots ,{\beta }_{\kappa }\}=\text{kp-detect}\left(\mathbf{X}\right)$	# detect keypoints
4. role assignment:	let $\mathbf{t}={\beta }_1$	# location of the pointing tip
	let $P=\{j,\dots ,\kappa \}$	# indices of keypoints on the palm
5. sign recognition:	$i^{\ast }=arg{min}_{i\in P}\left|\right|{\beta }_i-\mathbf{t}\left|\right|$
	$\text{sign}=\text{look-up}\left(i^{\ast }\right)$
6. output:	static PoH sign

6. Experiments and Results

Our main goal was to evaluate an application of Part Affinity Fields (PAFs) to static point-on-hand (PoH) sign recognition. Mitigation of the effect from hand-to-hand interaction using a bottom-up design of PAFs and other related factors (ground-truth preparation and effect of a link mechanism) were also investigated.

Our experiments are run on a computer with CPU 2x Intel Xeon E5-2620 v4 2.10GHz, 16 cores/32 threads, ram 503 GB (Ubuntu 22.04, Linux 5.15.0-164-generic x86_64) and NVIDIA GeForce GTX 1080 Ti, 11 GB VRAM (NVIDIA 570.211.01, CUDA 12.8) using Python 3.12.4.

Dataset. Our investigation is based on a PoH dataset available on Kaggle at https://www.kaggle.com/datasets/s0ngkran/poh-vr-1k (accessed on 28 July 2024). The dataset provides one thousand 720 × 720-pixel color images, along with proper keypoint annotations (19 keypoints/image) and sign labels (covering all 11 PoH signs; Figure 2). The data are separated into subsets of 698, 153, and 149 images for training, validation and testing, respectively. Each subset is well balanced over all PoH signs.

Additionally, for a signer factor comparing to the work by Vijitkunsawat and Racharak [10], the PoH_DME dataset was used for our out-of-sample test. It is accessible through: https://www.kaggle.com/datasets/s0ngkran/poh-dme-clean (accessed on 16 January 2026). It provides 191 annotated color images (of size 720×720, covering 11 PoH signs) collected from 9 signers (different from those that participated in the construction of the PoH dataset).

To lessen the burden on our resources, all images (from both datasets) were scaled down to 360×360 pixels for use in all our experiments.

Experiments. All experiments were set up with the original PAF configuration but trained on our dataset with ground truths prepared using certain values for part and limb sigma’s (Section 4.1) ${\sigma }_s={\sigma }_l=23.2$. The model was trained with regression loss using Adam with a batch size of 10 and a learning rate of ${10}^{-4}$ for training periods ranging from 896 to 2174 epochs (six repeats); the training processes were actually run longer, but the models were selected based on criteria of achieving the best validation losses (validation losses were between 0.010353 and 0.012213 over six repeats). The data points were fed with dynamic augmentation: each training example was randomly chosen to be augmented with a 50% chance of using original data and a 50% chance of being augmented. For the augmented set, (1) a sample was rotated by $\theta$, where $\theta \in [-{45}^{\circ },{45}^{\circ }]$ was chosen randomly from a uniform distribution; (2) with a 50% chance, it underwent color jitter to adjust its brightness to $\alpha$, where $\alpha \in [0.8,1.2]$ was chosen randomly from a uniform distribution; (3) with a 10% chance, it was converted into grayscale (The number of color channels remained the same, 3.); (4) finally, with a 50% chance, it was normalized at each color channel. Each experiment was repeated six times.

Predicted signs were assessed against ground-truth sign labels, using classification accuracy as a performance metric.

Addressing static PoH. The effectiveness on application to PoH recognition is shown in Figure 3. The PAF-based system achieved an average accuracy of 72.26% over six repeats. (The best performing model achieved an accuracy of 81.21%.) This was compared to accuracies of 42.95% and 58.39% on the same test set using VGG and MPH, implemented as mentioned in the work by Sanalohit and Katanyukul [1]. (Our results are consistent with theirs.)

Figure 3. Static PoH recognition accuracy using VGG, MPH, and PAFs.

Table 2 shows the performance of the PAF-based approach, MPH, and a more recent detector, X-Pose. While our PAF-based approach was trained on the PoH dataset, MPH and X-Pose were used off the shelf without fine-tuning. X-Pose weights are from https://openxlab.org.cn/models/detail/IDEA-Research/UniPose/tree/main (file: unipose_swint.pth, accessed on 16 January 2026).

Table 2. Performance comparison.

Method	PoH Acc.	PoH DME Acc.	Time/Image (s.)
MPH	0.584	0.434	0.12–0.13
X-Pose	0.470	0.483	0.79–0.81
PAF	0.723	0.482	1.97–2.02

For each image, X-Pose was used with the same textual prompt: a standard hand prompt, along with 21 predefined keypoints in X-Pose inference (https://github.com/IDEA-Research/X-Pose/blob/master/inference_on_a_image.py, accessed on 16 January 2026). Figure 4 illustrates examples of results from these keypoint detectors.

Figure 4. Illustration of keypoints detected by different methods. MPH (left) is used off the shelf. PAF (middle) is trained on the PoH dataset. X-Pose (right) is used off the shelf with a standard prompt for “hand”. Red dots represent hand keypoints. Green lines represent links between keypoints. Blue dots and lines represent keypoints and links not related to TFS signs (additional outputs from off-the-shelf detectors).

Figure 5 illustrates examples of results of PAFs when the sign prediction is correct versus when it is incorrect. Note that the accuracy of sign prediction may not rigidly correspond to the accuracy of keypoint detection. Each PoH sign has only two main crucial keypoints. Those two keypoints are the pointing tip and the target keypoints. The sign prediction may still be accurate if the two main keypoints of the sign are detected well enough and detections of other keypoints are not too far off to interfere with them. Conversely, if the detections of the two main keypoints are too far off, the sign prediction will be inaccurate, regardless of how accurate the other detections are. Correct sign prediction (Figure 5, left) shows that detections of the two main keypoints were sufficient. Incorrect sign prediction (Figure 5, right) shows a similar level of accurate detections of most keypoints, but a detection of the target keypoint (shown as a blue dot) was too far off and caused incorrect sign recognition.

Figure 5. Illustration of keypoints detected by PAFs: correct sign prediction (left) versus incorrect sign prediction (right). Green dots represent annotated ground truths. Red dots represent predicted keypoints. A pink dot and a blue dot highlight the predicted keypoints of two main keypoints of the sign: the pointing tip (pink) and its target (blue). Signing on the left spells “V6”: the blue dot is on the “V6” position. Signing on the right intends to spell “V11”, but the detection of “V11” is too far off: the blue dot is too far under the true “V11” position.

Figure 6 shows box plots of the MDEs of correct and incorrect sign predictions, revealing a correlation between keypoint detection quality and sign recognition performance. The Mann–Whitney U test confirmed the difference between MDEs of correct and incorrect sign prediction (p-value of 1.237 × 10⁻⁵).

Figure 6. Mean distance errors and prediction correctness. Y-axis is log-scaled. A red line represents a median. A green triangle represents an average.

Mean distance error (MDE) is an average relative error, computed as follows:

$$\begin{array}{c}\hfill \mathrm{MDE}={\displaystyle \frac{1}{J}}\sum _j{\displaystyle \frac{{ϵ}_j}{D}}\times 100\%\end{array}$$

(5)

where ${ϵ}_j=\left|\right|{\mathbf{b}}_j-{\mathbf{a}}_j\left|\right|$ and J is a number of all keypoints, here $J=19$ (including wrists, elbows, and shoulders). Variables ${\mathbf{b}}_j$ and ${\mathbf{a}}_j$ are locations of the detected keypoint of type j and the corresponding ground truth. Reference distance D is defined as the distance between metacarpophalangeal (MCP) joints of the index finger and the middle finger, i.e., $D=\left|\right|{\mathbf{a}}_6-{\mathbf{a}}_7\left|\right|$, where ${\mathbf{a}}_6$ and ${\mathbf{a}}_7$ are annotated x-y coordinates of the two reference keypoints (V6 and V7 in Figure 2). Log mean distance error is computed as $log\left({\displaystyle \frac{1}{J}}{\sum }_j{\displaystyle \frac{{ϵ}_j}{D}}\right)$. For interpretation, $\mathrm{MDE}=1$ or $log\mathrm{MDE}=0$ means that the average error is about the length between the two closest keypoints on the open palm. $\mathrm{MDE}=0.5$ or $log\mathrm{MDE}\approx -0.69$ means that the average error is about half the length between the two closest keypoints.

PAFs on hand–hand interaction. To investigate the hypothesis that a bottom-up design is more resilient against hand-to-hand interaction, a bottom-up approach, PAFs, was compared to a top-down design, MPH. Accuracy on samples with different % overlap, defined by Sanalohit and Katanyukul [1], quantifies the hand-to-hand interaction in signing, as shown in Figure 7. Intermediate performance represented by log mean distance error is shown in Figure 8.

Figure 7. Accuracy of MPH and PAFs on different degrees of hand–hand interaction. Supports (#samples) are shown in brackets.

Figure 8. Log mean distance errors of MPH and PAFs on different degrees of hand–hand interaction. A red line represents a median. A green triangle represents an average.

MPH operates as a top-down workflow: When MPH cannot detect a hand or hands, hand keypoints are not located. In that case, MDE will be computed using ${ϵ}_j={ϵ}_{max}$; in our case ${ϵ}_{max}=\sqrt{{360}^2+{360}^2}$. For this reason, $\sqrt{{360}^2+{360}^2}$ appears to be the largest possible distance for a 360×360 image.

Note that even when MPH can detect one hand (out of two), all distance errors ${ϵ}_j={ϵ}_{max},\forall j$, since those keypoints from the detected hand cannot be properly matched to the ground truths using the hand-role determination rule, used by Sanalohit and Katanyukul [1]. An alternative is to use a cut-off distance from the annotation. Such cut-off distance can be a relative distance, chosen to be large enough for useful precision. With the cut-off distance, any localization error, either from a no-localization output or localization with an error exceeding the cut-off, could be set as the cut-off distance; i.e., ${ϵ}_j=\tau$ for no localization on the $j^{th}$ part and ${ϵ}_j=min\left\{\right||{\mathbf{b}}_j-{\mathbf{a}}_j\left|\right|,\tau \}$ when there is localization for the $j^{th}$ part. This alternative is more forgiving than the first option. However, choosing too small a cut-off may obscure what we want to investigate; choosing a large cut-off may lead to a similar outcome as the first option, but with a more complicated process to implement. Therefore, for simplicity, we have chosen to handle a no-localization output with the first option.

The Mann–Whitney U test confirms the difference between MDEs of MPH on a low degree and a high degree of overlapping (p-value of 0.0124), but does not support the difference between MDEs of PAFs on a low degree and on a high degree of overlapping (p-value of 0.1821). In addition, the Mann–Whitney U test does not support the difference between MDEs of MPH and PAF, both on a low degree of overlap (p-value of 0.0556). This supports our hypothesis that MPH, representing top-down designs, suffers from performance deterioration due to a high degree of hand-to-hand interaction. The performance of PAFs, representing bottom-up designs, does not differ significantly. This supports our hypothesis that the PAF method has performance comparable to MPH on low-degree hand-to-hand interaction, but it tolerates a high degree of hand-to-hand interaction better than MPH.

Factor of ground-truth sigma size. The preparation of ground-truth confidence maps (${\mathbf{S}}_j^{\ast }$, $\forall j$; Equation (1)) and vector fields (${\mathbf{L}}_c^{\ast }$, $\forall c$; Equations (2)–(4)) was crucial to the training process and consequently to the quality of the final model. The maps were prepared using Gaussian distributed values centered on the annotated locations. Part sigma ${\sigma }_s$ (Equation (1)) along with limb width ${\sigma }_l$ (Equation (3)) played a crucial role in controlling how much forgiveness was accommodated for the inherently fuzzy annotation.

To determine the appropriate setting, PAFs with different sigma sizes were investigated. This investigation set ${\sigma }_s={\sigma }_l=\sigma$, where $\sigma$ was experimented for six treatments: $\sigma =1x$, $\sigma =2x$, $\sigma =4x$, $\sigma =7x$, $\sigma =10x$, and $\sigma =13x$. The reference size x is the mean of D of all images: $x=11.6$ pixels. The results are shown in Figure 9.

Figure 9. Effects of using different sigma sizes. $x=11.6$ pixels. A red line represents a median. A green triangle represents an average.

The result indicated that $\sigma =2x$ ($=23.2$ pixels) was the optimal choice. That is, part sigma and limb width were set to roughly double the length between two close keypoints (MCP joints of the index and middle fingers; V6 and V7 in Figure 2).

Factor of ground-truth sigma multiplicity. Building on the previous investigation and on the observation that different limbs vary in width, we hypothesized that relaxing the single-sigma scheme may allow the ground truth to be more consistent with the sizes of the corresponding limbs and parts and could lead to improved part localization.

Previous investigations employed a single-sigma scheme, which uses a single value for limb width: $\sigma ={\sigma }_l={\sigma }_s$ for all limbs and parts. To relax this practice, ${\sigma }_l$ and ${\sigma }_s$ were allowed to differ, and each keypoint could have its own ${\sigma }_s$; and each link could have its own ${\sigma }_l$.

To preliminarily assess this idea, the MDE was obtained using a multiple-sigma scheme compared with the MDE of a single-sigma scheme under a similar setting. The single-sigma scheme used ${\sigma }_s={\sigma }_l=2x$ for all limbs and parts. The sizes of ${\sigma }_s$ and ${\sigma }_l$ were set to ostensibly fit the corresponding part or link: ${\sigma }_s\approx 3x$ for elbows and shoulders, ${\sigma }_s\approx 2.6x$ for wrists, ${\sigma }_s\approx 1.3x$ for the MCP joint of the index finger on the pointing hand, ${\sigma }_s\approx 1x$ for all other keypoints, ${\sigma }_l\approx 6x$ for the shoulder–elbow link, ${\sigma }_l\approx 5.2x$ for the wrist-to-elbow and shoulder-to-shoulder links, ${\sigma }_l\approx 4.3x$ for the wrist-to-palm-center and palm-center-to-MCP-middle-finger links, ${\sigma }_l\approx 3.4x$ for the MCP-index-finger-to-wrist link on the pointing hand, ${\sigma }_l\approx 2.6x$ for the wrist-to-MCP-thumb link, and ${\sigma }_l\approx 1.7x$ for all other links. Note that the annotation is point-wise. To create ground-truth maps (${\mathbf{S}}_j^{\ast },\forall j$ and ${\mathbf{L}}_c^{\ast },\forall c$), annotations had to be processed (Equations (1) and (4)). The rationale for ${\sigma }_s$ and ${\sigma }_l$ was to account for variation across human annotations. Our selections above were chosen to make ground-truth maps closely correspond to the sizes of parts and links as much as possible. Figure 10 illustrates these choices. Note that we aim to put our choice into perspective. The real effects of ${\sigma }_s$ and ${\sigma }_l$ are more complex: ${\sigma }_s$ affects the Gaussian span in a part heatmap; ${\sigma }_l$ affects link width in a vector field.

Figure 10. Sigma multiplicity: sigma size is shown as link width (A) for a link or as circle radius (B) for a keypoint.

Figure 11 shows the results. The Mann–Whitney U test does not indicate a difference between the MDEs of the two schemes (p-value of 0.905). The average MDEs were 96.719% and 465.835% for the single- and multiple-sigma schemes, respectively. This preliminary result ruled out further investigation of the multiple-sigma hypothesis.

Figure 11. Effect of sigma multiplicity. A red line represents a median. A green triangle represents an average.

Effect of link mechanism. Recall that the main purpose of link vector fields is to guide a matching process in a multi-person scenario. A sign recognition scenario, at least at this stage, is assumed to have a single signer clearly seen in the viewing frame without external occlusion. This scenario eliminates the need for a matching process. However, training the model to simultaneously predict both part confidence maps and link vector fields may synergize the process; otherwise, processing link vector fields consumes additional resources for no reason.

This study aimed to examine a linkage mechanism in a PoH recognition scenario. The PAF-based system (used in previous experiments: ${\sigma }_s={\sigma }_l=2x$) served as the control treatment for comparison. The treatment being studied was a similar system, but without a link mechanism. It was trained on ground truth prepared using ${\sigma }_s=2x$. Figure 12 shows the result. The Mann–Whitney U test could not confirm a difference between the control (using a link mechanism) and the underlying treatment (using no link mechanism), with a p-value of 0.9307. The average MDEs were 96.719% and 96.497%, obtained from the control and the underlying treatment, respectively. The ablation result is shown in Table 3.

Figure 12. Effect of link mechanism. A red line represents a median. A green triangle represents an average.

Table 3. Ablation results.

Link	MDE	PoH Acc.	Time/Image (s.)
✓	0.967	0.723	1.97–2.02
(no link)	0.965	0.749	0.51–0.53

This finding suggests that in a single-person scenario, a link mechanism does not contribute much to the overall effectiveness of a PAF system. Together with the hand-to-hand interaction investigation, it was revealed that the PAF system decisively outperformed the MPH system, which does not have a link mechanism as a factor; we gained stronger evidence to support the advantage of a bottom-up design over a top-down design in hand-to-hand interaction.

Additional finding. While examining prediction errors obtained from our PAF system, we identified notable errors in wrist localization. Further investigation on this issue revealed an association between wrist localization error and final prediction performance, as shown in Figure 13.

Figure 13. Wrist localization errors and the prediction correctness. Y-axis is log-scaled. A red line represents a median. A green triangle represents an average.

Figure 13 illustrates data in two groups: (1) samples with incorrect sign prediction and (2) samples with correct sign prediction. It is apparent that when the PAF system makes an incorrect prediction, Wrist Distance Error (WDE) appears larger than when it gets the prediction right. Similar to MDE, WDE is calculated as: $\mathrm{WDE}={\displaystyle \frac{\left|\right|{\mathbf{b}}_w-{\mathbf{a}}_w\left|\right|}{D}}\times 100\%$, where ${\mathbf{b}}_w$ is the predicted location for the wrist and ${\mathbf{a}}_w$ is the corresponding ground truth. The Mann–Whitney U test confirms the difference, with a p-value of 2.545 × 10⁻⁶. The average WDEs are 3.47 and 1.0217 in the incorrect prediction and correct prediction groups, respectively. At this point, it is too early to speculate on any factor that might contribute to this wrist localization error. We have made this observation for a potential further study.

Generalization. The trained PAF-based system was tested on an out-of-sample PoH_DME dataset, whose signers were not involved in the training set. This was to examine the effect of different signers, as emphasized by Vijitkunsawat and Racharak [10]. Figure 14 presents the result. The average accuracy for the in-sample test was 72.2%, while for the out-of-sample test, it was 48.2%. This should lead to a drop from 95% to 40% by using the MEMP approach [10]. Our result agreed with that obtained by Vijitkunsawat and Racharak [10]. Since generalization is essential in practical applications, it should be assessed along with model accuracy.

Figure 14. Prediction accuracy on test with in-samples and out-of-samples (new signers). A red line represents a median. A green triangle represents an average.

Besides the signer factor, in-sample images have a natural background and indoor-lighting condition, while out-of-sample images feature a natural background, but with mixed lighting conditions. Some images were taken with indoor lighting, and some with outdoor lighting. This difference may also have contributed to such findings. Separating these factors may require dedicated studies employing well-designed methods and experiments.

7. Conclusions and Discussion

This study addressed static PoH recognition using PAFs, achieving an average accuracy of 72.26%, compared with the 58.39% accuracy obtained by Sanalohit and Katanyukul [1] using MPH. This performance remains inadequate for practical use. In addition, a test on different datasets showed a significant drop in accuracy, indicating limited model generalization. This generalization issue may stem from various factors, including signers, background, and lighting conditions. These factors influence hue, sharpness, contrast, background noise, as well as high-level patterns, e.g., shape, in a signing image. Without a mechanism that accounts for these factors, generalization and practical application are difficult to achieve.

The PAF method itself lacks a mechanism specifically designed to handle these issues. However, concepts such as denoising and contrastive learning could mitigate the problem and may offer a better approach to learned representation. Contrastive learning gives a framework to find sound representation mapping by maximizing agreement in a positive pair and minimizing agreement in a negative pair. Along with denoising and data augmentation, contrastive learning can also be used to enhance feature learning in various ways. PSSCL [30] and UCRT [31] employ contrastive learning to handle noisy labels in a conventional classification task. Our PAF-based method does not classify signs in a conventional way: it depends on keypoint detection and predicts signs based on the distance between designated keypoints. However, the backbone of PAFs, e.g., VGG in our study, can benefit from improved PSSCL or UCRT pre-training. However, addressing PoH recognition may not need to follow this pipeline of keypoint detection followed by keypoint-distance-based rules. Another potential use of contrastive learning is to develop a robust feature-producing model, before fine-tuning for a downstream task, such as keypoint detection. E.g., similar to Chen et al. [27], variation could be added to the original images for pre-training the model to recognize augmented images of the same origin. In addition, hand and finger occlusion could be treated directly through contrastive learning, e.g., pre-training the model to recognize occlusion or the same keypoint under a variety of occlusion types. This could be a promising research direction, tackling a challenge in PoH sign recognition.

In addition to contrastive learning, high-level prior knowledge can be utilized: E.g., PoH signing involves one hand in an open palm pose, while the other is in a pointing pose. This posture and hand anatomy could be used, for example, as an inductive bias in a model structure or as loss regularization during training.

Regarding the overall view of PoH sign recognition, MPH and X-Pose did not produce better outcomes than our PAF-based method. This reveals a challenge in PoH sign recognition. Although these notable detectors are used off the shelf, there have been reports of impressive results on various datasets. Figure 4 and Figure 5 suggest some causes of the difficulty: while many body keypoints appear to be detected with sufficient accuracy, hand and finger keypoints are detected with much lower accuracy. Self-occlusion and finger articulation may play a large part in the difficulty. Beyond PoH sign recognition, a complete recognition for TSL requires significant work, from a reliable generalized solution for static and dynamic PoH, integration of all TFS recognitions, semantic signing, and the final integration. That is a long road ahead with many challenges in both computing and linguistic aspects.

Regarding architecture design, our investigation supports the bottom-up hypothesis: a bottom-up design manages hand-to-hand interaction more effectively than a top-down design. This is supported by (1) the fact that the PAF system delivers more stable performance across different levels of hand–hand interaction and is thus more effective than the MPH system (Figure 7 and Figure 8) and (2) the fact that an extra link mechanism does not significantly contribute to the effectiveness of the PAF system (Figure 12). This study also investigates practical issues and proposes a single-sigma scheme with a sigma value that should be approximately twice the distance between the two closest keypoints, e.g., the MCP joints of the index and middle fingers. Additionally, the association between wrist localization error made by the PAF system and the final prediction performance is observed. Generalization remains an issue. The choice of the recognition model, recognition performance, the underlying links among signers, signing background, lighting condition, and their effects, as well as mitigation or remedy, deserve dedicated study and are crucial to advancing toward practical application.

While this study seems to advocate a bottom-up design over a top-down structure, these two approaches may not always be mutually exclusive. A creative combination of top-down and bottom-up designs could produce exceptional synergy.