Review Reports - TFS Point-on-Hand Sign Recognition Using Part Affinity Fields

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

This manuscript is dedicated to solving the urgent problem of automatic recognition of Thai Dactylic Alphabet (TFS), in particular the specific Point-on-Hand (PoH) scheme. The authors investigate the limitations of existing top-down approaches, such as MediaPipe Hands (MPH), in situations with intensive interaction between hands and propose the use of Part Affinity Fields (PAF) as a bottom-up architecture. The methodology is based on the use of OpenPose for keypoint regression, and the results demonstrate a significant improvement in recognition accuracy (from 58% to 72.26%) compared to the baseline models.

A detailed critical analysis of the structure and content of the presented work is given below.

Critical remarks:

The Abstract section (page 1, lines 14-15) states an accuracy of 72%. Although this is better than MPH, for modern computer vision systems, this indicator is critically low for practical use. The authors should clearly justify why this result is considered successful, or acknowledge this as a significant limitation of the proposed model in its current state.
The dataset size (1000 images for 11 characters, page 8, line 284) is too small for deep learning by modern standards. Such a sample size can lead to overfitting and the model’s inability to generalize to real data. Justification for the representativeness of the sample or augmentation methods should be provided.
Section 4 (Part Affinity Fields) contains too much well-known theory from the original paper by Cao et al. (2019). The authors should shorten the description of the basic OpenPose equations (Eq. 4-11) and focus on specific modifications or parameters that were adapted specifically for the recognition of Thai PoH gestures on pages 6-8.
In Figure 1 (page 2), the gesture images are too small, which makes it impossible to analyze the key points discussed in the text in detail. Since the article is devoted to precise localization, it is necessary to provide enlarged examples of complex PoH gestures with key point skeletons superimposed to demonstrate the work of PAF.
The authors claim the advantage of the bottom-up design, but the comparison is made only with one system, MPH. For scientific validity, it is necessary to add a comparison with other modern hand detectors or alternative architectures (for example, ViT-based pose estimation) to confirm the effectiveness of PAF at this time.
In section 5 (page 9, line 302), the description of the pose prediction process by the “closest point to the tip of the index finger” is oversimplified. A classification algorithm or pseudocode should be provided that explains how the keypoint vectors are transformed into a class label (letter of the alphabet) after the heatmaps are obtained.
The MDE (Mean Distance Error) metric value in Figure 5 (page 10) is difficult to interpret without reference to the image resolution in physical units or percentages. It should be clearly stated what pixel error is critical for correct character recognition, so that the reader can estimate the practical error of the system.
The Mann-Whitney test results on page 10 (lines 337-340) indicate that there is no statistically significant difference for PAF between low and high degrees of hand overlap. However, the authors use this as evidence of “robustness”. This is a logical fallacy: no difference does not always mean high quality; it could mean equally mediocre performance in both cases.
The use of the term “out-of-samples” on pages 4 and 12 needs to be more clearly defined. The authors should clarify whether this data includes other users, different lighting, or different backgrounds. Without detailing the shooting conditions for the validation dataset, the 48.2% accuracy result (page 13) looks like a model failure.
Section 5. Experiments (page 8) lack a description of the training hyperparameters (learning rate, batch size, optimizer, number of epochs). Without this data, the experiment cannot be reproduced, which is contrary to the principles of open science and the requirements of the MDPI Applied Sciences journal for describing the methodology.
Page 10 (lines 331-332) mentions the use of the maximum error ϵ_max in cases where MPH does not detect a hand. This approach can artificially inflate the error for MPH, creating an unequal comparison. The authors should review the methodology for handling missed detections to ensure objectivity of the comparative analysis.
Formula 12 (page 9) contains the variable D as a “reference distance”. It is necessary to clearly explain how D is calculated for each frame (for example, the distance between certain joints), since changing the scale of the hand in the image will significantly affect the relative value of the MDE.
The authors note on page 11 (lines 364-370) that the dimensions σ_s and σ_l were set “arbitrarily”. This approach undermines scientific rigor. It is necessary to provide the results of an ablation study or explain the logic of choosing these values based on the anatomical features of the hands.
Importantly, in section 6 (Conclusions), the authors indicate an accuracy of 72.26%, but do not explain whether this is sufficient for real-world communication. Given that a single letter error can change the meaning of a word, such a low figure requires critical discussion in the context of the usability of the system. This section should clearly highlight the main scientific contributions of the study (contributions) so that editors and readers can quickly assess the novelty of the work.
The list of references contains references to works from 1986, 1990, and 2002 (lines 448, 451, 468). Although these are classic works, the Related Works section should be more focused on SOTA solutions from 2023-2025, especially in the field of Transformers and Graph Neural Networks for gesture recognition.
The text on page 12 (line 382) mentions the use of a PAF system with parameters σ_s=σ_l=23.2. However, Figure 6 shows that the best result is given by σ=2x. There is a discrepancy between the text description and the visualization of the results, which needs to be corrected.
There is no analysis of the computational complexity (runtime performance). Since PAF is a heavy architecture, it is important to know the frame rate (FPS) on standard hardware, especially compared to MediaPipe, which is positioned as a real-time solution (page 12, lines 389-393).
Line 218 on page 7 describes the binary mask W(p). The authors should clarify how this mask is generated during testing if annotations are missing. If it is used only during training, this should be clearly delineated in the description of the inference architecture.
The conclusion on page 13 (lines 425-427) about the relationship between wrist localization error and final accuracy is interesting, but not supported by sufficient statistics. A correlation analysis between the MDE of keypoints and the probability of correct classification of a sign should be added.
The paper does not discuss the effects of background and lighting, although the dataset contains images with different conditions (Kaggle dataset). Since PoH gestures require “precise localization”, the sensitivity of PAF to background noise should be analyzed in the Discussion section.
The text contains technical OCR or LaTeX artifacts (e.g., “cf.” on line 158 on page 5, or a reference to “citetVR2024” on line 428 on page 13). The manuscript should be carefully checked for citation formatting errors.

The main reasons for the rejection are the insufficient accuracy of the system, the critically small training data set, and the lack of comparison with modern SOTA models outside of MediaPipe. The work requires significant refinement of the experimental part and methodology.

Author Response

``The Abstract section (page 1, lines 14-15) states an accuracy of 72\%. Although this is better than MPH, for modern computer vision systems, this indicator is critically low for practical use. The authors should clearly justify why this result is considered successful, or acknowledge this as a significant limitation of the proposed model in its current state.''

The abstract has been reviewed. Thank you for your suggestion.
``The dataset size (1000 images for 11 characters, page 8, line 284) is too small for deep learning by modern standards. Such a sample size can lead to overfitting and the model's inability to generalize to real data. Justification for the representativeness of the sample or augmentation methods should be provided.''

Section 6 Experiments and Results has been edited to describe the augmentation used in the training process. Thanks for pointing this out.
``Section 4 (Part Affinity Fields) contains too much well-known theory from the original paper by Cao et al. (2019). The authors should shorten the description of the basic OpenPose equations (Eq. 4-11) and focus on specific modifications or parameters that were adapted specifically for the recognition of Thai PoH gestures on pages 6-8.''

We have revised Section 4. Thanks for pointing this out.
``In Figure 1 (page 2), the gesture images are too small, which makes it impossible to analyze the key points discussed in the text in detail. Since the article is devoted to precise localization, it is necessary to provide enlarged examples of complex PoH gestures with key point skeletons superimposed to demonstrate the work of PAF.''

Figure 1 has been re-drawn and enlarged illustration of PoH signing is added to Figure 2.
``The authors claim the advantage of the bottom-up design, but the comparison is made only with one system, MPH. For scientific validity, it is necessary to add a comparison with other modern hand detectors or alternative architectures (for example, ViT-based pose estimation) to confirm the effectiveness of PAF at this time.''

A performance of a modern pose estimator --- X-Pose --- has been added in Table 2.
``In section 5 (page 9, line 302), the description of the pose prediction process by the “closest point to the tip of the index finger” is oversimplified. A classification algorithm or pseudocode should be provided that explains how the keypoint vectors are transformed into a class label (letter of the alphabet) after the heatmaps are obtained.''

The description has been revised and put in a dedicated section (Section 5) along with Table 1 to elaborate the entire process. Thanks for pointing this out.
``The MDE (Mean Distance Error) metric value in Figure 5 (page 10) is difficult to interpret without reference to the image resolution in physical units or percentages. It should be clearly stated what pixel error is critical for correct character recognition, so that the reader can estimate the practical error of the system.''

MDE is an average relative error (Equation 5). The relative error is to mitigate the effect of an image scale. The corresponding paragraph (below Equation 5) has been edited to clarify this point. In addition, an example of MDE interpretation has been added. Thanks for pointing this out.
``The Mann-Whitney test results on page 10 (lines 337-340) indicate that there is no statistically significant difference for PAF between low and high degrees of hand overlap. However, the authors use this as evidence of “robustness”. This is a logical fallacy: no difference does not always mean high quality; it could mean equally mediocre performance in both cases.''

We have fixed it. Now the interpretation reads "PAF has a comparative performance to that of MPH in a low degree of hand-hand interaction, but it can tolerate against a high degree of hand-hand interaction better than MPH can."
``The use of the term “out-of-samples” on pages 4 and 12 needs to be more clearly defined. The authors should clarify whether this data includes other users, different lighting, or different backgrounds. Without detailing the shooting conditions for the validation dataset, the 48.2\% accuracy result (page 13) looks like a model failure.''

A paragraph at the end of Section 6 has been added to provide the detail on lighting conditions. Thanks for pointing this out.
``Section 5. Experiments (page 8) lack a description of the training hyperparameters (learning rate, batch size, optimizer, number of epochs). Without this data, the experiment cannot be reproduced, which is contrary to the principles of open science and the requirements of the MDPI Applied Sciences journal for describing the methodology.''

The description of the training hyperparameters as well as other training detail has been added to the Section. (The experiment section has become Secton 6, due to addition of Section 5 to clarify sign decision.) Thanks for pointing this out.
``Page 10 (lines 331-332) mentions the use of the maximum error $\epsilon_{\max}$ in cases where MPH does not detect a hand. This approach can artificially inflate the error for MPH, creating an unequal comparison. The authors should review the methodology for handling missed detections to ensure objectivity of the comparative analysis.''

We have added a discussion about an alternative to the paragraph. Thanks for pointing this out.
``Formula 12 (page 9) contains the variable D as a “reference distance”. It is necessary to clearly explain how D is calculated for each frame (for example, the distance between certain joints), since changing the scale of the hand in the image will significantly affect the relative value of the MDE.''

We have elaborated the paragraph to clarify how reference distance $D$ is obtained. Thanks for pointing this out.
``The authors note on page 11 (lines 364-370) that the dimensions $\sigma_s$ and $\sigma_l$ were set “arbitrarily”. This approach undermines scientific rigor. It is necessary to provide the results of an ablation study or explain the logic of choosing these values based on the anatomical features of the hands.''

We have revised the paragraph and added Figure 10 to illustrate our choice.
``Importantly, in section 6 (Conclusions), the authors indicate an accuracy of 72.26\%, but do not explain whether this is sufficient for real-world communication. Given that a single letter error can change the meaning of a word, such a low figure requires critical discussion in the context of the usability of the system. This section should clearly highlight the main scientific contributions of the study (contributions) so that editors and readers can quickly assess the novelty of the work.''

The conclusions have been revised.
``The list of references contains references to works from 1986, 1990, and 2002 (lines 448, 451, 468). Although these are classic works, the Related Works section should be more focused on SOTA solutions from 2023-2025, especially in the field of Transformers and Graph Neural Networks for gesture recognition.''

Section 4.2 is added to provide additional review on more recent keypoint detection. Thanks for suggestion.
``The text on page 12 (line 382) mentions the use of a PAF system with parameters $\sigma_s = \sigma_l=23.2$. However, Figure 6 shows that the best result is given by $\sigma$=2x. There is a discrepancy between the text description and the visualization of the results, which needs to be corrected.''

Figure caption has been clarified. Due to additional figures, the figure has now become Figure 9. Thanks for pointing this out.
``There is no analysis of the computational complexity (runtime performance). Since PAF is a heavy architecture, it is important to know the frame rate (FPS) on standard hardware, especially compared to MediaPipe, which is positioned as a real-time solution (page 12, lines 389-393).''

Table 2 has been added to present elapse times along with other measures.
``Line 218 on page 7 describes the binary mask W(p). The authors should clarify how this mask is generated during testing if annotations are missing. If it is used only during training, this should be clearly delineated in the description of the inference architecture.''

Section 4 has been revised. A role of the binary mask $W(p)$ should be clearer with this revision.
``The conclusion on page 13 (lines 425-427) about the relationship between wrist localization error and final accuracy is interesting, but not supported by sufficient statistics. A correlation analysis between the MDE of keypoints and the probability of correct classification of a sign should be added.''

Figure 6 along with significance test have been added.
``The paper does not discuss the effects of background and lighting, although the dataset contains images with different conditions (Kaggle dataset). Since PoH gestures require `precise localization', the sensitivity of PAF to background noise should be analyzed in the Discussion section.''

The discussion has been revised. Thanks for point it out.
``The text contains technical OCR or LaTeX artifacts (e.g., `cf.' on line 158 on page 5, or a reference to `citetVR2024' on line 428 on page 13). The manuscript should be carefully checked for citation formatting errors.''

These have been fixed. Thanks for pointing this out.

Reviewer 2 Report

Comments and Suggestions for Authors

In the manuscript titled " TFS Point-on-Hand Sign Recognition using Part Affinity Fields", No.: applsci-4099730, authors investigate an application of a bottom-up-design key point regression, Part Affinity Fields (PAF) to sign language recognition. It achieves significant improvements compared to state-of-the-art (SOTA) methods. However, I have some concerns outlined below:

The paper mainly compares the proposed PAF-based approach with VGG and MediaPipe Hands (MPH). However, the set of baselines is still limited. It is recommended that the authors include additional quantitative comparisons with more recent or representative keypoint detection or hand gesture recognition methods.
It would be beneficial to include qualitative visualizations, such as predicted keypoints and PAF vector fields overlaid on test images, or comparisons between correct and incorrect predictions.
The paper touches on issues related to annotation inaccuracies, signer variability, and performance degradation, which can be partially interpreted as challenges arising from noisy or inconsistent labels. It is recommended that the authors cite and discuss recent representative works on noisy-label learning and robust training, such as psscl: a progressive sample selection framework with contrastive loss and urct: a two-stage noisy label learning framework with uniform consistency selection and robust training, in the related work or discussion section. In addition, a brief discussion on whether ideas such as contrastive learning, or consistency-based robust training could be integrated with the current PAF-based PoH recognition framework would further strengthen the paper and point to promising future research directions.
Although the paper demonstrates the effectiveness of PAF for static PoH recognition, the scope of applicability could be clarified further. It is recommended that the authors more explicitly discuss the limitations of the proposed approach.

Author Response

``The paper mainly compares the proposed PAF-based approach with VGG and MediaPipe Hands (MPH). However, the set of baselines is still limited. It is recommended that the authors include additional quantitative comparisons with more recent or representative keypoint detection or hand gesture recognition methods.''

X-Pose has been added to a set of the baselines. Thank you for your suggestion.
``It would be beneficial to include qualitative visualizations, such as predicted keypoints and PAF vector fields overlaid on test images, or comparisons between correct and incorrect predictions.''

Figure 5 has been added.
``The paper touches on issues related to annotation inaccuracies, signer variability, and performance degradation, which can be partially interpreted as challenges arising from noisy or inconsistent labels. It is recommended that the authors cite and discuss recent representative works on noisy-label learning and robust training, such as psscl: a progressive sample selection framework with contrastive loss and urct: a two-stage noisy label learning framework with uniform consistency selection and robust training, in the related work or discussion section. In addition, a brief discussion on whether ideas such as contrastive learning, or consistency-based robust training could be integrated with the current PAF-based PoH recognition framework would further strengthen the paper and point to promising future research directions.''

That is a very interesting subject. The discussion has been revised. Thank you for your suggestion.
``Although the paper demonstrates the effectiveness of PAF for static PoH recognition, the scope of applicability could be clarified further. It is recommended that the authors more explicitly discuss the limitations of the proposed approach.''

The discussion has been revised. Thanks for pointing this out.

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

The authors took into account almost all of my comments, and the article has improved.

Reviewer 2 Report

Comments and Suggestions for Authors

The authors have fully addressed all my concerns; I recommend acceptance of the paper.

Comments on the Quality of English Language

The authors have fully addressed all my concerns; I recommend acceptance of the paper.