TransMODAL: A Dual-Stream Transformer with Adaptive Co-Attention for Efficient Human Action Recognition

Comments 1: Please justify the selection of RT-DETR as a person detection model used in the research. Why not any other form yolo family, RCNN. Is the RT-DETR easy replaceable by other model in the proposed architecture? Same question for Vitpose++ part. Can it be yolo-pose, pose-net, open-pose?

Response 1: We thank the reviewer for this important question. We selected RT-DETR and ViTPose++ based on their state-of-the-art performance and efficiency at the time of this research. To clarify our rationale and the modularity of our approach, we have added a new subsection, "3.1.1. Upstream Model Selection and Hyperparameter Justification," which details these choices. We explicitly state that the pipeline is modular and other high-performing detectors or pose estimators could be substituted. In this new section, we clarify that:

1. RT-DETR was chosen for its excellent balance of high accuracy and real-time performance, representing the state-of-the-art in DETR-based object detectors.
2. ViTPose++ was selected as a top-performing, transformer-based model for pose estimation, aligning well with the overall architecture.
3. We explicitly confirm that our pipeline is modular. Both the detector and the pose estimator function as pre-processing steps and can be readily substituted with other high-performing models (such as those from the YOLO family or OpenPose), provided they output the required bounding boxes and 17-channel COCO keypoints. Mention exactly where in the revised manuscript this change can be found – page 5, paragraph 4, and lines 204-214.

Proposed improvements to the manuscript 2: The PoseEncoder is relatively simplistic, using flattened 2D keypoints and a single Transformer encoder layer, which limits its ability to model complex spatial relationships between joints. This can be improved by adopting a graph-based or spatiotemporal pose model, such as ST-GCN, which captures joint dependencies more effectively.

Response of Proposed improvements to the manuscript 2: More sophisticated pose encoders, such as Graph Convolutional Networks (GCNs), could capture inter-joint dependencies more effectively. Our current design prioritizes simplicity and efficiency, demonstrating that even a lightweight encoder can yield strong results when combined with our novel fusion mechanism. We believe that exploring graph-based models is an excellent direction for future research. Accordingly, we have acknowledged this as a valuable avenue for improvement in the "Conclusion and Future Work" section and have added a citation to the foundational ST-GCN paper. – page 15, paragraph 1, and lines 497-500.

Proposed improvements to the manuscript 3: The evaluation is also limited to smaller or moderately sized datasets (KTH, UCF101, HMDB51), which do not fully test generalization. To improve this, the authors should include experiments on larger-scale benchmarks like Kinetics400 or Something-Something V2.

Response of Proposed improvements to the manuscript 3: We thank the reviewer for this important suggestion. We agree that evaluating TransMODAL on larger-scale benchmarks like Kinetics-400 and Something-Something V2 is a critical next step to fully assess its scalability and generalization capabilities. While conducting these extensive new experiments is beyond the scope of the current manuscript, we have made sure to explicitly acknowledge this limitation and highlight it as a primary direction for future work. This is stated in the "Conclusion and Future Work" section. – page 15, paragraph 1, and lines 495-497.

Proposed improvements to the manuscript 4: The model claims computational efficiency, there is no detailed comparison of inference speed, FLOPs, or model size versus state of the art flow-based or dual-stream models. Adding such comparisons would better position TransMODAL in terms of practical deployment.

Response of Proposed improvements to the manuscript 4: We thank the reviewer for this valuable suggestion. To better contextualize our model's efficiency, we have added a new Table 6 and a corresponding paragraph in the "Analysis and Discussion" section (Section 5). This table provides a direct comparison of TransMODAL's trainable parameters, FLOPs, and accuracy against prominent two-stream models that use computationally expensive optical flow (I3D and R(2+1)D), highlighting its competitive efficiency. – page 14, paragraph 2, and lines 458-468.

Proposed improvements to the manuscript 5: While qualitative results are strong, more in-depth attention visualizations and a broader ablation study (eg, alternative fusion mechanisms or deeper pose encoders) would provide greater insight into the architecture's design choices and performance.

Response of Proposed improvements to the manuscript 5: We thank the reviewer for these excellent suggestions for extending the analysis. We agree that a broader ablation study on different fusion mechanisms or deeper pose encoders, as well as more in-depth visualizations of the cross-attention weights, would provide valuable insights into the model's behavior. Given that these experiments would require significant implementation and retraining, we believe they constitute a substantial body of work that is best addressed in a future study. We have, however, incorporated these valuable suggestions into our "Conclusion and Future Work" section to explicitly highlight them as important and promising avenues for further research. – page 15, paragraph 1, and lines 497-500.

Comments 1: The author should refer to how other papers on deep neural networks describe the method. The general practice is to provide the architecture figure of the entire network in the third part. Please note that it is an architecture figure, not a flowchart. A flowchart is used to describe an algorithm. The architecture figure describes the input and output feature map size, fusion method, convolution used, and other information for each module, in order for readers to reproduce the author's work. Figure 1 uses a flowchart to describe the proposed network, which appears very unprofessional and lacks necessary information.

Response 1: We thank the reviewer for this valuable and detailed feedback. We agree. A professional architectural diagram is the standard for deep learning papers and provides significantly more clarity for reproducibility than the previous flowchart. We have, accordingly, completely replaced the former Figure 1 with a new, detailed architectural diagram. This new figure illustrates the internal components of our proposed modules, including the convolutional layers and residual connection in the PoseEncoder, the dual cross-attention mechanism in CoAttentionFusion, the learnable scoring layers in AdaptiveSelector, and the final MLP in the ActionClassifier. The transformation of tensor dimensions is clearly marked at each stage to provide all necessary information for readers to understand and reproduce our work. – page 5, paragraph 4, and lines 199-203.

Comments 5: The author claims to have proposed two modules, a CoAttention Fusion module and an efficient Adaptiveselector. However, detailed network structure diagrams were not provided for these two modules. Readers are unable to determine the size of the output and input feature maps. It is also impossible to know the propagation process of the forward propagation flow.

Response 5: We thank the reviewer for this crucial feedback. We agree with this valuable point completely. A detailed illustration of our two novel modules, CoAttentionFusion and AdaptiveSelector, is essential for clarity and reproducibility. The original Figure 1 was intended to provide a high-level overview of the entire system, but it did not adequately detail the internal forward propagation logic and feature map transformations within our core contributions. To address this, in addition to revise sections 3.3.1 and 3.3.2, we have added a new Figure 2 to the manuscript, which is dedicated to illustrating the detailed architecture of these two modules. This new figure visually depicts the parallel cross-attention paths, feed-forward networks, and residual connections within CoAttentionFusion, as well as the two-stage learnable scoring and token pruning process in AdaptiveSelector. All input and output tensor shapes are clearly labeled to show the forward propagation flow. Consequently, all subsequent figures have been renumbered (the former Figure 2 is now Figure 3, etc.), and all in-text references have been updated accordingly. – pages 7, paragraphs 2,3... , and lines 254-284 and pages 8, paragraphs 1, and lines 285-299.

Comments 6: Section 1.1 is included in the Introduction of the first part, but there is no section 1.2. Please prepare your manuscript carefully.

Response 6: We thank the reviewer for their careful attention to the manuscript's formatting. We agree that the single subheading "1.1. Problem Statement" was unnecessary and created an inconsistency in the structure. We have, accordingly, removed this subheading and integrated the text directly into the main body of the Introduction to improve the flow and correct the formatting. – page 1, paragraph 5, and lines 34-42.

Comments 7: Remove the black dots before lines 77 and 87, as this is not a PowerPoint presentation.

Response 7: We thank the reviewer for pointing out this formatting issue. We agree that the bullet points are not appropriate for a formal manuscript. We have removed the bulleted list and have rewritten the two contributions as a single, cohesive paragraph to improve the flow and professionalism of the text. – page 2, paragraph 4, and lines 73-94.

Comments 8: In the second part of the relevant literature, the author only listed the literature without summarizing it, and did not prove the problem that needs to be solved in this article through the summary of the literature.

Response 8: We thank the reviewer for this excellent feedback. We agree that the "Related Work" section should more clearly synthesize the existing literature to build a compelling argument for our research. The previous version described relevant works but did not sufficiently connect their limitations to the problems that TransMODAL is designed to solve. To address this, we have revised the end of each subsection within the "Related Work" section (Sections 2.1, 2.2, and 2.3) to include a concluding summary. These summaries now explicitly state the key takeaway from each line of research and highlight the specific gaps—such as the limited receptive fields of CNNs, the modality limitations of RGB-only Transformers, and the open questions in fusion strategies—that motivate our work. This creates a clearer logical progression that culminates in the problem statement our paper addresses. – page 3, paragraph 3, and lines 116-118 and pages 3, paragraphs 4, and lines 134-137 and pages 4, paragraphs 1,  and lines 145-147.