Entropy-Gated Prediction Agreement for Two-View Video Action Recognition

Abstract

Human action recognition (HAR) often struggles to capture important temporal cues distributed across an entire video when relying solely on a single sampled clip. To overcome this limitation, this study proposes a framework that constructs two temporal views from the same video and explicitly learns the prediction consistency between them. Specifically, the prediction-level agreement (AG) loss was introduced to align the class probability distributions of the two views. In addition, conditional gating was applied to adaptively control the contribution of AG loss according to the sample-wise prediction confidence, thereby reducing unstable alignment in temporally ambiguous or information-insufficient segments. The proposed framework was evaluated using both convolutional neural network (CNN)- and Transformer-based backbones on three representative action-recognition benchmark datasets, and it generally improved the performance over the single-view baseline across backbone–dataset combinations. Further empirical analyses, including training behavior, motion magnitude, temporal prediction stability, and qualitative case studies, were conducted to examine the effectiveness and behavior of the proposed two-view framework from multiple perspectives.

1. Introduction

Human action recognition (HAR) has been widely studied as a key research topic in various application domains, including intelligent surveillance, sports analysis, robotic vision, and autonomous systems. Because videos inherently contain spatiotemporal information along the temporal axis, HAR presents unique challenges compared to single-image recognition, as it requires modeling not only the spatial appearance but also the contextual temporal dynamics of actions [1,2].

Early studies employed frame-level features extracted using 2D convolutional neural networks (CNNs) [3]; however, these approaches were limited in effectively capturing temporal information. Subsequently, 3D CNN-based architectures were introduced to jointly model spatial and temporal features [4]. For example, SlowFast proposes a dual-pathway structure with different temporal resolutions to learn semantic information and dynamic motion patterns [5]. More recently, Transformer-based models have been adopted for HAR, and approaches such as TimeSformer have demonstrated the ability to model global spatiotemporal relationships by disentangling spatial and temporal attention mechanisms [6,7].

Despite these advances, several HAR models still adopt a single-view training strategy in which only one clip is sampled from a video during training. Although this approach is practical in terms of computational efficiency and implementation simplicity, it may fail to sufficiently capture the action cues distributed throughout the entire video. In particular, when key actions are concentrated within specific temporal regions, or when certain segments contain incomplete information, models trained on a single segment may yield uncertain predictions. This ultimately raises the question of how different temporal segments within the same video can be effectively utilized during training.

Existing studies that leverage temporal information in video-based action recognition can be categorized into three types. First, multi-view learning exploits the relationships among different views to improve the representation robustness. Second, multi-segment or multi-branch approaches combine multiple temporal segments or information sources to enhance the video representation and prediction performance. Third, consistency-based learning employs the prediction consistency across different inputs, repeated inferences, or multiple models as a learning signal. However, these studies have mainly focused on representation alignment, information fusion, and prediction stabilization. The problem of directly learning the relationship between prediction distributions obtained from different temporal views of the same video in a fully supervised setting has been relatively underexplored.

To address this limitation, this paper proposes a two-view framework comprising two temporal views extracted from the same video. Different temporal segments can provide complementary information for the same action instance; however, the clarity of action cues and reliability of predictions may vary across segments. Some segments may fail to adequately capture the core action, resulting in uncertain and potentially unstable predictions, depending on their temporal position within the same video. To alleviate this issue, this paper incorporated agreement (AG) regularization into the training objective such that the prediction distributions of the two temporal views maintained consistent relationships. However, applying the same alignment strength to all samples may cause excessive alignment, even for segments with insufficient information.

Therefore, this paper introduced conditional gating, which adaptively controls the strength of the alignment based on prediction uncertainty. This design allows AG regularization to be actively utilized for reliable segments, while suppressing excessive alignment for ambiguous segments. Therefore, this study proposes a unified learning framework that combines a temporal two-view structure, AG regularization, and conditional gating.

The main contributions of this study are summarized as follows:

• This study proposes a two-view framework for video action recognition that directly aligns the prediction probability distributions of different temporal views extracted from the same video. AG regularization is introduced to encourage prediction-level agreement, while conditional gating adaptively controls its contribution based on sample-wise prediction confidence.
• Through experiments on three representative action-recognition benchmark datasets using both CNN- and Transformer-based backbones, this paper demonstrates that the proposed method provides consistent performance improvements across various model architectures and datasets.
• The proposed two-view framework is analyzed through group-wise training metrics, motion magnitude-based performance analysis, prediction-consistency and reliability analysis, and qualitative case analysis. In addition, the single-view baseline, nAG without agreement regularization, and the proposed gAG are compared to distinguish the effects of two-view supervision and gated agreement regularization.

The remainder of this paper is organized as follows. Section 2 reviews the related work and discusses how our approach differs from the existing methods. The proposed method is described in Section 3. Section 4 presents and analyzes the experimental setup and results. Section 5 discusses the key implications and limitations of this study. Finally, Section 6 concludes the paper and suggests directions for future research.

2. Related Work

This section reviews previous studies related to the proposed method from three perspectives: multi-view learning, multi-branch or multi-segment action recognition, and consistency-based learning.

First, multi-view learning was developed to improve the representation robustness by utilizing different views generated from the same sample. Refs. [8,9] are contrastive learning methods that learn representations from augmentation-based views, whereas Ref. [10] learns the relationships between different temporal clips in videos through contrastive learning and aligns them in the representation space. In contrast, the present study directly addresses the relationship between the prediction probability distributions at the final classification stage rather than focusing on representation alignment.

Second, in the field of HAR, multi-branch and multi-segment architectures have been actively studied to exploit multiple temporal information sources. Refs. [3,5,11] are representative approaches that utilize different modalities, temporal resolutions, or segment-level information. In particular, SlowFast is based on a dual-pathway architecture that learns semantic and motion-related information separately, and stable performance has been reported on benchmark datasets such as UCF101 [12] and HMDB51 [13] and HAA500 [14]. In contrast, TimeSformer is a Transformer-based backbone that learns video representations using only space-time self-attention without convolution. However, these approaches mainly focus on feature fusion, dual-pathway modeling, or prediction aggregation and do not directly incorporate the relationship between prediction distributions from different temporal views into the training objective.

Table 1. Representative previously reported Top-1 accuracy results on UCF101, HMDB51, and HAA500, used as reference performance levels for each dataset.

Dataset	Method	Top-1 (%)	Ref.
UCF101	VideoMAE V2/ViT-g	99.05	[15]
UCF101	3D CNN + BERT temporal modeling	98.69	[16]
HMDB51	VideoMAE V2/ViT-g	86.08	[15]
HMDB51	3D CNN + BERT temporal modeling	85.10	[16]
HAA500	TSN Two-Stream	64.40	[14]
HAA500	I3D Three-Stream	49.87	[14]

summarizes representative previous results to contextualize dataset difficulty. Since these methods differ in pretraining data, input modality, backbone scale, test-time views, and evaluation protocols, they are not used for direct comparison. The main evaluation is instead conducted under the controlled Baseline–nAG–gAG setting.

Third, consistency-based learning has been widely studied in the context of semi-supervised learning and regularization. Refs. [17,18,19,20,21] are representative methods that use the prediction consistency across different conditions, repeated inferences, and multiple models as learning signals. In video action recognition, attempts have been made to exploit temporal consistency and prediction stability, as described in Ref. [22]. In addition, recent studies have used temporal-view consistency for test-time adaptation in video action recognition [23], temporal consistency as a learning signal for domain adaptation [24], and confidence-weighted divergence to align the prediction distributions between different views [25]. However, these studies focused on test-time adaptation, domain adaptation, and multicamera settings. Therefore, they differ from the present study in both problem setting and application, as our method directly aligns class-wise prediction probability distributions between two views extracted from the same video in a fully supervised setting and adaptively controls the alignment strength through sample-wise conditional gating.

Recent video action recognition studies have addressed view bias and view ambiguity in multi-view settings. Ref. [26] proposed a debiasing strategy to reduce excessive dependence on specific temporal thumbnails by strengthening semantic action representations. Ref. [27] addressed the view fuzz problem using a dual-recommendation disentanglement network that separates view-related and action-related features. These studies are meaningful in that they improve robustness against view-dependent representation bias. However, they mainly focus on multi-view feature disentanglement, representation correction, or information fusion across different views.

In contrast, this study focuses on temporal-view consistency within a single RGB video rather than camera-view variation. The proposed framework constructs two temporal views with different temporal coverage and regularizes their prediction-level agreement without modifying the backbone architecture. Entropy-based gating is further introduced to reduce the influence of uncertain view pairs, thereby mitigating excessive regularization when one view contains insufficient action evidence. In this respect, the proposed method provides a lightweight temporal consistency regularization strategy for supervised action recognition while preserving a single-view inference structure.

3. Methods

3.1. Overview of the Proposed Framework

The proposed framework constructs two temporal views from the same video and jointly learns supervised classification for the anchor view and AG between the two views. Here, AG refers to a learning signal that encourages the prediction probability distributions of the two views to maintain consistent class-level semantics. Figure 1 illustrates the overall training pipeline.

First, two temporal windows are extracted from the input video using overlapping temporal sampling. Each window is then converted into a fixed-length clip through uniform temporal resampling to form the input for the two-view framework. The generated views are fed into a shared backbone network. Both views are used for supervised classification with the same ground-truth label, and their prediction distributions are additionally used to compute the AG loss. Accordingly, the two views jointly contribute to the classification objective, while the AG term encourages cross-view prediction consistency.

Furthermore, AG loss is applied to prevent the output probability distributions of the two views from becoming excessively different, thereby reducing the prediction variation caused by changes in the temporal position. However, because partial observations may lead to uncertain predictions in certain temporal views, this study introduces conditional gating to adjust the contribution of AG loss according to the sample-wise prediction confidence. Consequently, the proposed framework is designed to jointly learn the classification performance and prediction stability from different temporal observations of the same video.

In summary, this paper proposes a framework that learns AG regularization between two views in a fully supervised setting. In addition, this paper introduces adaptive gating based on prediction uncertainty to mitigate the excessive regularization that may occur when the same alignment constraint is uniformly imposed on all view pairs. This allows the proposed method to explicitly learn the prediction relationships between the two views while flexibly adjusting the alignment strength for each sample.

3.2. View Generation

3.2.1. Overlapping Temporal Sampling

This study trained a two-view framework by generating two temporal windows with different time ranges from a single-input video. The input video $V$ is defined in Equation (1), where $x_t$ denotes the frame at temporal index $t$, and $T$ denotes the total number of frames in the video.

$$V=\{x_0,x_1,\dots ,x_{T-1}\}$$

(1)

The center index $m$ of the entire video and the center-based extension length $o$ are defined in Equations (2) and (3), respectively, where $\alpha \in \left[\mathrm{0,1}\right]$ is a hyperparameter that controls the degree of overlap. Here, $\alpha$ is not the actual overlap ratio itself, but a hyperparameter that controls how much each window is extended around the center. Therefore, the actual overlap ratio may vary depending on the video and window lengths.

$$m=⌊{\displaystyle \frac{T}{2}}⌋$$

(2)

$$o=\mathrm{r}\mathrm{o}\mathrm{u}\mathrm{n}\mathrm{d}\left(\alpha m\right)$$

(3)

The two temporal windows, $W^{\left(1\right)}$ and $W^{\left(2\right)}$, are defined in Equations (4) and (5), respectively. Accordingly, the first view always included the earlier part of the video, whereas the second view always included the later part, with both views partially sharing a temporal context around the center.

$$W^{\left(1\right)}=[0, \mathrm{m}\mathrm{i}\mathrm{n}(T-1, m+o\left)\right]$$

(4)

$$W^{\left(2\right)}=\left[\mathrm{m}\mathrm{a}\mathrm{x}\right(0,m-o), T-1]$$

(5)

The actual degree of overlap between the two views is measured using the overlap ratio $r$, as defined in Equation (6).

$$r={\displaystyle \frac{\mid W^{\left(1\right)}\cap W^{\left(2\right)}\mid }{\mathrm{m}\mathrm{i}\mathrm{n}(\mid W^{\left(1\right)}\mid , \mid W^{\left(2\right)}\mid )}}$$

(6)

Although it is possible to fix the overlap ratio directly, this strategy requires a joint design of the positions and lengths of the two windows. This makes view construction less intuitive and complicates determining whether performance differences are caused by the overlap itself or by differences in the absolute positions or lengths of the windows. In contrast, the proposed strategy maintains simple and consistent view construction: the first view reflects the earlier part of the video, the second view reflects the later part, and $\alpha$ controls only the shared temporal context around the center.

3.2.2. Uniform Temporal Resampling

Although the two temporal windows $W^{\left(1\right)}$ and $W^{\left(2\right)}$ cover different temporal ranges, the backbone network requires fixed-length input clips. Therefore, this paper applies uniform temporal resampling to convert each window into a clip with the same number of frames. For each window $W^{\left(k\right)}=[s_k,e_k]$, $N$ frames are uniformly sampled over the entire interval, where $N$ denotes the number of frames in the input clip generated from each temporal window. Figure 2 shows an example of resampling a video clip.

In this process, the integer index of the $i$-th frame selected from the $k$-th window is denoted by ${\widehat{t}}_i^{\left(k\right)}$. This index is obtained by converting the uniformly spaced sampling position within $[s_k,e_k]$ to the nearest integer frame index, as shown in Equation (7). The corresponding frames are then arranged in temporal order to form a fixed-length clip $v^{\left(k\right)}$ for each view, as defined in Equation (8).

$${\widehat{t}}_i^{\left(k\right)}=\mathrm{r}\mathrm{o}\mathrm{u}\mathrm{n}\mathrm{d}\left(s_k+{\displaystyle \frac{i}{N-1}}(e_k-s_k)\right),i=0,\dots ,N-1$$

(7)

$$v^{\left(k\right)}=\{x_{{\widehat{t}}_0^{\left(k\right)}},x_{{\widehat{t}}_1^{\left(k\right)}},\dots ,x_{{\widehat{t}}_{N-1}^{\left(k\right)}}\}$$

(8)

This process evenly reflects the temporal range of each window, while converting two views extracted from different temporal segments into fixed-length clips with the same input format for the backbone network. The generated two-view clips are then used in the training objective, which combines classification loss and AG regularization.

3.3. Training Objective

3.3.1. Two-View Classification with Shared Backbone

The two views generated from the same video produce predictions through a shared backbone network, forming the basis for supervised classification and AG regularization. Specifically, the two views $v^{\left(1\right)}$ and $v^{\left(2\right)}$ are fed into the same backbone $f_{\theta }$, which outputs prediction logits $z^{\left(1\right)}$ and $z^{\left(2\right)}$, respectively, as defined in Equation (9). Because the two views share the same backbone parameters, the differences between the outputs are caused by differences in the temporal context rather than differences in the model structure.

$$z^{\left(k\right)}=f_{\theta }\left(v^{\left(k\right)}\right)$$

(9)

In this study, $v^{\left(1\right)}$ and $v^{\left(2\right)}$ were used as two temporal views, and the cross-entropy (CE) loss with the ground-truth label $y$ was applied to the prediction of each view, as shown in Equations (10) and (11).

$$L_{\mathrm{c}\mathrm{e}1}=CE(z^{\left(1\right)},y)$$

(10)

$$L_{\mathrm{c}\mathrm{e}2}=CE(z^{\left(2\right)},y)$$

(11)

Both temporal views contribute to supervised classification. During training, they are processed sequentially: the gradient from $L_{ce2}$ is first accumulated using the second view, after which $L_{ce1}$ and the AG loss are computed using the first view and the stored prediction of the second view. Thus, the AG regularization is applied during the optimization of the first-view prediction. The next subsection describes how consistency between the prediction distributions of the two views is encouraged.

3.3.2. Prediction-Level Agreement

Although these two views contain different temporal contexts, they must maintain consistent predictions regarding class-level semantics. However, conventional CE-based learning focuses on improving the classification accuracy of individual views and does not directly control the prediction consistency between them. Consequently, the model may overly depend on local cues from a specific temporal position or produce unstable predictions when the temporal position changes.

To alleviate this limitation, this paper introduced AG regularization between the two views. Each view produces prediction logits through the shared backbone, which were then converted into probability distributions for AG loss computation. Specifically, the prediction distribution $p^{\left(k\right)}$ of each view is defined using softmax with temperature ${\tau }_{agr}$, as shown in Equation (12).

$$p^{\left(k\right)}=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}({\displaystyle \frac{z^{\left(k\right)}}{{\tau }_{\mathrm{a}\mathrm{g}\mathrm{r}}}})$$

(12)

Both views are supervised using the same ground-truth label and therefore contribute to the classification objective. For AG regularization, the prediction of the second view is stored and used as a detached reference when the first view is processed. Consequently, the AG gradient is propagated through the first-view prediction only, while both prediction distributions are used to measure cross-view inconsistency.

To this end, the discrepancy between the two distributions was computed using symmetric Kullback–Leibler divergence (KL-Div) [28,29], as defined in Equation (13). where $L_{agr,i}$ is the AG loss of the $i$-th sample. This term measures the bidirectional discrepancy between the two prediction distributions, while the detached second-view prediction serves as a fixed reference during optimization. It thereby regularizes the first-view prediction to reduce prediction variation caused by temporal differences within the same video.

$$L_{\mathrm{a}\mathrm{g}\mathrm{r},i}={\displaystyle \frac{1}{2}}\left[D_{\mathrm{K}\mathrm{L}}(p_i^{\left(1\right)}\parallel p_i^{\left(2\right)})+D_{\mathrm{K}\mathrm{L}}(p_i^{\left(2\right)}\parallel p_i^{\left(1\right)})\right]$$

(13)

By using temperature-scaled probability distributions, the proposed method reduces excessive alignment toward only a few dominant classes caused by overly sharp predictions, and instead captures the overall relationship between the two distributions more smoothly. However, if this regularization is applied with the same strength to all the samples, noisy alignment signals from information-insufficient or uncertain views may also be enforced. To address this issue, the next subsection introduces conditional gating to adaptively control the contribution of AG loss according to the sample-wise prediction reliability.

3.3.3. Entropy-Based Gating

Instead of applying AG regularization equally to all samples, this paper introduced conditional gating to adjust its contribution according to sample-wise prediction reliability. For each sample i, a gate weight ${\lambda }_i\in \left[\mathrm{0,1}\right]$ is defined and multiplied by the sample-wise AG loss to form the gAG loss. A larger ${\lambda }_i$ increases the contribution of AG loss for that sample, whereas a smaller value reduces its influence. In addition, for samples whose constructed temporal windows cannot share a valid overlapping region, the corresponding gate weight is set to zero, thereby preventing unreliable agreement constraints from contributing to the training objective.

Thus, conditional gating mitigates the performance degradation caused by excessive alignment by adjusting the strength of AG regularization according to the prediction reliability. The gAG loss for batch $B$ is defined by Equation (14). Instead of simply averaging the AG losses of all samples in the batch, different weights were assigned based on their reliability to construct the batch-level agreement regularization.

A sample-wise gating mechanism suppresses unreliable agreement signals. Samples with gate weights below the threshold ($\tau$) are excluded from the gAG loss, and their contribution is set to zero. That is, AG regularization is effectively applied only to samples satisfying ${\lambda }_i\ge \tau$. Consequently, conditional gating flexibly adjusts the contribution of AG regularization according to the sample-wise prediction reliability and alleviates the performance degradation caused by excessive alignment.

$${\tilde{L}}_{\mathrm{a}\mathrm{g}\mathrm{r}}={\displaystyle \frac{1}{B}}{\displaystyle \sum _{i=1}^B}{\lambda }_i{L}_{\mathrm{a}\mathrm{g}\mathrm{r},i}$$

(14)

3.3.4. Final Objective

The final objective function is defined as the sum of the classification loss for the anchor view and the gAG loss, as shown in Equation (15). Here, $L_{ce}$ is the CE loss for the anchor view, and $\beta$ is a hyperparameter that controls the global contribution of the gAG loss. That is, while $\beta$ determines the overall strength of AG regularization, ${\lambda }_i$ can be interpreted as a local coefficient that determines how much the AG loss should be trusted and reflected for each sample.

$$L={\displaystyle \frac{1}{2}}(L_{\mathrm{c}\mathrm{e}1}+L_{\mathrm{c}\mathrm{e}2})+\beta {\tilde{L}}_{\mathrm{a}\mathrm{g}\mathrm{r}}$$

(15)

Consequently, the proposed objective was designed to maintain the supervised classification criterion while more stably learning two-view prediction consistency through reliability-based selective AG regularization.

4. Experiments

In this section, this paper evaluates the effectiveness of the proposed two-view framework using two backbone networks and three benchmark datasets: TimeSformer-UCF101 (TSF-UCF), TimeSformer-HMDB51 (TSF-HMDB), TimeSformer-HAA500 (TSF-HAA), SlowFast-UCF101 (SLF-UCF), SlowFast-HMDB51 (SLF-HMDB), SlowFast-HAA500 (SLF-HAA). This paper first describes the datasets, backbones, and experimental settings, and then presents the performance comparison, convergence behavior, and further analyses. All experiments were conducted using the official split and evaluation protocol of each dataset, and Top-1 accuracy (Acc.) was used as the primary performance metric.

4.1. Materials and Experimental Settings

4.1.1. Datasets

Experiments were conducted on three representative video-based human action recognition (HAR) benchmarks: UCF101, HMDB51, and HAA500. UCF101 contains 101 action classes with relatively clear scene context and stable camera motion, making it a comparatively less challenging benchmark. HMDB51 consists of 51 action classes and provides a more difficult setting due to low video quality, camera motion, occlusion, and large intra-class variations. HAA500 further increases the difficulty by introducing 500 fine-grained atomic action classes, where subtle pose and motion differences are critical for classification. The detailed statistics of each dataset are summarized in

Table 2. Dataset statistics used in the experiments.

Dataset	Class	Training	Testing	Total
UCF101	101	9537	3783	13,320
HMDB51	51	3570	1530	5100
HAA500	500	8000	2000	10,000

.

4.1.2. Backbones

In this study, TimeSformer and SlowFast pre-trained on Kinetics-400 [30] were used as backbone networks [31,32,33,34]. TimeSformer is a Transformer-based backbone that directly learns global dependencies along spatial and temporal dimensions using self-attention. SlowFast is a CNN-based backbone that jointly models long-term semantic and short-term dynamic information using a two-pathway architecture with different temporal resolutions. By evaluating both Transformer- and CNN-based models with different architectural characteristics, this paper aims to examine whether the proposed two-view framework can be applied generally rather than being limited to a specific backbone family. The detailed backbone configurations are listed in

Table 3. Backbone configurations used in the experiments.

Backbone	Pretraining	Model Variant	Input Clip
TimeSformer	Kinetics-400	divST_8×32_224_K400	32 frames
SlowFast	Kinetics-400	R101_16×8	64 frames (fast pathway)

.

4.1.3. Common Experimental Settings

For a fair comparison, the baseline and two-view framework models were trained under the same experimental settings. All experiments were conducted for 30 epochs using a batch size of 2, stochastic gradient descent (SGD) optimizer, initial learning rate of ${10}^{-3}$, and dropout rate of 0.5. Step decay was used for learning rate scheduling, where the learning rate was multiplied by 0.1 every five epochs. In addition, to reduce the memory usage, the two temporal views were processed sequentially, and gradient accumulation was used before updating the shared backbone parameters. The implementation details are presented in

Table 4. Implementation environment.

Item	Setting
OS	Ubuntu 20.04 LTS
Framework	Anaconda 22.9.0, CUDA 11.1, PyTorch 1.8.0
Python	Python 3.8 (TimeSformer)/Python 3.7 (SlowFast)
GPU	Nvidia RTX 3090 Ti (UCF101, HAA500)/Nvidia RTX TITAN (HMDB51)

.

4.2. Baseline Performance

To verify the effectiveness of the two-view framework, single-input baseline models were first trained and used as references for all the comparisons. For both backbones, the baseline models were trained using the same dataset and training conditions. Figure 3 shows the training loss and test-set Top-1 accuracy curves across checkpoints.

Table 5. Performance comparison of the single-view baseline and the two-view nAG and gAG models. For the baseline, Top-1 denotes the single-view Top-1 accuracy. For nAG and gAG, Top-1 denotes the two-view Top-1 accuracy, Gain denotes the improvement over the baseline, and Time denotes the average training time per epoch (hour/epoch) measured under the same experimental environment.

Backbone–Dataset	Baseline	nAG			gAG
	Top-1	Top-1	Gain	Time	Top-1	Gain	Time
TSF-UCF	94.66	96.46	$+$1.80	0.88	96.59	$+$1.93	0.86
SLF-UCF	95.14	95.82	$+$0.68	1.44	95.88	$+$0.74	1.21
TSF-HMDB	70.33	72.81	$+$2.48	0.42	73.07	$+$2.74	0.42
SLF-HMDB	77.03	78.02	$+$0.99	0.81	79.27	$+$2.24	0.80
TSF-HAA	77.10	82.50	$+$5.40	0.95	82.10	$+$5.00	0.93
SLF-HAA	63.20	69.00	$+$5.80	2.80	68.95	$+$5.75	2.72

reports the best test Top-1 accuracy of the single-view baseline for each backbone–dataset combination.

Based on official split 1, each epoch checkpoint was evaluated on the test set, and the model with the highest Top-1 accuracy was selected as the final baseline. Overall, the training tended to converge stably across all backbone–dataset combinations. These baseline results are compared with the training results of the two-view framework described in the following subsection.

4.3. Two-View Performance

The performance of the two-view framework was evaluated against the single-view baseline. Along with the proposed gAG model, nAG, which denotes two-view training without the gAG loss, was included to distinguish the effect of two-view supervision from that of agreement regularization.

The two-view framework uses two temporal windows generated from a single video and converted into fixed-length clips. The parameter $\alpha$ controls temporal window generation rather than directly representing the actual overlap ratio. For a fair comparison, the same hyperparameter settings were used across all experiments: $\alpha$ was fixed at 0.5, the softmax temperature for the AG loss was set to 2.0, and the global AG weight $\beta$ was set to 0.05. For all gAG experiments, the gating threshold ($\tau$) was fixed at 0.6.

In Figure 4, for both backbones, the training loss decreases rapidly in the early stage and then remains low, whereas the Top-1 accuracy generally increases and converges. This suggests that the proposed two-view framework forms a stable optimization process across different backbones. The degree of performance improvement and detailed behavior of the learning curves differ depending on the backbone and dataset. TimeSformer tends to show a relatively stable performance improvement as training progresses, whereas SlowFast shows rapid convergence after early fluctuations. Nevertheless, the learning curves generally exhibit stable convergence across all combinations, indicating that the proposed framework can provide consistent training behavior across different backbones.

Table 5 compares the performance of the single-view baseline with the two-view nAG and gAG models. Overall, both nAG and gAG outperform the baseline across all backbone–dataset combinations, indicating that the two-view learning framework can exploit richer temporal cues than single-view training.

Compared with the baseline, gAG improved Top-1 accuracy by $+$1.93 and $+$0.74 percentage points on UCF101, $+$2.74 and $+$2.24 percentage points on HMDB51, and $+$5.00 and +5.75 percentage points on HAA500 for TimeSformer and SlowFast, respectively. The largest gains were observed on HAA500, suggesting that two-view supervision can be particularly useful for complementing temporal evidence in fine-grained atomic action recognition with short video durations. The reported Time value represents the average training time per epoch measured under the same experimental environment and serves as a practical reference for comparing the training costs of nAG and gAG.

Overall, Table 5 shows that two-view learning consistently improves performance over the single-view baseline. The detailed differences between nAG and gAG, including checkpoint-level stability and prediction consistency, are further analyzed in the next subsection.

4.3.1. Prediction-Consistency and Reliability Analysis

In this subsection, we analyze the effect of gated agreement on prediction consistency and reliability under the same two-view setting. nAG and gAG are compared using Top-1 accuracy, symmetric KL divergence, and prediction confidence. KL-Div. measures the discrepancy between the class probability distributions of the two temporal views, where a lower value indicates higher cross-view prediction consistency. Confidence denotes the predicted probability of the selected class, and is used to examine whether the performance change is associated with more reliable predictions.

First, ∆Top-1 shows that the performance effect of gAG is dependent on the backbone–dataset combination. The largest improvement is observed in SLF-HMDB (+1.25), whereas TSF-HAA shows the largest decrease (−0.40). This suggests that gated agreement can improve recognition accuracy when the two temporal views provide reliable complementary evidence, but may be less effective when the shared temporal evidence is insufficient or ambiguous.

Second, ∆Std. indicates that the effect of gAG on checkpoint-level stability is not uniform. Although gAG reduces the standard deviation in several settings, the average ∆Std. is close to zero, suggesting that gated agreement affects stability differently depending on the backbone and dataset.

Third, $∆$KL-Div. does not decrease consistently across all settings. SLF-HMDB shows both improved Top-1 accuracy and reduced KL-Div., indicating a favorable case where prediction consistency and recognition performance improve together. In contrast, TSF-HAA shows reduced KL-Div. but lower Top-1 accuracy, suggesting that lower prediction discrepancy alone may not guarantee better performance when the shared temporal evidence is not sufficiently discriminative.

Finally, $∆$Conf. remains close to zero in most settings. SLF-HMDB achieves the largest Top-1 improvement with almost no confidence change, whereas TSF-HAA shows decreases in both $∆$Conf. and ∆Top-1. This suggests that the effect of gAG is not simply due to increased prediction confidence, but is more related to the reliability of temporal-view evidence.

The differences between gAG and nAG in terms of Gain, KL-Div., Confidence, and Entropy are summarized in

Table 6. Auxiliary analysis of gAG relative to nAG. $∆$Top-1, $∆$Std., $∆$KL-Div., and ∆Conf. denote the gAG−nAG differences in Top-1 accuracy, checkpoint-level standard deviation, symmetric KL divergence, and prediction confidence, respectively. Negative $∆$KL-Div. values indicate reduced two-view prediction discrepancy.

Backbone–Dataset	$∆$Top-1	$∆$Std.	$∆$KL-Div.	$∆$Conf.
TSF-UCF	$+$0.13	$-$0.02	$+$0.010	$+$0.0069
SLF-UCF	$+$0.06	$+$0.17	$+$0.003	$+$0.0005
TSF-HMDB	$+$0.26	$-$0.14	$+$0.004	$-$0.0079
SLF-HMDB	$+$1.25	$+$0.38	$-$0.002	$+$0.0002
TSF-HAA	$-$0.40	$-$0.40	$-$0.003	$-$0.0054
SLF-HAA	$-$0.05	$-$0.01	$\approx$0.000	$-$0.0002

.

Overall, the comparison between nAG and gAG shows that gated agreement functions as a selective regularizer rather than a uniform constraint that forces all two-view predictions to be identical. Although gAG improved Top-1 accuracy in most backbone–dataset combinations, the changes in KL divergence and confidence were condition-dependent, indicating that its effect varies with the backbone architecture and dataset characteristics. The contrasting results of SLF-HMDB and TSF-HAA further support this interpretation: gAG improved both accuracy and two-view consistency in SLF-HMDB, whereas it reduced checkpoint-level variation and KL divergence in TSF-HAA without improving Top-1 accuracy. These findings suggest that gated agreement is most effective when reliable and discriminative temporal evidence is available, while its benefit can be limited for ambiguous or phase-dependent views.

4.3.2. Motion Magnitude and Two-View Performance Analysis

This subsection analyzes how the performance difference between the baseline and two-view framework models changes according to the magnitude of video motion. The optical flow [35] magnitude between consecutive frames was computed for each video, and the average value was defined as the motion score. Optical flow was only used as an analysis metric to quantify the degree of dynamic change in the videos. All test samples were divided into low-, medium-, and high-motion groups based on the motion score, and the Top-1 accuracy of the baseline and best-performing two-view framework models for each backbone–dataset combination was compared in each group.

The line plots in Figure 5 show the Top-1 accuracy of the baseline and two-view framework models in each motion group, whereas the bar plots represent the absolute performance gain of the two-view framework model over the baseline (gain, %p). Thus, the line plots indicate the overall recognition performance level in each motion group, whereas the bar plots show the additional improvement provided by the two-view framework over the baseline at that performance level.

This experiment was conducted to examine whether the effect of two-view learning varies with the motion magnitude of videos. Overall, gAG achieved higher Top-1 accuracy than the baseline in most motion groups, indicating that two-view learning contributes to performance improvement across different motion levels. However, the magnitude of the gain showed dataset-dependent patterns. In TimeSformer–UCF101, both models maintained high accuracy across all motion groups, while the gain was largest in the low-motion group and gradually decreased toward the high-motion group. In contrast, TimeSformer–HMDB51 and TimeSformer–HAA500 showed more pronounced gains in the low- and high-motion groups than in the medium-motion group. These results suggest that two-view learning is generally effective across motion levels, but its benefit depends on the dataset-specific motion characteristics and the distribution of temporal cues.

4.3.3. Sample-Level Analysis

This section analyzes whether the prediction consistency between the two temporal views is related to sample-level classification correctness. For each test sample, the symmetric KL divergence between the prediction distributions of the two views was computed. The samples were then divided into “Correct” and “Wrong” groups according to the Top-1 prediction result, and the distributions were compared across datasets, backbones, and training strategies.

Figure 6 shows the KL divergence distributions for UCF101, HMDB51, and HAA500. In all datasets, correctly classified samples are concentrated in relatively low-divergence regions, whereas wrongly classified samples show higher divergence and larger variation. This trend indicates that when the two temporal views produce similar prediction distributions, the final prediction is more likely to be correct. In contrast, large view-wise prediction discrepancies are frequently associated with ambiguous or insufficient temporal evidence, leading to unstable predictions.

The tendency is particularly clear in the “Wrong” groups, where large outliers appear more frequently than in the “Correct” groups. This suggests that two-view prediction consistency can be interpreted as an indicator of temporal prediction stability. Overall, these results support the motivation of the proposed framework, suggesting that improving agreement between reliable temporal views can enhance the stability and accuracy of action recognition.

4.3.4. Group-Wise Analysis of Training Metrics Across Performance Levels

In this subsection, the epochs of each backbone–dataset combination are sorted by the test Top-1 accuracy and divided into bottom-5, middle-5, and top-5 groups. The average cross-entropy loss (CE), mean gate weight $\left(\lambda \right)$, and gAG loss were then compared across the groups.

Figure 7 shows the changes in these training metrics according to the performance level. Overall, CE tended to decrease as the performance improved across the datasets, indicating that higher-performing epochs were generally associated with more stable classification optimization. This trend was particularly pronounced in HAA500, where the SlowFast CE sharply decreased from the bottom-5 to the middle-5 and top-5 groups.

CE reflects the supervised discriminative learning of each temporal view, whereas the gate weight $\lambda$ determines how strongly the agreement term is activated for each sample. The gAG loss is therefore not a pure error term like CE, but a gated consistency signal whose magnitude depends on both prediction discrepancy and sample-wise reliability. In contrast to CE, the mean gate weight and gAG loss did not show a strictly monotonic pattern in all cases. This suggests that gAG loss should not be interpreted as better simply when lower. Rather, the gated agreement term acts as an auxiliary consistency constraint whose effect depends on the reliability and temporal compatibility of the two views. Therefore, Figure 7 indicates that higher-performing groups were mainly characterized by lower classification loss, while agreement regularization was selectively reflected during training.

4.3.5. Qualitative Success and Failure Case Analysis

A qualitative analysis was conducted on the TSF-UCF results to examine when two-view learning improves or degrades recognition performance. Samples with different predictions between the baseline and gAG were selected and categorized into success and failure cases. For each case, the input frames from the two temporal views, predicted classes, confidence (Conf.), and entropy (Ent.) are presented.

Figure 8a,b show success cases in which the proposed method corrects the misclassifications of the baseline. In the BlowDryHair and TableTennisShot cases, the baseline incorrectly predicts Haircut and JumpRope, respectively, whereas gAG correctly predicts the ground-truth classes by exploiting action cues commonly shared by the two views. Both cases show high Conf. and low Ent., suggesting that agreement learning can serve as a reliable regularization signal when the two temporal views share consistent class-discriminative evidence.

In contrast, Figure 8c,d show failure cases. In the Nunchucks and CricketBowling cases, the action cues are not stably shared between the two views due to dark backgrounds, ambiguous object motion, or the emphasis on only a specific motion phase. As a result, gAG misclassifies them as PizzaTossing and Shotput, respectively, with low Conf. and high Ent. This suggests that when shared cues between views are insufficient or temporal phases are inconsistent, agreement learning may reinforce uncertain predictions.

Therefore, gAG can improve recognition performance by enhancing prediction consistency for reliable two-view pairs, but its effect may be limited for ambiguous view pairs. These results support the need for confidence- and entropy-aware gating to mitigate unreliable agreement.

4.3.6. Summary of Experimental Findings

The overall experimental results show that the proposed two-view framework generally improves Top-1 accuracy over the single-view baseline. In particular, the largest gains are observed on HAA500, suggesting that two-view supervision can help complement temporal evidence in fine-grained atomic action recognition.

However, the comparison between nAG and gAG shows that the effect of agreement regularization is condition-dependent. gAG improves the average Top-1 accuracy over nAG, but the improvement is not uniform across all backbone–dataset combinations. The KL-divergence analysis further shows that reduced prediction discrepancy does not always directly lead to higher accuracy, suggesting that agreement is most useful when the two temporal views share reliable and class-discriminative evidence.

The qualitative success and failure cases support this interpretation. Reliable view pairs with consistent action cues tend to benefit from agreement, whereas ambiguous view pairs or temporally mismatched phases can limit its effect. These findings suggest that the proposed gated agreement should be viewed as a selective regularization mechanism rather than a uniform constraint applied equally to all two-view pairs.

5. Discussion

The results indicate that two-view learning can improve action recognition performance by exploiting temporal evidence from different segments of the same video. Nevertheless, the effect of agreement regularization was not uniform across all experimental conditions. This suggests that prediction-level agreement should be applied with consideration of the reliability of each view pair, rather than being treated as a universally beneficial constraint.

The current framework also has several limitations. First, the experiments were conducted in a supervised trimmed-video setting. Therefore, further validation is required to assess its applicability to untrimmed videos, temporal localization, or more complex real-world scenarios. Second, the framework uses only two temporal views because simultaneous multi-view processing substantially increases GPU memory usage. Although sequential processing with gradient accumulation enabled CE supervision for both views, extending the framework to more than two views remains an important future direction. Third, entropy was used as a simple proxy for prediction uncertainty. More advanced confidence estimation methods or learnable gating functions may further improve the robustness of agreement regularization.

These limitations provide meaningful directions for future research. Extending the proposed framework to multi-view temporal coverage, untrimmed video understanding, and adaptive gating mechanisms may further improve the generality and reliability of two-view prediction agreement learning.

6. Conclusions

This study proposes a two-view framework that directly learns prediction-level temporal alignment to address prediction discrepancies between different temporal views of the same video in video-based action recognition. The proposed method combines supervised classification of the anchor view with symmetric agreement regularization between the two views, and mitigates the performance degradation caused by excessive alignment by adjusting the alignment strength for each sample through entropy-based conditional gating.

Experiments on two representative benchmark datasets and structurally different backbones showed that the proposed method consistently improved performance over the baseline across various settings. These results indicate that the improvement is related to prediction-level consistency learning between temporal views, rather than simply multi-view input expansion.

Consequently, this study demonstrates that combining prediction-level alignment with adaptive gating is an effective training strategy for handling temporal variations in video action recognition. It also suggests future directions for multi-view extensions, untrimmed video settings, and more sophisticated confidence modeling.