Action-Region Mixing: A Video Augmentation Approach for Reducing Scene Bias in Action Recognition

Abstract

Video action recognition models can rely heavily on recurring scene context rather than action-relevant cues, limiting their generalization to background changes. foreground–background mixing has therefore emerged as a useful augmentation strategy for reducing such scene dependence. However, existing semantic mixing approaches typically rely on off-the-shelf object detectors and associated preprocessing and postprocessing pipelines to localize action-related regions, while bounding-box-based representations may introduce unrelated background content into the mixed samples. To address these limitations, we propose ActionRegionMix, a prompt-guided, action-aware video augmentation framework that leverages a mask-generating Multimodal Large Language Model (MLLM) to extract action-related regions at the pixel level in an open-vocabulary manner. The extracted action regions are transferred across videos and composited with backgrounds from other training samples, introducing contextual variation while preserving action-relevant content. Experiments on UCF101 and HMDB51 show that ActionRegionMix consistently improves recognition performance over the no-augmentation baseline and the compared bounding-box-based semantic augmentation methods. These results demonstrate the effectiveness of pixel-level action-region mixing as a video augmentation strategy, offering a promising approach to enhance background diversity and mitigate reliance on scene-specific cues.

1. Introduction

The success of modern action recognition models depends heavily on large-scale labeled video datasets. 3D convolutional neural networks (CNNs) [1,2] effectively capture joint spatiotemporal features, but their large number of parameters makes them prone to overfitting when training data is limited [3]. Vision Transformers (ViTs) [4] capture long-range dependencies through attention mechanisms, yet these models also typically require substantial training data to generalize effectively [5,6]. Meeting this data requirement is particularly challenging in the video domain, where the additional temporal dimension makes data collection and annotation more demanding than for static images [7,8,9]. Consequently, obtaining sufficiently diverse labeled videos remains a major challenge. Data augmentation offers a practical way to increase the diversity of training samples without collecting and annotating additional videos, helping reduce overfitting and improve generalization.

Conventional video augmentation approaches commonly use image-based transformations, such as random cropping, horizontal flipping, and color jittering, to increase training data diversity [10,11]. These transformations introduce spatial and appearance variations but offer limited semantic diversity, as they do not explicitly modify the semantic content or composition of a video. Mixing-based methods, such as Mixup [12] and CutMix [13], have also been extended to video data [14]. These methods combine information from different videos, typically without considering the semantic boundaries of action-relevant regions. Because they do not explicitly distinguish these regions from the background, models trained on the augmented samples may still exhibit scene bias, relying on background cues rather than the action itself for recognition [15]. To reduce this bias, ActorCutMix [16] and InteractionCutMix [17] guide region mixing using actor regions and actor regions together with action-related objects, respectively. However, reliance on detectors with predefined object categories and additional pre-processing and post-processing steps can limit the flexibility and scalability of these approaches across diverse actions.

To address these limitations, we propose ActionRegionMix, an action-aware data augmentation framework that replaces detector-based region selection with pixel-level segmentation using a mask-generating multimodal large language model (MLLM) [18]. By incorporating the target action class name into a text prompt, ActionRegionMix guides the MLLM to segment action-relevant regions in an open-vocabulary manner. During training, the generated masks are used to extract action foregrounds and combine them with backgrounds randomly selected from other videos within the current mini-batch through direct spatiotemporal mixing. The action recognition network is trained jointly on the original videos and the synthesized background-swapped sequences. This strategy encourages reliance on action cues rather than background context, aiming to reduce scene bias and improve generalization.

The key contributions of this study can be briefly summarized as follows:

• We introduce an action-conditioned region selection strategy for video augmentation using prompt-guided MLLM segmentation. Unlike detector-based methods that identify actors or objects from predefined categories, our approach uses the target action class name to extract pixel-level action-relevant foregrounds in an open-vocabulary manner.
• We develop ActionRegionMix, a video augmentation framework that uses these action-conditioned masks to combine action-relevant foregrounds with backgrounds from other videos. Training on the original and augmented videos encourages the model to rely on action cues rather than scene-specific context.

2. Related Work

2.1. Video Augmentation for Action Recognition

Data augmentation is a fundamental approach for reducing overfitting and improving the generalization capability of deep learning models. In action recognition, conventional methods typically involve spatial transformations such as random cropping and horizontal flipping, appearance-based transformations such as color jittering, and temporal augmentations such as random frame sampling [10,11,19]. While these techniques have become standard practice, they primarily modify the appearance or sampling of existing videos rather than introduce new combinations of visual content. To generate more diverse training samples, VideoMix [14] extends the image-based CutMix strategy [13] to videos by mixing spatiotemporal cuboids, with labels weighted according to their volume proportions.

While VideoMix selects regions without explicitly considering their semantic content, semantic mixing methods aim to preserve action-relevant regions during augmentation. ActorCutMix [16] uses a pretrained human detector to identify actor regions and combines the foreground of one video with the background of another. This encourages the model to recognize actions across different backgrounds and reduces its reliance on scene cues. However, actor-focused augmentation may overlook objects involved in the action. InteractionCutMix [17] addresses this limitation by extracting rectangular regions containing both the actor and the interacting object, preserving human–object interactions during mixing. Learn2Augment [20] instead uses a pretrained Mask R-CNN [21] to segment people and objects in both videos, removes and inpaints these regions in the background video, and composites the foreground regions onto the resulting background. Its main focus is learning to select useful video pairs for augmentation, reducing the search space of candidate compositions. Despite their differences, these methods depend on the categories and localization capabilities of pretrained detection or segmentation models.

Although our focus is on reducing scene bias in action recognition, video augmentation also supports other objectives within action recognition and other video understanding tasks [11]. Surreact [22] generates synthetic human action videos to improve recognition from unseen viewpoints. Beyond action recognition, Vista [23] spatially and temporally combines existing videos to construct instruction-following data for long-duration and high-resolution video understanding.

Among the augmentation strategies discussed above, the effectiveness of both ActorCutMix and InteractionCutMix is inherently bounded by two main factors: the limitations of underlying object detectors and the rigidity of bounding boxes. First, as explicitly noted in recent studies [17], off-the-shelf detectors are typically trained on static images and exhibit suboptimal performance when applied to dynamic or low-resolution video frames, often necessitating cumbersome post-processing steps. Furthermore, their closed-set vocabularies restrict the augmentation process to a limited set of predefined categories. Second, relying on rectangular bounding boxes inevitably introduces spatial artifacts. As illustrated in Figure 1, ActorCutMix isolates only the actor, mostly missing the interacting object. Conversely, while InteractionCutMix includes the interacting object, its rigid bounding boxes inevitably carry over unwanted background artifacts. To overcome these categorical and spatial limitations, we propose ActionRegionMix. By leveraging the spatial grounding capabilities of Multimodal Large Language Models (MLLMs), ActionRegionMix operates in an open-vocabulary manner and segments both the actor and interacting objects at the pixel level, significantly reducing the background noise compared to rigid bounding boxes.

Figure 1. Comparison between bounding-box based augmentation approaches (ActorCutMix and InteractionCutMix) and our method.

2.2. Multimodal Large Language Models for Video Segmentation

Recent advances in Multimodal Large Language Models (MLLMs) [24] have enabled segmentation approaches that combine vision-language understanding with pixel-level grounding. These methods use natural language descriptions or instructions to identify and segment objects or regions, supporting queries that may require reasoning about visual content.

Several recent studies have explored the use of language descriptions to generate segmentation masks, connecting visual grounding with language understanding. Frameworks such as LISA [25], GLaMM [26], and OMG-LLaVA [27] integrate segmentation with language-based reasoning or response generation, supporting tasks ranging from identifying referred regions to generating grounded descriptions. Related developments in the video domain include open-vocabulary instance segmentation, as explored by OpenVIS [28], and language-guided video segmentation using MLLMs, as investigated by VISA [29] and VideoLISA [30]. CoT-RVS [31] and AgentRVOS [32] investigate reasoning-based and agentic pipelines, respectively, using pretrained models without additional task-specific training. These studies extend segmentation beyond fixed category labels or enable regions to be specified through natural language, providing a basis for extracting semantically relevant content from images and videos.

For ActionRegionMix, we adopt Sa2VA [18], which integrates SAM 2 [33] with an MLLM to support dense grounded understanding of both images and videos. Its combination of language-conditioned reasoning and video segmentation allows us to specify action-relevant foreground regions through textual instructions without additional fine-tuning on the target action-recognition datasets. In particular, we use this capability to target both the actor and interacting objects, rather than restricting foreground extraction to predefined detector categories or rectangular regions. The resulting pixel-level masks guide foreground–background mixing, with the aim of preserving action-relevant content while varying the surrounding scene.

3. Method

The proposed framework introduces an action-aware video augmentation pipeline that leverages the capabilities of mask-generating Multimodal Large Language Models (MLLMs). Designed to decouple human actions from their contextual backgrounds, our method synthesizes new training samples with diverse contexts aiming to reduce scene bias. As illustrated in Figure 2, the proposed pipeline operates in three main stages. First, in the prompt-based action segmentation stage (Section 3.1), we inject the target action class name into a text prompt to guide the MLLM to segment action-related regions and generate pixel-level segmentation masks in an open-vocabulary manner. Next, during the augmented video generation stage (Section 3.2), we employ a direct spatiotemporal mixing strategy where the isolated action foreground is extracted using the generated mask and directly integrated with a diverse contextual background randomly selected from other videos within the current mini-batch. Finally, the action recognition network is trained simultaneously on both the original video samples and their newly synthesized, background-swapped counterparts, ensuring that exactly one augmented sequence is introduced for every original video.

Figure 2. Overview of the proposed action-aware data augmentation framework. (1) The MLLM segments the action foreground based on a text prompt. (2) The foreground is integrated with a random background and the network is trained on both original and augmented data.

3.1. Prompt-Based Action Segmentation

In this stage, we extract action-relevant foreground regions by conditioning a multimodal segmentation model on the action class label of the input video. Specifically, we employ Sa2VA-4B [18], an end-to-end vision-language framework that grounds semantic text queries into dense pixel-level masks. In our pipeline, we guide this segmentation process using an action-specific prompt $\mathcal{P}$, formulated as: <image> Please segment all of the region related to the {class_name} action. Here, the <image> tag serves as a visual placeholder token, and {class_name} corresponds to the ground-truth action label tokenized into lowercase, space-separated words. Guided by this prompt, the model localizes the target action and generates binary spatial masks across the video frames.

Given a raw video sequence, we first decode it into a set of T frames $\mathcal{F}=\{f_1,f_2,\dots ,f_T\}$. To satisfy the token processing constraints of Sa2VA-4B, we partition $\mathcal{F}$ into M consecutive and non-overlapping temporal chunks $\{{\mathcal{C}}_1,{\mathcal{C}}_2,\dots ,{\mathcal{C}}_M\}$, each containing K frames. The model processes each chunk independently using the fixed prompt $\mathcal{P}$, producing per-chunk binary mask tensors. These masks are then concatenated along the temporal dimension to construct the complete spatio-temporal mask for the entire video:

$${\mathcal{M}}_{\mathrm{video}}={\displaystyle \underset{i=1}{\overset{M}{\mathrm{Concat}}}}\left(\Psi ({\mathcal{C}}_i,\mathcal{P})\right),$$

(1)

where $\Psi (·)$ denotes the frozen Sa2VA-4B inference model, and $\mathrm{Concat}$ represents concatenation along the temporal dimension. The resulting volume ${\mathcal{M}}_{\mathrm{video}}\in {\{0,1\}}^{T\times H\times W}$ is a binary tensor with spatial dimensions H and W. Pixels with a value of 1 represent regions actively involved in the target action, while 0 denotes background or unrelated areas.

Finally, the generated binary masks are stored at their original spatial resolution, maintaining a direct one-to-one pixel correspondence with the source video frames for downstream augmentation. Figure 3 illustrates the segmentation behavior of Sa2VA across different action categories, showing successful foreground extraction (top row), partial segmentation of action-relevant regions (middle row), and localization failure (bottom row). In the inspected examples, these errors tend to occur within individual processing chunks rather than persist throughout the entire video, leaving action-relevant information available in other temporal segments. This temporal locality limits the impact of segmentation errors on the action information retained in the augmented videos.

Figure 3. Segmentation outputs of Sa2VA across different action categories, showing complete capture (top), partial capture (middle), and failure (bottom).

3.2. Augmented Video Generation

The primary objective of this phase is to synthesize novel training samples by isolating the action foreground of a source video and compositing it onto a dynamically selected background. This strategy mitigates background bias, thereby encouraging the recognition model to rely primarily on motion patterns instead of static scene-specific biases.

Given a mini-batch of B source video clips, let $\mathcal{X}\in {\mathbb{R}}^{C\times T\times H\times W}$ denote a source clip and $\mathcal{M}\in {\{0,1\}}^{1\times T\times H\times W}$ its corresponding binary mask, where C, T, H, and W represent the number of channels, frames, height, and width, respectively. Each source clip $\mathcal{X}$ is dynamically paired with a distinct background source ${\mathcal{X}}_{bg}$ and its corresponding mask ${\mathcal{M}}_{bg}$ obtained by randomly shuffling the mini-batch, such that the foreground action from one sample is composited onto the background of another. To isolate the background context, we first suppress the original action region in ${\mathcal{X}}_{bg}$:

$${\mathcal{X}}_{\mathrm{bg}\_\mathrm{clean}}={\mathcal{X}}_{bg}\odot (1-{\mathcal{M}}_{bg}),$$

(2)

where ⊙ denotes the element-wise product. This operation effectively blacks out the actively moving regions in the background source.

Next, we extract the foreground action from the source video, expressed as $\mathcal{X}\odot \mathcal{M}$, and composite it onto the clean background. The final augmented video clip ${\mathcal{X}}_{\mathrm{aug}}$ is formulated as

$${\mathcal{X}}_{\mathrm{aug}}=\left({\mathcal{X}}_{\mathrm{bg}\_\mathrm{clean}}\odot (1-\mathcal{M})\right)+(\mathcal{X}\odot \mathcal{M}),$$

(3)

where ⊙ denotes the element-wise Hadamard product, and + represents the pixel-wise composition of mutually exclusive spatial regions. Suppressing the donor action in ${\mathcal{X}}_{bg}$ prevents conflicting motion cues from leaking into the composite clip, while non-overlapping zero-valued regions serve as a form of cutout regularization without requiring computationally heavy video inpainting. Because the background sources are dynamically paired within each mini-batch, this composition strategy introduces rich spatio-temporal diversity on the fly. Finally, the original clips and their augmented counterparts are concatenated along the batch dimension to form the final training input:

$${\mathcal{X}}_{\mathrm{total}}=[{\mathcal{X}}_1,\dots ,{\mathcal{X}}_B,{\mathcal{X}}_{\mathrm{aug},1},\dots ,{\mathcal{X}}_{\mathrm{aug},B}]\in {\mathbb{R}}^{2B\times C\times T\times H\times W}.$$

(4)

This ensures that every forward pass consistently processes exactly $2B$ samples, preserving the gradient statistics across all training steps. Unlike standard mixing strategies that interpolate ground-truth labels based on blending ratios, we maintain strict hard labels. The augmented clip ${\mathcal{X}}_{\mathrm{aug}}$ inherits the exact class label y of the source clip $\mathcal{X}$, expanding the target vector to ${\mathbf{y}}_{\mathrm{total}}=[y,y]\in {\mathbb{R}}^{2B}$. This methodological design encourages the action recognition network to rely more on foreground motion cues while aiming to reduce its dependence on background context.

4. Experiments

4.1. Datasets

Experimental evaluations were conducted on the UCF101 [34] and HMDB51 [35] datasets, which are widely utilized in action recognition tasks. The UCF101 dataset includes a total of 13,320 videos that are organized into 101 action classes, while the HMDB51 dataset includes a total of 6766 videos that are organized into 51 action classes. For C3D experiments, we report the mean accuracy across the three official train/test splits of each dataset. Experiments using 3DResNet-18 were conducted on UCF101 split 1.

4.2. Implementation Details

All experiments were conducted on a single NVIDIA GeForce RTX 4070 Ti SUPER GPU using the PyTorch 2.5.1 [36] framework. We employed two 3D CNN backbones: C3D [1] and 3DResNet-18 [3]. The 3DResNet-18 backbone was trained from scratch on UCF101 Split-1 for ablation studies investigating the effects of augmentation ratios, mask representations, and background cleaning, as well as robustness to background changes. For comparisons with existing video augmentation methods, we adopted the C3D backbone and followed the experimental protocol of InteractionCutMix [17] to ensure comparability across methods. C3D was evaluated with and without Sports1M pretraining [9], and results were averaged across the three official train/test splits of the corresponding datasets.

For 3DResNet-18 experiments, models were trained for 100 epochs using the Adam optimizer with an initial learning rate of $1\times {10}^{-4}$. Input clips consisting of 8 consecutive frames were randomly sampled from each video and resized to $342\times 256$. Random cropping to $224\times 224$ and horizontal flipping with a probability of 50% were subsequently applied to both video frames and their corresponding masks. Nearest-neighbor interpolation was used for mask resizing to preserve binary values. Training was performed with a mini-batch size of 8 and 4 gradient accumulation steps, resulting in an effective batch size of 32, with Automatic Mixed Precision (AMP) reducing memory consumption.

For C3D experiments, models were trained for 100 epochs using Stochastic Gradient Descent (SGD) with a momentum of 0.9 and a weight decay of 0.0005. The initial learning rate was set to 0.001 and reduced by a factor of 0.1 at epochs 30, 60, and 90 using a multi-step learning rate scheduler. Training employed a mini-batch size of 16 with 4 gradient accumulation steps, yielding an effective batch size of 64. Each input clip consisted of 16 consecutive frames resized to $171\times 128$ and cropped to $112\times 112$, following the standard C3D configuration. Cross-Entropy loss was used as the training objective for both backbones.

During evaluation, 10 clips were uniformly sampled along the temporal dimension of each test video. Center cropping was applied with a spatial resolution of $224\times 224$ for 3DResNet-18 and $112\times 112$ for C3D. Video-level predictions were obtained by averaging the predicted class probabilities across the 10 clips.

The Sa2VA-4B segmentation model [18] was employed exclusively for offline preprocessing. Segmentation masks were pre-computed using the publicly available model from Hugging Face [37] and stored in .npy and .npz formats. During training, the masks were loaded alongside their corresponding video frames, eliminating the need for segmentation inference within the training loop. This design prevents the computational cost of segmentation from accumulating across training epochs. Mask generation required approximately 10.98 s per video on average, corresponding to approximately 41 h for the entire UCF101 dataset on a single NVIDIA GeForce RTX 4070 Ti SUPER GPU.

4.3. Experimental Results

The proposed ActionRegionMix framework is evaluated for downstream action recognition at different augmentation ratios. Table 1 presents the classification results on the UCF101 dataset using the 3DResNet-18 backbone trained from scratch. The baseline model, trained strictly on the complete training set without any synthesized data (0% augmented ratio), yields a Top-1 accuracy of 53.69%. To investigate how performance scales with synthesized samples, we expand the training set by generating background-swapped pairs for a subset of the videos while preserving the complete original training set (100%) in all configurations. When synthesizing background-swapped counterparts for 10% of the training videos, the accuracy increases to 58.47%, providing an absolute gain of 4.78%. Furthermore, when full augmentation is applied, generating a background-swapped sequence for every original video (100% augmented ratio), the model achieves a peak Top-1 accuracy of 66.16%. This 12.47% improvement demonstrates that pairing original videos with pixel-level background-swapped variants significantly enhances model generalization.

Table 1. Classification results using the 3DResNet-18 backbone trained from scratch under different training data compositions.

Method	Dataset	Training Data	Top-1 (%)
Baseline	UCF101	100% Train	53.69
ActionRegionMix	UCF101	100% Train + 10% Augmented	58.47
ActionRegionMix	UCF101	100% Train + 100% Augmented	66.16

To evaluate the effectiveness of ActionRegionMix relative to existing scene-debiasing augmentation methods, we compare it with ActorCutMix [16] and InteractionCutMix [17]. For these comparisons, we followed the experimental setup of InteractionCutMix. To ensure consistency with our use of hard labels and no action-ratio-based filtering, we use its reported results for the hard-label setting with ${\rho }_{\min }=0$ and without REPP post-processing. Table 2 reports the video-level Top-1 and Top-5 classification accuracies on UCF101 and HMDB51 using the C3D backbone. The supervised baselines represent models fine-tuned without any scene-debiasing augmentation. As indicated in Table 2, ActionRegionMix consistently outperforms all the compared methods. When utilizing the Sports1M-pretrained C3D backbone on the UCF101 dataset, our method achieves a Top-1 accuracy of 85.10%. This corresponds to a 2.60% absolute improvement over the baseline (82.50%). In contrast, while ActorCutMix achieved 83.08% and obtained a gain of +0.58%, the subsequent study, InteractionCutMix, achieved 83.93% and obtained a gain of +1.43%. Notably, our approach nearly doubles the performance gain provided by InteractionCutMix on this dataset. The improvements are even more pronounced on the HMDB51 dataset, which is inherently more challenging due to severe scene variations and complex motion dynamics. The baseline Top-1 accuracy on HMDB51 is 52.61%. ActionRegionMix increases this metric to 56.32%, delivering an absolute gain of 3.71%. Meanwhile, InteractionCutMix and ActorCutMix achieve relatively limited gains of 1.18% and 0.66%, respectively. Additionally, when evaluated without pretraining on Sports1M, our framework achieves 61.18% Top-1 and 83.17% Top-5 accuracy, providing an initial baseline for scratch training. The relatively modest gain observed in the Sports1M-pretrained setting (+2.60% absolute gain on UCF101) is consistent with a well-established pattern in transfer learning: stronger initialization reduces the marginal contribution of any single training-time intervention, as the model already possesses rich spatiotemporal representations. This effect is particularly pronounced here because Sports1M pretraining exposes the model to visually diverse scenes, partially alleviating the scene bias that ActionRegionMix is designed to address at the data level. The more informative indicator of augmentation effectiveness is therefore the from-scratch setting, where ActionRegionMix yields a substantially larger gain (+12.47% absolute gain in Table 1), directly reflecting the augmentation’s capacity to compensate for limited background diversity in the absence of pretrained priors.

Table 2. Comparison with existing video augmentation methods on UCF101 and HMDB51. Our method is evaluated under the same experimental settings as InteractionCutMix [17].

Method	Dataset	Backbone	Pretraining	Top-1 (%)	Top-5 (%)
No augmentation a	UCF101	C3D	Sports1M	82.50	95.61
ActorCutMix [16] b	UCF101	C3D	Sports1M	83.08	95.59
InteractionCutMix [17] a	UCF101	C3D	Sports1M	83.93	96.03
Ours	UCF101	C3D	No Pretraining	61.18	83.17
Ours	UCF101	C3D	Sports1M	85.10	97.01
No augmentation a	HMDB51	C3D	Sports1M	52.61	82.68
ActorCutMix [16] b	HMDB51	C3D	Sports1M	53.27	84.38
InteractionCutMix [17] a	HMDB51	C3D	Sports1M	53.79	84.90
Ours	HMDB51	C3D	Sports1M	56.32	87.10

Note. ^a Results taken from [17]. ^b Reimplementation results reported in [17]. Bold text indicates the best-performing cases across the dataset for each performance metric.

Table 3 reports the Top-1 accuracy of the baseline and ActionRegionMix under both original and background-swapped test conditions, using a 3DResNet-18 backbone trained from scratch. In the background-swapped variants, foreground motion remains unchanged while the background is replaced with one from another clip, evaluating model robustness when scene context alters while foreground action stays constant. Background swap pairs were sampled with a fixed random seed to ensure reproducibility. The baseline model achieves 53.69% Top-1 accuracy on original test videos but suffers a dramatic accuracy drop of 36.27% when test backgrounds are replaced (falling to 17.42%), indicating a strong reliance on scene-specific cues. In contrast, ActionRegionMix reduces this drop to 11.05%, demonstrating that pixel-level foreground–background decoupling during training substantially mitigates scene bias.

Table 3. Action recognition performance under background swapping on UCF101 evaluation clips.

Model	Original (%)	Background-Swapped (%)	Drop (%)
Baseline	53.69	17.42	−36.27
ActionRegionMix	66.16	55.11	−11.05

ActionRegionMix achieves 66.16% Top-1 accuracy on original videos and 55.11% on background-swapped variants, representing a decrease of 11.05 percentage points, compared with 36.27 percentage points for the baseline. Although background replacement also reduces ActionRegionMix’s accuracy, the model preserves a much larger fraction of its performance under altered scenes. Since foreground motion remains unchanged throughout this comparison, the smaller accuracy drop indicates that ActionRegionMix is less affected by changes in scene context. This advantage also holds when the 3DResNet-18 backbone is trained from scratch, showing that ActionRegionMix can improve robustness to background changes without pretrained weights. A per-class analysis confirms that this robustness is consistent across the majority of action categories: ActionRegionMix reduces the accuracy drop relative to the baseline in 85 out of 101 classes, with the largest gains observed in scene-dependent categories such as rafting, golf swing, and bowling. Categories showing minimal improvement tend to be those where the action pattern itself is visually distinctive regardless of background context.

Table 4 presents a component-level ablation study evaluating the individual contributions of mask type and background cleaning on UCF101 classification performance. The baseline model trained without data augmentation achieves a Top-1 accuracy of 53.69%. Introducing bounding-box-based foreground swap without background cleaning increases accuracy to 61.93%, yielding a 8.24% absolute gain over the baseline. Notably, adding background cleaning to the bounding-box variant slightly reduces accuracy to 60.19%, suggesting that the cleaning step does not consistently benefit coarse mask representations: since bbox-derived masks already contain background pixels, the cleaning operation may suppress contextually relevant regions rather than isolating clean foreground content. To further quantify this difference, we measured the average spatiotemporal mask coverage ratio across the UCF101 training set: pixel-level masks cover 20.94% of the video volume on average, compared to 38.14% for their bounding-box equivalents. This 17.20 pp gap confirms that bounding-box masks enclose substantially more background content by construction, which the cleaning step cannot fully compensate for. Replacing bounding boxes with MLLM-guided pixel-level segmentation without background cleaning further elevates performance to 65.72%, and the full configuration with background cleaning achieves 66.16%, representing a 12.47% overall gain over the unaugmented baseline. Across both mask types, pixel-level segmentation consistently outperforms bounding-box-based localization, confirming that tighter foreground extraction reduces background leakage and yields cleaner composites by allowing the network to isolate human–object interactions while suppressing non-essential spatial background context.

Table 4. Ablation study on the effects of foreground mask representation and background cleaning on action recognition accuracy on UCF101.

Variant	Top-1 (%)
No Augmentation	53.69
BBox Foreground Swap	61.93
BBox Foreground Swap + BG Cleaning	60.19
Pixel Foreground Swap	65.72
Pixel Foreground Swap + BG Cleaning	66.16

The segmentation quality of Sa2VA-4B varies across action categories depending on the visual distinctiveness of the action region. For categories involving clear human–object interactions with well-defined spatial boundaries, the model produces precise pixel-level masks that closely follow the contours of the action region. In contrast, categories with ambiguous foreground–background boundaries or atypical body configurations may exhibit partial or inaccurate segmentation. As described in Section 3.1, videos are processed in independent temporal chunks to reduce computational requirements, which may introduce temporal inconsistencies, including occasional mask flickering at chunk boundaries. In the inspected examples, segmentation errors tend to be localized within individual chunks rather than persist throughout entire sequences, leaving action-relevant information available in other temporal segments. Furthermore, because the masks are pre-computed, these segmentation errors and temporal inconsistencies remain fixed across training epochs rather than introducing additional stochastic variations. Despite these limitations, the consistent improvements in recognition accuracy suggest that the generated masks provide sufficiently informative action regions for effective augmentation.

Overall, these results highlight the advantages of MLLM-guided pixel-level segmentation over bounding boxes. By isolating foreground interactions and reducing bounding-box artifacts, ActionRegionMix provides a more effective semantic mixing strategy, leading to consistent performance improvements over the evaluated scene-debiasing augmentation methods.

5. Conclusions

In this study, we introduced ActionRegionMix, a video augmentation framework designed to reduce scene bias in action recognition. Unlike existing scene-debiasing approaches, our method integrates an open-vocabulary MLLM into the augmentation pipeline to extract pixel-level action regions, including both actors and their interacting objects, through natural language prompting. By combining these regions with backgrounds from other training videos, ActionRegionMix introduces contextual diversity while preserving action-relevant content. Experimental results on UCF101 and HMDB51 demonstrate consistent performance improvements over existing bounding-box-based augmentation methods, while additional evaluations under background-swapped conditions indicate reduced reliance on scene-specific cues. These findings highlight the potential of MLLM-guided pixel-level augmentation for mitigating scene bias in action recognition.

Future work could explore adapting the segmentation model through task-specific fine-tuning to improve the reliability of action-region extraction across more diverse and challenging action categories. Additionally, alternative supervision strategies, such as soft labeling, could be investigated to further improve learning from augmented videos.