Fourier–Transformer Mixer Network for Efficient Video Scene Graph Prediction ^†

Abstract

In video scene graph prediction, the aim is to capture structured object interactions that occur over time in dynamic visual content. While recent spatiotemporal attention-based models have improved performance, they often suffer from high computational costs and limited structural consistency across long sequences. Therefore, we developed a Fourier transformer mixer network (FTM-Net), a modular, frequency-aware architecture that integrates spatial and temporal modeling via spectral operations. It incorporates a resolution-invariant Fourier Mixer for global spatial encoding and a Fast Fourier Transform (FFT)-Net-based temporal encoder that efficiently represents long-range dependencies with less complexity. To improve structural integrity, we introduce a spectral consistency loss function that synchronizes high-frequency relational patterns between frames. Experiments conducted utilizing the Action Genome dataset demonstrate that FTM-Net surpasses previous methodologies in terms of both Recall@K and mean Recall@K while markedly decreasing parameter count and inference duration, providing an efficient, interpretable, and generalizable approach for structured video comprehension.

1. Introduction

Comprehending dynamic object interactions in video is difficult because it requires organized, temporally coherent reasoning [1,2]. Video scene graph prediction (VSgP) is used to model temporal subject–predicate–object interactions, facilitating tasks such as action detection and captioning [3]. Nonetheless, current methodologies frequently encounter difficulties pertaining to global spatial context, extended sequence modeling, and the generalization of unusual relations; these difficulties are primarily attributable to the limitations of convolutional neural network (CNN)-based encoders and the inefficiencies of Transformers. To resolve this issue, we present Fourier transformer mixer network (FTM-Net), a modular, frequency-aware system that integrates a resolution-invariant Fourier Mixer for spatial encoding with an efficient Fast Fourier Transform net (FFTNet) temporal module [4]. We also propose spectral consistency loss to improve temporal coherence in the frequency domain. FTM-Net attained superior performance on the Action Genome dataset, providing enhanced accuracy, generalization, and computational efficiency.

2. Materials and Methods

2.1. Related Work

Scene graph generation (SGG) in static photos has been extensively researched. However, its application to videos poses significant challenges due to the requirement for temporal reasoning. Recent video-based SGG approaches represent spatiotemporal interactions. Nonetheless, they frequently segregate spatial and temporal processing and depend on computationally intensive Transformers. Conversely, frequency-domain methodologies, including the Fourier neural operator, Fourier-based transformer alternative, and FFTNet, enable efficient global modeling with less complexity [5]. Inspired by this, we introduce FTM-Net, a modular and resolution-invariant framework that integrates frequency-based spatial and temporal modeling for structured video prediction. FTM-Net comprises a Fourier Mixer for spatial encoding, an FFTNet temporal encoder for long-range dynamics, and a prediction head for relational inference. To improve fine-grained temporal coherence—especially for infrequent predicates—we also propose a spectral consistency loss component that synchronizes high-frequency relational components across frames [6].

2.2. Methodology

Given a sequence of frames ${{\mathit{x}}_{\mathit{t}}}_{\mathit{t}=\mathbf{1}}^{\mathit{T}}$, where x_t ∈ R^H×W×C, the model predicts the scene configuration ${\hat{\mathit{y}}}_{\mathit{t}\mathbf{+}\mathbf{1}}$ at t + 1.

$${\hat{y}}_{t+1}=f_{\mathit{\Theta }}\left(x_{1:T}\right)$$

(1)

Instead of single triples, the model outputs confidence scores for each object pair (s _ i, o _ i), resulting in the following.

$${\hat{y}}_{t+1}={(s_i,r_i,o_i)}_{i=1}^N$$

(2)

The Fourier Mixer applies 2D FFT on each x_t, followed by spectral filtering.

$${\hat{X}}_t={\mathcal{F}}_r\left(x_t\right)$$

(3)

This applies a real-valued 2D Fast Fourier Transform (FFT) to the input frame x_t, converting it from the spatial domain to the frequency domain.

$${\stackrel{\sim }{X}}_t\left[k\right]=W_k·{\hat{X}}_t\left[k\right]$$

(4)

Each frequency component k is filtered by a learnable spectral weight matrix {W} _ k to capture semantic patterns.

$$W_k=A·diag\left(\varphi \right(k\left)\right)·B$$

(5)

The filter {W} _ k is factorized using two matrices A and B with a frequency-aware diagonal term k for efficient parameterization.

$$z_t={\mathcal{F}}_r^{-1}\left({\stackrel{\sim }{X}}_t\right)$$

(6)

Then, global average pooling is defined as follows.

$${\mathit{z}}_{\mathit{t}}^{\mathit{t}\mathit{o}\mathit{k}\mathit{e}\mathit{n}}=\mathit{G}\mathit{A}\mathit{P}\left({\mathit{z}}_{\mathit{t}}\right)$$

(7)

In the FFTNet temporal encoder, L layers are defined as follows.

$${\mathit{z}}_{\mathit{t}}^{(\mathit{l}+\mathbf{1})}=\mathit{L}\mathit{N}({\mathit{z}}_{\mathit{t}}^{\left(\mathit{l}\right)}+{\mathcal{F}}_{\mathit{t}}^{-\mathbf{1}}({\mathit{P}}^{\left(\mathit{l}\right)}·{\mathcal{F}}_{\mathit{t}}\left({\mathit{z}}_{\mathit{t}}^{\left(\mathit{l}\right)}\right)\left)\right)$$

(8)

The prediction head combines z _ T^{(L)} with object features, leading to Equation (9).

$$\hat{\mathit{i}}=\mathit{M}\mathit{L}\mathit{P}\left(\right[{\mathit{z}}_{\mathit{T}}^{\left(\mathit{L}\right)};\mathit{f}\mathit{e}\mathit{a}\mathit{t}({\mathit{s}}_{\mathit{i}},{\mathit{o}}_{\mathit{i}})\left]\right)$$

(9)

Sigmoid is used for multi-label classification as follows.

$${\mathit{L}}_{\mathit{t}\mathit{a}\mathit{s}\mathit{k}}=-{\mathit{\Sigma }}_{\mathit{j}=1}^{\left|\mathit{R}\right|}\left[{\mathit{r}}_{\mathit{j}}\mathbf{l}\mathbf{o}\mathbf{g}\mathit{\sigma }\right({\hat{\mathit{r}}}_{\mathit{j}})+(\mathbf{1}-{\mathit{r}}_{\mathit{j}}\left)\mathbf{l}\mathbf{o}\mathbf{g}\right(\mathbf{1}-\mathit{\sigma }\left({\hat{\mathit{r}}}_{\mathit{j}}\right)\left)\right]$$

(10)

Applying FFT to the prediction and ground truth $\hat{\mathbf{x}}$ _ {t + 1}, x _ {t + 1}, the spectral consistency loss is calculated using Equation (11).

$${\mathit{L}}_{\mathit{s}\mathit{p}\mathit{e}\mathit{c}}=(\mathbf{1}/|{\mathit{K}}_{\mathbf{h}\mathbf{i}\mathbf{g}\mathbf{h}}\left|\right){\mathit{\Sigma }}_{\mathit{k}\in {\mathit{K}}_{\mathit{h}}\mathit{i}\mathit{g}\mathit{h}}\left|\right|{\hat{\mathit{X}}}_{\mathit{t}+\mathbf{1}}\left[\mathit{k}\right]-{\mathit{X}}_{\mathit{t}+\mathbf{1}}\left[\mathit{k}\right]|{|}_2^2$$

(11)

The total loss is estimated as follows.

$${\mathit{L}}_{\mathit{t}\mathit{o}\mathit{t}\mathit{a}\mathit{l}}={\mathit{L}}_{\mathit{t}\mathit{a}\mathit{s}\mathit{k}}+{\mathit{\lambda }}_{\mathit{s}\mathit{p}\mathit{e}\mathit{c}}·{\mathit{L}}_{\mathit{s}\mathit{p}\mathit{e}\mathit{c}}$$

(12)

The overall workflow is presented in Figure 1.

Figure 1. Comprehensive architecture of FTM-Net.

For each t, FFT and spectral filtering are performed using Equation (13).

$${\hat{\mathit{X}}}_{\mathit{t}}={\mathcal{F}}_{\mathit{r}}\left({\mathit{x}}_{\mathit{t}}\right),{\mathit{z}}_{\mathit{t}}=\mathit{G}\mathit{A}\mathit{P}\left({\mathcal{F}}_{\mathit{r}}^{-\mathbf{1}}\right({\stackrel{\sim }{\mathit{X}}}_{\mathit{t}}\left)\right)$$

(13)

The training Algorithm 1 was created as follows.

Algorithm 1 End-to-end training workflow of FTM-Net

Require: Video frames {x_t}_{t=1}^T, ground-truth relations y{t+1} Ensure: Top-ranked relation triples ${\hat{y}}_{\{t+1\}}$ = {(si, ri, oi)}

1: for t = 1 to T do 2:          ${\stackrel{\sim }{X}}_t$ ← $\mathcal{F}r$(xt)                                            ▷ 2D FFT 3: for each frequency mode k ∈ $\mathcal{K}$ do 4:                  ${\hat{X}}_t$[k] ← Wk ${\stackrel{\sim }{X}}_t$[k]                      ▷ Spectral filtering 5: end for 6:          zt ← GAP($\mathcal{F}r$−1(${\hat{X}}_t$))                          ▷ Spatial token 7: end for 8: {zt}_{t=1}^T ← FFTNet({zt}_{t=1}^T)          ▷ Temporal encoding 9: for each object pair (si, oi) do 10:         ${\hat{r}}_i$ ← MLP([zT; feat(si, oi)])         ▷ Relation logits 11: end for 12: Compute ${\mathcal{L}}_{task}$ via binary cross-entropy 13: Compute ${\mathcal{L}}_{spec}$ via FFT-based spectral loss 14: ${\mathcal{L}}_{total}$ ← ${\mathcal{L}}_{task}$ + λspec ${\mathcal{L}}_{spec}$ 15: Update model parameters θ using AdamW optimizer 16: return ${\hat{y}}_{\{t+1\}}$ = Top-K {(si, ri, oi)} from ${\hat{r}}_i$

Algorithm 1 outlines the end-to-end optimisation procedure of FTM-Net, including spectral filtering, temporal encoding, and relation inference.

3. Evaluation of the Method

3.1. Experimental Setup

We assessed FTM-Net using the Action Genome dataset, comprising 10,000 video clips tagged with 35 object classes and 50 predicates, categorized into attention, spatial, and contact types. Videos were divided into 70, 15, and 15% for training, testing, and validating and segmented into 16-frame clips at a rate of 2 frames per second. Object characteristics were derived with Faster R-CNN pretrained on Visual Genome. Evaluation was conducted using the “with constraint” procedure, utilizing the complete 50-class predicate set, which facilitates multi-label prediction [7]. We evaluated the PredCLS and SGCLS tasks utilizing Recall@20/50 and mean Recall@50, averaging the outcomes across three seeds. FTM-Net was evaluated against a graph convolutional network tailored for visual relationship detection (VRD-GCN), MotifNet, GPS-Net, SGG, spatio-temporal transformer for video-based (STTran), and TRACE using uniform preprocessing methods [8]. In training, PyTorch 1.9.0 on an RTX 3090, the AdamW optimizer, cosine decay, warm-up, and a batch size of 32 were used for 100 epochs. The temporal encoder comprised four FFTNet layers (dimension = 256) and thirty-two frequency modes, with a spectral loss weight of 0.1.

3.2. Comparison with State-of-the-Art Models

We compared FTM-Net with five strong baselines—VRD-GCN, GPS-Net, MotifNet, STTran, and TRACE—on the Action Genome dataset under both PredCLS and SGCLS settings [9,10]. All models were employed using the same object detector and frame segmentation pipeline for fair comparison. FTM-Net achieved the highest Recall@K and Mean Recall@K scores (Table 1). While the absolute gains in Recall@50 over TRACE were moderate (+1.9 PredCLS, +1.5 SGCLS), the improvements in mR@50 were more substantial, indicating better performance on rare predicates. These gains stem from FTM-Net’s frequency-aware design: the Fourier Mixer provides resolution-invariant spatial encoding, FFTNet captures long-range temporal dynamics, and spectral consistency loss reinforces fine-grained relation prediction.

Table 1. Performance comparison with respect to Action Genome.

Method	R@20 (PredCLS)	R@50 (PredCLS)	mR@50 (PredCLS)	R@20 (SGCLS)	R@50 (SGCLS)	mR@50 (SGCLS)
VRD-GCN	15.4	22.6	7.8	10.5	18.9	5.2
MotifNet	19.7	27.3	9.2	14.1	22.4	6.4
GPS-Net	21.3	30.1	10.6	15.6	24.7	7.8
STTran	24.5	33.2	12.3	17.8	27.0	8.6
TRACE	25.7	34.9	13.0	18.5	28.4	9.1
FTM-Net	27.5	36.8	15.4	19.7	29.9	11.3

3.3. Ablation Study

We conducted ablation studies under the PredCLS setting on Action Genome to assess the contribution of FTM-Net’s components. We tested three variants: (1) removing spectral consistency loss, (2) replacing the Fourier Mixer with a shallow CNN, and (3) substituting FFTNet with a 1D CNN—while keeping the prediction head fixed. Removing spectral loss led to the largest drop in mR@50 (–1.4), highlighting its importance for modeling fine-grained structures (Figure 2). The Fourier Mixer improved both R@50 and mR@50 via resolution-invariant spatial encoding, while FFTNet captured long-range temporal dependencies efficiently. We evaluated a minimal version that excludes both Fourier Mixer and FFTNet, using basic CNNs for spatial and temporal modeling. As expected, this configuration yielded the weakest performance, reaffirming the benefit of frequency-aware modeling in both domains.

Figure 2. Ablation study results obtained with respect to Action Genome (PredCLS).

3.4. Qualitative Analysis

Figure 3 illustrates FTM-Net’s predictions and ground truth with respect to two sample frames. The model accurately captures spatial and functional relations, including man–holding–umbrella and hat–on–man. Errors arise from part–whole confusion (e.g., head–of misclassified as head–on) or visual proximity (e.g., pants–on–horse as a false positive). These results highlight strong generalization with occasional ambiguity in fine-grained or overlapping relations.

Figure 3. Qualitative analysis of FTM-Net forecasts.

3.5. Efficiency and Model Complexity

We compared FTM-Net’s efficiency with VRD-GCN, STTran, and TRACE. FTM-Net achieved the best balance of accuracy and efficiency, with just 29.4M parameters, 18.7 giga floating-point operations (GFLOPs), and a 42 ms inference time per 16-frame clip. Unlike Transformer models (O(T²)), FFTNet offers scalable temporal modeling (O(T log T)) with full parallelism (Table 2). Its compact, modular design is ideal for deployment on edge or real-time platforms.

Table 2. Model efficiency comparison with respect to 16-frame video clips.

Method	Number of Parameters (M)	GFLOPs	Inference Time (ms)
VRD-GCN	32.1	12.8	35.6
STTran	47.9	38.4	74.3
TRACE	41.5	33.2	68.1
FTM-Net (Ours)	29.4	18.7	42.0

4. Conclusions and Future Work

We introduced FTM-Net, a modular, frequency-sensitive framework for effective video scene graph prediction. FTM-Net integrates a resolution-invariant Fourier Mixer for spatial encoding with an FFTNet-based temporal module, effectively capturing global semantics and long-range dynamics with minimal complexity [9,10]. A spectral consistency loss component enhances structural precision, especially for infrequent or unclear relations. Experiments using Action Genome demonstrated the model yielded superior performance in both Recall@K and mR@K while achieving a smaller model size and lower inference time. Ablation and qualitative analyses validated the efficacy and comprehensibility of each module. In future endeavors, we intend to expand FTM-Net to accommodate streaming or untrimmed films for online graph prediction. We investigated adaptive frequency selection, spectrum attention fusion, and the integration of multimodal signals (e.g., audio and text) with self-supervised pretraining to improve generalization across various video comprehension tasks [11].