Disentangling Interaction and Intention for Long-Tail Pedestrian Trajectory Prediction

Abstract

Pedestrian trajectory prediction remains a challenging task, particularly in long-tail scenarios where goal distributions are sparse and inter-agent behaviors are uncertain. In this work, we propose to disentangle the trajectory prediction task into two complementary components: interaction modeling and intention modeling. For interaction modeling, we introduce an adaptive meta-strategy that proactively extracts latent and rare-yet-critical interaction patterns often overlooked by conventional trajectory-only approaches. For intention modeling, we propose Continuous Waypoint Slot-Driven Prototypical Contrastive Learning (PCL). It adapts prototype learning to the multi-modal reality where conventional PCL fails to model diverse and continuous goal distributions. Capitalizing on the complementary strengths of both components, we orchestrate a unified frequency-based fusion module that seamlessly integrates interaction and intention modeling, yielding enhanced overall prediction accuracy. In particular, our method is model-agnostic and can be seamlessly incorporated into a wide range of existing prediction frameworks. Extensive experiments on several datasets demonstrate that our approach not only achieves consistent performance gains in standard settings, but also significantly alleviates degradation on hard or long-tail trajectory samples.

1. Introduction

Trajectory prediction, essential for intelligent systems [1,2,3], has evolved from kinematic models [4,5,6] and traditional ML [7,8,9] to data-driven deep learning approaches [9,10,11,12,13,14] that better capture complex interactions. A recent study [15] employed contrastive learning to mitigate accuracy degradation in rare trajectories.

Although these studies [15,16] concentrate on the hard case issue, they rarely analyze the two fundamental sources of long-tail behaviors: abnormal social interactions and rare goal intentions. Consequently, by applying a monolithic autoencoder to inherently continuous trajectories without disentangling these underlying causes, their approaches suffer from limited reconstruction quality [17] and produce ill-suited discrete representations. Furthermore, their evaluation, often restricted to a single algorithm, leaves generalization to other representative methods unclear. As a result, they struggle to capture rare events while maintaining accuracy on common cases. Such rare samples often stem from either abnormal inter-agent interactions or uncommon goals that require distinct modeling strategies. To address this, we propose explicitly decoupling interaction and intention into modular components, allowing independent adaptation and improved generalization across both frequent and rare scenarios.

In modeling social interactions, we address two key limitations: the lack of explicit, interpretable uncertainty representation beyond Gaussian noise [18] and the failure to capture rare-but-critical abnormal behaviors (e.g., collision avoidance). To overcome these, a richer representation using distance (to describe interaction intensity), speed (to describe motion intent), and angle (to describe social structure) is necessary to provide a more physically meaningful and dynamic description than a static snapshot of relative positions. As shown in Figure 1a, pedestrians exhibiting extremely high speeds or remaining stationary can indicate abnormal interactions that an ego agent should pay more attention to. In intention modeling, although many studies generate goal candidates to assist intention modeling, they [15,16,19] do not emphasize the continuity and long-tail characteristic of continuous trajectory goals, such as those in Figure 1b, resulting in limited interpretability and performance. In this work, we propose an approach that jointly considers interactions and intentions together.

Meteorologists predict future weather patterns by analyzing satellite imagery that captures the evolving formations of anomalous cloud masses rather than examining the distribution of individual droplets [20]. Inspired by this high-level perspective, our work focuses on extracting abnormal social interactions and intentions by identifying structured patterns rather than modeling each in isolation. To address the challenges in modeling pedestrian intention and interaction in complex scenarios, we propose the following approach: (1) We first decouple social interaction from intention. (2) Social interactions are mapped to a Distance–Velocity–Angle (DVA) space modeled as multivariate normal distributions. A Gaussian Mixture Model (GMM) classifies interactions, enabling offline generation of an abnormal interaction database for scenario-specific aggregation. (3) An enhanced PCL framework incorporates a GMM-based module that generates pseudo-labels of rare intentions to supervise motion prediction while maintaining continuity, which eliminates ill-suited discrete representation and the need for training a separate autoencoder. (4) To ensure robust performance across both challenging cases and the entire dataset, we integrate (2) and (3) into a frequency-sensitive decoder which regards rare intentions predicted by (3) as goal-driven to generate complete trajectories. Our method demonstrates consistent improvements over several recent representative trajectory prediction models on multiple benchmarks.

The contributions of our work are summarized as follows: (1) a framework of decoupled interaction–intentions for motion forecasting; (2) a DVA space that models interaction uncertainty coupled with offline GMM-based preprocessing for efficient extraction and aggregation of abnormal interactions; (3) an improved prototypical contrastive learning method for rare intentions under continuous goal labels; (4) a frequency-sensitive decoder to combine abnormal interactions and intentions together which could seamlessly connect to existing methods. To our knowledge, this is the first work studying the uncertainty and long-tail of two important factors (interactions and intentions) together. Extensive experiments and in-depth analysis confirm that our method consistently outperforms several recent representative methods.

2. Related Work

Social Interaction Modeling. Social pooling methods [21,22,23] propagate neighboring pedestrians’ temporal features to the target agents. NADP [24] is a decoupled pedestrian trajectory prediction network that uses a near-aware attention module to extract core spatiotemporal features for prediction. While early attention-based approaches [11,23,25] employ factorized structures $O\left(N^2\right)$ to model social interactions, recent methods like T-MLSTG [26] (GNN-based), PMITra [27] (GNN-based), QCNet [28] (GNN-based), IA-STDGNN [29] (DGNN-based), and FJMP [30] (directed graphs) exhibit strong structural biases. SocialCircle [31] innovatively adopts angle-space aggregation, inspired by marine echo localization, but its uniform averaging of meta-components in angle space may overlook critical abnormal interactions in another feature dimension, which may be a key focus for motion prediction.

Intention Capturing in Trajectory Prediction. Agents have uncertain intentions (e.g., a person standing at an intersection can either go straight or turn left, as long as they behave in accordance with traffic rules). Most of the present methods regard the future trajectory of the ego agent as an M mixture distribution, where M denotes ’multi-modalities’. To model such uncertainty, PGP [18] decomposes it into lateral variability of anchor-based map candidates and longitude variability, which can be regarded as the random noise of normal distribution. MELON [32] decouples trajectory decoding into specialized modules with adaptive spatiotemporal uncertainty quantification and a streaming prediction scheme, achieving state-of-the-art performance on complex urban traffic datasets. PPT [33] uses a progressive learning architecture, modeling short-term goals and long-term goals in two stages. Current generative approaches [13,25,34,35,36,37,38] suffer from unreliable uncertainty modeling due to their uninterpretable noise sampling, highlighting the need for robust heuristic rules.

Long-Tail Distribution in Trajectory Prediction. Manual class balancing fails with increasing categories, inducing long-tail distributions. Previous studies [39,40] have addressed this imbalance for categorical targets. Prototypical contrastive learning (PCL) [41,42] addresses long-tailed data distributions by learning underlying features through instance discrimination, as demonstrated in previous work [41,43,44]. Pedestrian trajectories exhibit long-tail distributions (e.g., turns vs. straight paths) owing to inertial motion in unchanged scenes as well. FEND [15] proposes a future enhanced contrastive learning framework and a hypernetwork to recognize these long-tailed patterns. TrACT [16] proposes to incorporate richer information on training dynamics into a prototypical contrastive learning framework. Hi-SCL [19] fights long-tailed trajectory prediction with hierarchical wave-semantic contrastive learning. Unlike prior methods [15,16,19] that uniformly amplify minority patterns and distort the majority feature space, our approach selectively extracts long-tail signals as a plugin module, preserving the core representations and accuracy of normal data. As a continuous multi-modal regression problem, applying traditional prototype learning methods to pedestrian trajectory prediction is not feasible. Prior methods [15,16,19] often ignore the continuity imbalance of trajectory intention distributions, while applying direct binning method like [45] in the trajectory prediction task generates memory-intensive $O\left(N^2\right)$ 2D grids, making the precise calibration of bin size critical as well. Moreover, grid-based discretization of 2D coordinates fails to preserve the underlying data structure in clustering tasks, as rigid spatial partitioning disregards intrinsic density variations.

3. Method

3.1. Problem Formulation

Denote the past trajectory of ego pedestrian i with $t_h$ timesteps as $O^i=(p_1^i,p_2^i,\dots ,p_{t_h}^i)$, where 2D $p_t^i=(x_t^i,y_t^i)$. We aim to forecast their future trajectory $F^i=(p_{t_h+1}^i,p_{t_h+2}^i,\dots ,p_{t_h+t_f}^i)$ based on $O^i$ and the past trajectories of all their $N_a$ neighbors $O^{/i}$ = {$O^j$∣$1\le j$$\le N_a$}, where j ∈ neighbor(i). ‘Neighbors’ refers to non-ego agents (e.g., optionally filtered by distance). The motion forecasting task is to find an optimal model ${\theta }^{*}=\underset{\theta }{max}P\left(F^i\right|O^i,O^{/i})$. In real-world scenarios, pedestrians usually decide their trajectories in a three-step manner that can be decomposed into a realistic cognitive-behavioral process: risk perception → goal formation → motion execution. Previous studies [46,47,48] disentangle the latent space and construct a Bayesian Network in deep learning tasks. In our work, we disentangle lateral variations into a social interaction part $z_{social}$($z_{soc}$) and an individual intention part $z_{intention}$( $z_{int}$), where we manage to minimize $P\left(F^i\right|O^i,O^{/i})$ via Equation (1):

$$P\left(F^i\right|O^i,O^{/i})={\int }_{z_{soc}}{\int }_{z_{int}}P\left(z_{soc}\right|O^i,O^{/i})·P\left(z_{int}\right|z_{soc})·P\left(F^i\right|z_{int},z_{soc})d_{z_{soc}}{d}_{z_{int}}.$$

(1)

Equation (1) illustrates the decision-making process: A pedestrian first observes the surrounding environment $z_{soc}$, which subsequently informs and refines their intended goal $z_{int}$. The future trajectory ($F^i$) is then generated based on the combination of this contextual information and the refined goal. Figure 2a–c shows our overall framework, corresponding to the above three items in Equation (1). The key symbol definitions in the Methods section are given in

Table 1. Key symbol definitions.

Section	Symbol	Explanation
Section 3.2	$r_{dis}^{i\leftarrow j}$	Relative distance of pedestrian j to i.
	$r_{vel}^j$	Absolute velocity of pedestrian j to i.
	$r_{\theta }^{i\leftarrow j}$	Relative angle of pedestrian j to i.
	${\rho }_{dis\_vel}$	Correlation coefficient between relative distance and absolute velocity.
	${\rho }_{vel\_\theta }$	Correlation coefficient between absolute velocity and relative angle.
	${\rho }_{dis\_\theta }$	Correlation coefficient between relative distance and relative angle.
	${\Theta }_{abn}$	GMM to extract abnormal social interactions.
	$R_{meta}^{i\leftarrow j}$	Abnormal social interaction meta-component (relative distance, absolute velocity, and relative angle) of pedestrian j to i.
	$A^i$	Abnormal neighbor set for pedestrian i.
	$s_n^i$	The n-th aggregated Gaussian component of the abnormal neighbors feature for pedestrian i.
	${\lambda }_n$	Weight coefficient for the n-th abnormal social interaction component whose addition is 1.
Section 3.3	${\Theta }_{int}$	GMM to extract rare intentions.
	${\Theta }_{e1}$	Intention GMM for the first stage to capture glocal goal distributions.
	${\Theta }_{e2}$	Intention GMM for the second stage to capture long-tail intentions.
	$N_{int}$	Number of intention components in ${\Theta }_{int}$.
	$e^i$	Intention and endpoint to predict for pedestrian i.
	$P_{int}^i$	Clustered intention labels by ${\Theta }_{int}$ for pedestrian i.
	$v_i$	Intention embeddings for pedestrian i.
	$v_j$	Arbitrary sample in the same batch as i.
	$|N_i^k|$	Number of neighbors which belong to the k-th nearest intention cluster for sample i.
	$c_i^k$	Prototype of cluster which the k-th neighbor GMM component of pedestrian i belongs to.
	$\alpha$	Momentum coefficient in our PCL algorithm.
	$\tau$	Temperature coefficient in our PCL algorithm.

.

3.2. Abnormal Social Interaction Modeling

We focus on how to model the social interaction part $z_{soc}$, which is used to simulate risk perception.

DVA Multivariate Gaussian Space. Previous studies [31,49] treat the neighbor social interactions to ego pedestrian i as social meta-components $R_{meta}^i=\left\{r_{meta}^{i\leftarrow j}\right|1\le j\le N_a\}$. In this work, we construct meta-components as $r_{meta}^{i\leftarrow j}=\{$$r_{dis}^{i\leftarrow j}$,$r_{vel}^j$,$r_{\theta }^{i\leftarrow j}$}.

• Relative Distance $r_{dis}^{i\leftarrow j}$. Neighbor agents exert different influences on the target agent depending on their current distances from the ego one. Formally, for any neighbor j of ego pedestrian i,

  $$r_{dis}^{i\leftarrow j}={\parallel p_{t_h}^i-p_{t_h}^j\parallel }_2.$$

  (2)
• Absolute Velocity $r_{vel}^j$. Not only do high-velocity neighbors pose a greater threat, but static agents can too, especially if they are close by, as this may prompt the target pedestrian to take proactive avoidance measures. Formally, for any neighbor j of an ego pedestrian i,

  $$r_{vel}^j={\parallel p_{t_h}^j-p_0^j\parallel }_2.$$

  (3)
• Relative Angle $r_{\theta }^{i\leftarrow j}$. Pedestrians often take advantage of the relative orientation angle to judge their surroundings (e.g., if there are crowds to the north). Formally, for any neighbor j of an ego pedestrian i,

  $$r_{\theta }^{i\leftarrow j}=atan2(y_{t_h}^i-y_{t_h}^j,x_{t_h}^i-x_{t_h}^j).$$

  (4)

We project $R_{meta}$ into a multivariate Gaussian space $\{{\mu }_{meta},{\Sigma }_{meta}\}$ to reflect its uncertainty, where

$$\left\{\begin{array}{c}{\mu }_{meta}^{i\leftarrow j}=[{\mu }_{dis}^{i\leftarrow j},{\mu }_{vel}^{i\leftarrow j},{\mu }_{\theta }^{i\leftarrow j}];\\ \\ {\Sigma }_{meta}=\left(\begin{array}{ccc}{\sigma }_{dis}^2& {\rho }_{dis\_vel}{\sigma }_{dis}{\sigma }_{vel}& {\rho }_{dis\_\theta }{\sigma }_{dis}{\sigma }_{\theta }\\ {\rho }_{dis\_vel}{\sigma }_{dis}{\sigma }_{vel}& {\sigma }_{vel}^2& {\rho }_{vel\_\theta }{\sigma }_{vel}{\sigma }_{\theta }\\ {\rho }_{dis\_\theta }{\sigma }_{dis}{\sigma }_{\theta }& {\rho }_{vel\_\theta }{\sigma }_{vel}{\sigma }_{\theta }& {\sigma }_{\theta }^2\end{array}\right).\end{array}\right.$$

(5)

Abnormal Social Meta-Components. Abnormal social behaviors at the scene level have certain commonalities (e.g., in congested environments, agents may change direction to accelerate overtaking, while high-velocity agents will prioritize avoiding static neighbors on their paths forward). However, these interactions in DVA space may be long-tail and influenced by the combined effects of multiple independent variables (e.g., in the previous example, ‘high-velocity’ corresponds to absolute velocity $r_{vel}$ and ‘on the forward path’ corresponds to relative angle $r_{\theta }$—the two factors interact to create the aforementioned long-tail scenarios). To deal with these long-tail social behaviors, a GMM ${\Theta }_{abn}={\sum }_{n=1}^{N_{abn}}{\lambda }_n\mathcal{N}({\mu }_n,{\Sigma }_n)$ with $N_{abn}$ components is constructed to extract abnormal interactions. Inspired by [31], our abnormal interactions represent another spatial interactive context so that we can handle the abnormal interaction sequence along with the object trajectory ($N_{abn}$ = $t_h$) for easy alignment and concatenation as well. In detail, for all social meta-components in the training set of $N_{tr}$ pedestrians $R_{tr}=$$\{R_{meta}^1,R_{meta}^2,\dots ,R_{meta}^i,\dots ,R_{meta}^{N_{tr}}\}$, we use ${\Theta }_{abn}$ to fit them for the first time, the optimization goal L of which is to maximize the maximum-likelihood function in Equation (6) through an EM algorithm.

$$logL\left({\Theta }_{abn}\right|R_{tr})=\sum _{i=1}^{N_{tr}}log(\sum _{n=1}^{N_{abn}}{\lambda }_n\mathcal{N}(R_{meta}^i;\{{\mu }_n,{\Sigma }_n\})),$$

(6)

where ${\sum }_{n=1}^{N_{abn}}{\lambda }_n=1$ and $\forall$${\lambda }_n\ge 0$. $\mathcal{N}({\mu }_n,{\Sigma }_n)$ is the probability density function of a single trivariate Gaussian component representing the distribution of relative distance, absolute velocity, and relative angle.

We compute posterior probabilities $logP\left(R_{meta}^i\right|{\Theta }_{abn})=log{\sum }_{n=1}^{N_{abn}}{\lambda }_n·\mathcal{N}(R_{meta}^i;{\Theta }_{abn})$ for each social meta-component in $R_{tr}$, then extract abnormal interactions $R_{abn}=\left\{R_{meta}^i\right|logP\left(R_{meta}^i\right|{\Theta }_{abn})<{ϵ}_{abn}\}$ by filtering out normal ones where ${ϵ}_{abn}$ is the threshold for the definition of abnormal social interactions. After that, ${\Theta }_{abn}$ manages to fit the $R_{abn}$ set for the second time with the initial center of the normal social meta-components of the first fit stage via Equation (7):

$$logL\left({\Theta }_{abn}\right|R_{abn})=\sum _{i=1}^{|R_{abn}|}log(\sum _{n=1}^{N_{abn}}{\lambda }_n\mathcal{N}(R_{abn}^i;\{{\mu }_n,{\Sigma }_n\})),$$

(7)

where $\{{\mu }_n,{\Sigma }_n\}$ is named the n-th abnormal interaction base after the second fit. The workflow diagram in Figure 3 describes the above two-stage GMM fitting procedure, which can be conducted in an offline manner.

When training or performing inference online, $I_{abn}$ is the indicator function used to judge whether the pedestrian j is an abnormal neighbor to which the pedestrian ego i should pay attention. ${ϵ}_{abn2}$ is the filtering threshold for abnormal social interactions during training and evaluation.

$${\mathbb{I}}_{abn}^{i\leftarrow j}=\left\{\begin{array}{cc}0\hfill & logP\left(R_{meta}^{i\leftarrow j}\right|{\Theta }_{abn})\le {ϵ}_{abn2};\hfill \\ 1\hfill & \mathrm{Otherwise}.\hfill \end{array}\right.$$

(8)

Denote $A^i=\left\{R_{meta}^{i\leftarrow j}\right|{\mathbb{I}}_{abn}^{i\leftarrow j}=1,j\in neighbor\left(i\right)\}$ as the abnormal neighbor set for i. For an abnormal agent j, for i, $R_{meta}^{i\leftarrow j}$ will be decomposed and projected to several abnormal social component bases generated during training via Equation (9):

$$P\left(R_{meta}^{i\leftarrow j}\right)=\sum _{n=1}^{N_{abn}}{\lambda }_n^{i\leftarrow j}·\mathcal{N}(R_{meta}^{i\leftarrow j};\{{\mu }_n,{\Sigma }_n\}),$$

(9)

where ${\sum }_{n=1}^{N_{abn}}{\lambda }_n^{i\leftarrow j}=1$ and ${\lambda }_n^{i\leftarrow j}\ge 0$.

Under the assumption of mutual independence among abnormal interaction bases, we aggregate abnormal neighbors $s^i$ according to their projection lengths ${\lambda }_n^{i\leftarrow j}$ to these bases as in Equation (10), where n represents the n-th abnormal social interaction base. Note that we adopt the reparameterization trick [50] to generate the n-th component of $s^i$ here.

$$s_n^i\sim \mathcal{N}({\displaystyle \frac{1}{|A^i|}}{\Sigma }_j{\lambda }_n^{i\leftarrow j}{\mu }_n,{\displaystyle \frac{1}{{|A^i|}^2}}{\left({\Sigma }_j{\lambda }_n^{i\leftarrow j}\right)}^2{\Sigma }_n).$$

(10)

Our serialized abnormal social uncertainty feature $z_{abn\_soc}$ is defined as Equation (11), where $g_{embed}$ is denoted as an MLP layer with the tanh activation function. Note that if an ego agent has no abnormal interaction component, we pad its social abnormal feature with zero.

$$z_{abn\_soc}=\left\{\begin{array}{cc}{g}_{embed}\left(\left[s_1,s_2,\dots ,s_n\right]\right)\hfill & |A^i|>0;\hfill \\ g_{embed}\left(\left[0,0,\dots ,0\right]\right)\hfill & \mathrm{Otherwise}.\hfill \end{array}\right.$$

(11)

The characteristic of abnormal social interaction $z_{abn\_soc}$$\in R^{d_s}$ will be concatenated (∥) with the past trajectory characteristic of the ego agent $f_{beh}$$\in R^{d_i}$ produced by the backbone encoder. Then, a temporal attention module $g_{fuse}$ combined with a list of MLP modules is set to form our final social interaction feature $z_{soc}$:

$$z_{soc}=g_{fuse}\left(\left[f_{beh}\parallel z_{abn\_soc}\right]\right).$$

(12)

3.3. Rare Intention Modeling

In this part, we focus on modeling intention feature $z_{int}$, which is used to simulate goal formation through a novel prototypical contrastive learning (PCL) method to address rare intentions. We initialize a learnable waypoint slots set $S_{int}$ with length of multi-modalities M. Another multi-head attention layer $S_{attn}$ uses $S_{int}$ as query $\mathcal{Q}$, $z_{soc}$ as key $\mathcal{K}$, and value $\mathcal{V}$ to encode inputs of PCL $z_{int}$ to form $P\left(z_{int}\right|z_{soc})$ via Equation (13):

$$z_{int}=S_{attn}(\mathcal{Q}=S_{int},\mathcal{K}=z_{soc},\mathcal{V}=z_{soc}).$$

(13)

The number of our intention base is $N_{int}$. We adopt a two-stage GMM fitting paradigm where the first stage captures global goal distributions and the second stage focuses on long-tail intentions. In detail, we regard the last points of future trajectories in training set $e=[e^1,e^2,\dots ,e^{N_{tr}}]$ as goal intentions and first use a GMM with ${\displaystyle \frac{N_{int}}{2}}$ components ${\Theta }_{e1}$ as $\{{\mu }_{e1},{\Sigma }_{e1}\}$ to fit them in the training set. The goals with the lowest log-likelihood score $logP\left(e\right|{\Theta }_{e1})$ among those filtered by the $R_{int}$ ratio, denoted as $e_r$, are filtered out and defined as rare intentions. Then, we utilize another GMM ${\Theta }_{e2}$ with ${\displaystyle \frac{N_{int}}{2}}$ components to fit $e_r$. We then combine the two GMMs ${\Theta }_{e1}$ and ${\Theta }_{e2}$ to form the new, larger ${\displaystyle \frac{N_{int}}{2}}\ast 2=N_{int}$-component GMM ${\Theta }_{int}$ of Equation (14), where {$1,2,\dots ,n,\dots ,N_{int}$}, corresponding to Gaussian components $\{({\mu }_1^e,{\Sigma }_1^e)\dots ,({\mu }_{{\displaystyle \frac{N_{int}}{2}}}^e,{\Sigma }_{{\displaystyle \frac{N_{int}}{2}}}^e),({\mu }_{{\displaystyle \frac{N_{int}}{2}}+1}^{e_r},{\Sigma }_{{\displaystyle \frac{N_{int}}{2}}+1}^{e_r}),\dots ,({\mu }_{N_{int}}^{e_r},{\Sigma }_{N_{int}}^{e_r})\}$ through optimization goal Equation (15) for contrastive supervision later, which means the first ${\displaystyle \frac{N_{int}}{2}}$ Gaussian (${\Theta }_{e1}$) components of ${\Theta }_{int}$ are generated by all endpoints e while the second ${\displaystyle \frac{N_{int}}{2}}$ Gaussian components of ${\Theta }_{int}$ (${\Theta }_{e2}$) are generated by the $10\%$ long-tail endpoints $e^r$.

$$\left\{\begin{array}{cc}{\Theta }_{int}=\{{\Theta }_{e1},{\Theta }_{e2}\};\hfill & \\ {\Theta }_{e1}=\{({\mu }_1^e,{\Sigma }_1^e),\dots ,({\mu }_{{\displaystyle \frac{N_{int}}{2}}}^e,{\Sigma }_{{\displaystyle \frac{N_{int}}{2}}}^e\};\hfill & \\ {\Theta }_{e2}=\{({\mu }_{{\displaystyle \frac{N_{int}}{2}}+1}^{e_r},{\Sigma }_{{\displaystyle \frac{N_{int}}{2}}+1}^{e_r})\dots ,({\mu }_{N_{int}}^{e_r},{\Sigma }_{N_{int}}^{e_r})\}.\hfill & \end{array}\right.$$

(14)

$$\left\{\begin{array}{cc}logL({\Theta }_{e_1}|e)=\sum _{i=1}^{N_{tr}}log(\sum _{n=1}^{{\displaystyle \frac{N_{int}}{2}}}{\lambda }_n^e\mathcal{N}(e^i;({\mu }_n^e,{\Sigma }_n^e)));\hfill & \\ logL({\Theta }_{e_2}|e_r)=\sum _{i=1}^{{\displaystyle \frac{N_{tr}}{10}}}log(\sum _{n={\displaystyle \frac{N_{int}}{2}}+1}^{N_{int}}{\lambda }_n^{e_r}\mathcal{N}(e_r^i;({\mu }_n^{e_r},{\Sigma }_n^{e_r}))).\hfill & \end{array}\right.$$

(15)

In intention-clustered label $P_{int}$ prediction for sample i, ${\Theta }_{int}$ is used to find the intention Gaussian component $P_0$ with maximum posterior probability via Equation (16):

$$\begin{array}{c}\hfill P_{int}=\underset{P_0}{\arg \max }logP\left(e^i\right|{\Theta }_{int}^{P_0})=\underset{P_0}{\arg \max }\mathcal{N}(e^i;{\Theta }_{int}^{P_0})P_0\in (1,N_{int}).\end{array}$$

(16)

Having obtained the clustered labels $P_{int}$, we proceed to leverage them in our contrastive learning framework to supervise the learning of intention features $z_{int}$ defined in Equation (13). Notably, a final MLP $S_{proj}$ prevents conflicts between motion forecasting loss and contrastive learning loss, which means $z_{int}$ is passed through $S_{proj}$ as the last layer for the prototypical contrastive learning but not in intention prediction. We denote the PCL encoder module list as $f_{\theta }$ = [$S_{int}$, $S_{attn}$, $S_{proj}$].

Traditional unsupervised contrastive learning methods like MoCo [39] get feature input without gradients. To align with them, we pretrain the model with only Winner-Takes-All (WTA) ADE loss for trajectory points ${\mathcal{L}}_{pred}$, the calculation of which is described in Equation (17), where ${\widehat{p}}_t^{\left(M\right)}$ means the M-th-modality predicted trajectory point of timestep t outputted by our frequency-based decoder D (whose details are shown in Section 3.4) and $p_t$ means the ground truth.

$$\left\{\begin{array}{cc}{p}_t=D(z_{int},z_{soc});\hfill & \\ {\mathcal{L}}_{pred}({\widehat{p}}_t,p_t)=\underset{M}{min}\sum _{t=t_h+1}^{t_p+t_f}{\parallel {\widehat{p}}_t^{\left(M\right)}-p_t\parallel }_2.\hfill \end{array}\right.$$

(17)

The WTA strategy chooses one slot among M slots with the best ADE where the gradients exclusively backpropagate to preserve diversity. Then, we freeze (*) all parameters of the pretrained backbone prediction encoder $B_{enc}^{*}$ shown in Figure 2a, which eliminates the need to design a dual momentum encoder $f_{\theta }^{\prime }$ containing the dual parameters of the previous abnormal social interaction modeling when we train the PCL because the input of PCL $z_{soc}$ (which actually is the output of $B_{enc}^{*}$ as well) receives no grad.

Instead of conducting contrastive loss directly on the past trajectory context, we conduct our prototypical contrastive learning on multi-modal slots by using the WTA strategy in advance to maintain prediction diversity. Consequently, the feature clustering and updating procedure must be performed per iteration rather than per epoch because the specific slot with the best $f_{ADE}$ defined in Equation (17) to be used in loss computation remains undetermined and distinct for different samples. Since the clustering operates solely on x and y dimensions of intention, the computation is highly efficient.

We then focus on how to define our prototypical contrastive learning loss (PCL loss) ${\mathcal{L}}_{ProtoNCE}$. In Equation (21), ${\mathcal{L}}_{ProtoNCE}$ consists of instance-wise term $L_{ins}$ and instance-prototype term ${\mathcal{L}}_{proto}$. The first term brings the sample features within the class closer, while the second term maximizes inter-cluster separation. Standard PCL [39,40] assigns a sample to a single discrete class label $P_{int}=\left[P_0\right]$, whereas our goal intention follows a continuous distribution. For each sample i, we use a finite set of discrete components $P_{int}^i=[P_0^i,P_1^i,\dots ,P_{K-1}^i]$ with length K to mimic the continuity. As a result, our approach handles continuous intention distributions through K-nearest GMM component allocation in the PCL loss, effectively simulating distributional continuity with multiple discrete elements but not a single component. We verify the effectiveness of the modeling in our experiments. In detail, given an endpoint intention $e^i$, we only find the component $P_{int}$ with the highest posterior log likelihood via Equation (16).

For highly efficient computation, we want to directly look up the other $K-1$ components $[P_1,\dots ,P_{K-1}]$ based on $P_0$. In detail, we calculate the pairwise Kullback–Leibler (KL) divergence $K{L}_{int}\in R^{N_{int}\ast N_{int}}$ of any two multivariant Gaussian components in ${\Theta }_{int}$.

The indexes of the K-nearest Gaussian distributions $K_{id}\in R^{N_{int}\ast K}$ for each intention component are calculated to look up the KL divergence of the K-nearest neighbors (including itself) $K{L}_{id}\in R^{N_{int}\ast K}$ from $K{L}_{int}$, which is used to construct continuous labels and calculate our continuous contrastive loss later.

Denoting $v_i$ as the intention embeddings ($z_{int}$, defined in Equation (13)) of sample i, we construct positive sample feature pairs $(v_i,v_{i+})$ and negative sample feature pairs $(v_i,v_j)$. Notably, to simulate continuity of endpoints in the (x,y) two-dimensional space, $v_{i+}$ adopts a hierarchical structure containing all samples belonging to the K-nearest components of ${\Theta }_{int}$. In detail, $v_{i+}^k$ means arbitrary samples belonging to the k-th nearest component of sample i. We look up $K{L}_{id}$ and denote KL divergences between $P_{int}$ and $P_0$ of sample i as {$KL\left(P_0^i\right|P_0^i)$, $KL\left(P_1^i\right|P_0^i)$, …, $KL\left(P_{K-1}^i\right|P_0^i)$}. Equation (18) shows our ${\mathcal{L}}_{ins}$, where $v_j$ denotes an arbitrary sample in the same batch as i and r denotes batch size. $\sigma$ is a softmax function used to assign weights based on the KL divergence between Gaussian components, and $\tau$ is the temperature coefficient.

$${\mathcal{L}}_{ins}=-\sum _{i=1}^r\sum _{k=1}^K{\displaystyle \frac{1}{|N_i^k|}}\sum _{i_{+}=1}^{|N_i^k|}\sigma (-KL\left(P_k^i\right|P_0^i))·log{\displaystyle \frac{exp(v_i·v_{i_{+}}^k/\tau )}{{\sum }_{j=1}^{r}exp(v_i·v_j/\tau )}}.$$

(18)

The prototypical features $C=[c_1,\dots ,c_n,\dots ,c_{N_{int}}]$ are updated per batch sample according to their maximum-likelihood intention-clustered labels $P_0$ via Equation (19), where $\alpha$ is the momentum coefficient if the batch has any sample with the label and $\mathbb{I}(P_0^j==n)$ is the indicator function to judge whether $P_0$ of sample j belongs to the n-th intention Gaussian component.

$$c_n^{\prime }=\alpha ·c_n+(1-\alpha )·{\displaystyle \frac{{\Sigma }_j\mathbb{I}(P_0^j==n)·v_j}{\left|\mathbb{I}\right(P_0^j==n\left)\right|}}.$$

(19)

Equation (20) shows our ${\mathcal{L}}_{proto}$, where $c_i^k$ is the prototype of the cluster which the k-th nearest neighbor GMM component of sample i belongs to, and $c_j$ is the prototype of an arbitrary cluster j. In summary, our approach refines intention modeling by specifically targeting challenging edge-case scenarios through the training approach above. Algorithm 1 shows the whole process of our rare intention prototypical contrastive learning.

$${\mathcal{L}}_{proto}=-\sum _{i=1}^r\sum _{k=1}^K\sigma (-KL\left(P_k^i\right|P_0^i))·log{\displaystyle \frac{exp(v_i·c_i^k/\tau )}{{\sum }_{j=1}^{N_{int}}exp(v_i·c_j/\tau )}}.$$

(20)

$${\mathcal{L}}_{ProtoNCE}={\mathcal{L}}_{ins}+{\mathcal{L}}_{proto}.$$

(21)

Algorithm 1 Intention Prototypical Contrastive Learning

Input: KL divergence of K-nearest intention GMM components $K{L}_{id}$, past trajectories X, predicted trajectory in advance ${\widehat{p}}_t$, ground truth of future trajectory $p_t$, past timesteps $t_h$, future timesteps $t_f$, cluster centroid feature C, momentum coefficient $\alpha$. Parameter: intention GMM ${\Theta }_{int}$, frozen backbone encoder $B_{enc}^{*}$, PCL encoder $f_{\theta }$, momentum PCL encoder $f_{\theta }^{\prime }$.   1: Let $f_{\theta }^{\prime }=f_{\theta }$   2: C←0.   3: while not MaxEpoch do   4:    for x in DataLoader(X) do   5:      e = $p_{t_h+t_f}$. {Regard the last point of future trajectory e as intention.}   6:      Let $P=\underset{n}{topK}logP\left(e\right|{\Theta }_{int}^n)$ {Intention pseudo-labels $P_{int}$ as Equation (16).}   7:      $z_{soc}=B_{enc}^{*}\left(X\right)$.   8:      $z_{int}=f_{\theta }\left(z_{soc}\right)$, $z_{int}^{\prime }=f_{\theta }^{\prime }\left(z_{soc}\right)$.   9:      $M^{*}=\underset{M}{argmin}{\mathcal{L}}_{pred}({\widehat{p}}_t,p_t).$ {Lookup min ADE index $M^{*}$ in advance as Equation (17).} 10:      Update prototype features C according to Equation (19). 11:      Calculate ${\mathcal{L}}_{ProtoNCE}$ based on $z_{int}\left[M^{*}\right],z_{int}^{\prime }\left[M^{*}\right],C$ as Equations (18), (20) and (21). 12:      $\theta =SGD(\theta ,{\mathcal{L}}_{ProtoNCE})$. 13:      Momentum update ${\theta }^{\prime }$ based on $\theta$. 14:    end for 15: end while

3.4. Frequency-Sensitive Decoder Combining Interactions and Intentions

In this part, we propose a novel frequency-based decoder D to integrate the social interaction part $z_{soc}$ introduced in Section 3.2 and the intention part $z_{int}$ introduced in Section 3.3, corresponding to $P\left(F^i\right|z_{int},z_{soc})$, then obtain final motions $F^i=D(z_{int},z_{soc})$ to execute. We have $z_{soc}\in R^{t_h\ast d}$ and $z_{int}\in R^{M\ast d}$, where $t_h$ is past timesteps and M is the number of multi-modalities. As in Equation (22), we regress endpoints directly based on $z_{int}$ with an FC layer intention decoder $D_{int}$ to obtain goal endpoints $e\in R^{M\ast 2}$.

$$e=\sum _{t=t_h+1}^{t_h+t_f}ve{l}_t=D_{int}\left(z_{int}\right).$$

(22)

The key is to adopt an endpoint-driven method, and the rest of the trajectory points will be conditionally completed under the drive of the terminal location in the frequency domain through Discrete Fourier Transform (DFT). By decomposing a signal into its constituent sinusoids, DFT allows deep learning models to identify and leverage repetitive patterns and structures within the data that are often obscure in the original time domain [51]. To interpolate intermediate trajectory points, we demonstrate a gate-weighting, as shown in Equation (23):

$$W=[W_s,W_i]=D_{\sigma }(\underset{Socials}{\underbrace{D_{sw}((z_{int}@z_{soc})@z_{soc})}},\underset{Intentioni}{\underbrace{D_{iw}\left(z_{int}\right)}}),$$

(23)

where $D_{sw}$ and $D_{iw}$ are MLPs for social part s and intention part i, respectively, and $D_{\sigma }$ is an MLP with a softmax activation function used to normalize weights between the two parts. The output social weight and intention weight are gathered as $W=\left[W_s\parallel W_i\right]$, which is used to generate the Alternate-Current Component (ACC) of the velocity in the frequency domain after passing through an FC layer $D_{ac}$. The ACC is generated via Equation (24), which includes the Fourier components except the first Direct-Current Component (DCC):

$$ACC=D_{ac}(W_s\ast s+W_i\ast i).$$

(24)

Because of the theorem that the real part of the DCC after the Discrete Fourier Transform (DFT) equals the sum of all points in the temporal series [52], we concatenate goal endpoints e with the zero image part to form the DCC, which can be regarded as the accumulation of instantaneous velocity $ve{l}_t$ at all future steps. Finally, we concatenate the DCC with the ACC and employ an inverse Discrete Fourier Transform (iDFT) layer to reconstruct future velocity profiles whose cumulative sum is exactly the DCC, that is, the endpoint e, so that we obtain accurate predicted trajectory $F^i$, as shown in Equation (25):

$$F^i=CumSum(iDFT(\underset{DCC}{\underbrace{e+0j}},ACC)).$$

(25)

3.5. Loss Function

${\mathcal{L}}_{ProtoNCE}$ could hardly bring more benefits to easy samples, so we adopt a gate ${\theta }^{\prime }$ to stop PCL loss on easy samples. In contrast to indicating a deterministic hardness of the samples [15], we determine hardness of the samples based on ${\mathcal{L}}_{pred}$, predicted in advance, which can be dynamically adjusted during the training process. Having denoted the WTA strategy’s advanced calculated ${\mathcal{L}}_{pred}$, we define our loss function $\mathcal{L}$ via Equation (26), where $\lambda$ is an indicator value defined via Equation (27). ${\theta }^{\prime }$ is the threshold to filter out hard samples.

$$\mathcal{L}={\mathcal{L}}_{pred}+\lambda ·{\mathcal{L}}_{ProtoNCE}.$$

(26)

$$\left\{\begin{array}{cc}\lambda =1\hfill & {\mathcal{L}}_{pred}>{\theta }^{\prime }\mathrm{and}\mathrm{is}\mathrm{not}\mathrm{pretraining};\hfill \\ \lambda =0\hfill & {\mathcal{L}}_{pred}\le {\theta }^{\prime }\mathrm{or}\mathrm{is}\mathrm{pretraining}.\hfill \end{array}\right.$$

(27)

4. Experiment

4.1. Experimental Setup

Datasets. We use two pedestrian motion forecasting datasets, ETH [53]-UCY [54] and SDD [55], in this work. Recent studies on ETH-UCY primarily used cross-dataset validation, which means training on four scenarios and testing on the held out one. We retain this setting in our approach.

• ETH-UCY is a dataset of pedestrian walking scenes consisting of five sub-scenarios: eth, hotel, univ, zara1, and zara2. A sample interval of 0.4 s is conducted in the length of the previous observer, $t_h=$ 3.2 s/0.4 s = 8 steps, and future prediction of $t_f=$ 4.8 s/0.4 s = 12 steps.
• Stanford Drone Dataset (SDD) is a drone dataset of human behaviors on campus. A total of 60 drone videos are used to extract 290,243 trajectories (8 steps as observed steps and 12 steps as future steps to predict) partitioned into 60% to train, 20% to validate, and 20% to test.

As in previous studies [12,31,49,56], some preprocessor layers are used to transform the coordinates of the trajectories into scene-centric ones. ’Move’ means normalizing trajectory points $p_t$ at timestep t based on the current absolute position of ego agent $p_{t_h}^i$ via Equation (28) to get moved trajectory points $p_t^{\prime }=(x^{\prime },y^{\prime })$. ’Rotate’ means rotating the moved historical trajectory $p_t^{\prime }=(x^{\prime },y^{\prime })$ based on the current target agent’s heading $\theta$ to get final preprocessed trajectory points $p_t^{″}=(x^{″},y^{″})$ via Equation (29).

$$p_t^{\prime }=p_t-p_{t_h}^i.$$

(28)

$$p_t^{″}=\left(\begin{array}{c}{x}^{″}\\ y^{″}\end{array}\right)=\left(\begin{array}{cc}cos\theta & sin\theta \\ -sin\theta & cos\theta \end{array}\right)\left(\begin{array}{c}{x}^{\prime }\\ y^{\prime }\end{array}\right).$$

(29)

Backbone prediction networks. Since our method can be regarded as a plugin, we will briefly introduce some of the latest outperforming backbone prediction networks that are used:

• Multi-Style Network (MSN) [10] provides multi-style predictions with a series of style channels, each of which is bound to a unique behavior.
• View Vertically (${\mathit{V}}^{\mathbf{2}}$-Net) [12] transforms agents’ trajectories into the frequency domain to obtain potential characteristics which could not be extracted in the time domain.
• E-${\mathit{V}}^{\mathbf{2}}$ Net [56] introduces Haar Transform instead of Fourier Transform and proposes a bilinear structure to model dimension interactions.
• SocialCircle(Plus) (-SC/-SCP) [31,49] is an alternative structure which can be plugged into the existing SOTA methods mentioned above to promote their performance. Inspired by the location of marine animals, SocialCircle aggregates social interactions according to their relative directions. Moreover, SocialCirclePlus concentrates on physical interactions as well as social interactions.

These approaches predict key points followed by linear speed interpolation [10,12,31,49,56], achieving competitive ADE/FDE on ETH-UCY and the SDD.

Evaluation Protocol and Implementation Details. In order to eliminate the influence of hyperparameters on the experimental results, we fix all the official optimal hyperparameters of the backbone prediction networks to reproduce the results before our approach is activated. The epoch with the best ADE of key points is used to represent the experimental results of a single trial. As in previous work, we adopt leave-one-out cross-dataset validation for ETH-UCY to verify whether our method has a good generalization ability to new scenarios. Specifically, we choose four subsets of ETH-UCY (eth, hotel, univ, zara1, zara2) as a training set and the remaining one as a validation subset. Training and evaluation are carried out on a 24G VRAM NVIDIA 4090 GPU. In our model, the 50 nearest neighbors of the ego agent are used to compute social interactions. The number of abnormal meta-components n is set to the same as the observation steps (eight in our study) for convenient concatenation. ${ϵ}_{abn}$ during the offline stage is set to −2 and ${ϵ}_{abn2}$ is set to 0. It should be noted that we do not make any additional changes to the backbone structures other than the improvements described in the paper. It takes us about 30 min to train completely once on the univ dataset and about 3 h to train completely once on the SDD.

Metrics. The pedestrian motion forecasting task measures the prediction accuracy of the generated M (M = 20) trajectories with the best average displacement error $minAD{E}_{20}$ and the best final displacement error $minFD{E}_{20}$. Their calculation can be denoted as Equations (30) and (31), where the hat mark ‘^{^}’ means the predicted trajectory points.

$$minAD{E}_{20}=\underset{M}{min}\sum _{t=t_h+1}^{t_p+t_f}{\parallel {\widehat{p}}_t^{\left(M\right)}-p_t\parallel }_2.$$

(30)

$$minFD{E}_{20}=\underset{M}{min}{\parallel {\widehat{p}}_{t_h+t_f}^{\left(M\right)}-p_{t_h+t_f}\parallel }_2.$$

(31)

Training Details. 

Table 2. Implementation details we used in this work. K is short for key points. ✓ means the preprocessing steps we conducted before model training.

	Size		Preprocessor		Backbone	Hyperparameters
	$\parallel {\mathit{N}}_{\mathit{train}}\parallel$	$\parallel {\mathit{N}}_{\mathit{test}}\parallel$	Move	Rotate		bsz	lr	Epochs	$\mathit{K}$
eth	36,784	2614	✓	✓	$V^2$-Net-SCP	1000	$3\times {10}^{-4}$	200	4 8 11
					E-$V^2$-Net-SCP	1000	$3\times {10}^{-4}$	200	4 8 11
					MSN-SCP	1000	$3\times {10}^{-4}$	200	11
hotel	38,323	1075	✓	✓	$V^2$-Net-SCP	1000	$4\times {10}^{-4}$	200	4 8 11
					E-$V^2$-Net-SCP	1000	$3\times {10}^{-4}$	200	4 8 11
					MSN-SCP	1000	$4\times {10}^{-4}$	200	11
univ	15,064	24,334	✓	✓	$V^2$-Net-SCP	1000	$6\times {10}^{-4}$	300	4 8 11
					E-$V^2$-Net-SCP	1000	$1\times {10}^{-3}$	200	4 8 11
					MSN-SCP	1000	$3\times {10}^{-4}$	200	11
zara1	37,042	2356	✓	-	$V^2$-Net-SCP	1000	$3\times {10}^{-4}$	200	4 8 11
					E-$V^2$-Net-SCP	1000	$4\times {10}^{-4}$	200	4 8 11
					MSN-SCP	1000	$3\times {10}^{-4}$	200	11
zara2	33,488	5910	✓	-	$V^2$-Net-SCP	1000	$3\times {10}^{-4}$	200	4 8 11
					E-$V^2$-Net-SCP	1000	$2\times {10}^{-4}$	250	4 8 11
					MSN-SCP	1000	$3\times {10}^{-4}$	200	11
SDD	251,617	38,626	✓	-	$V^2$-Net-SC	1000	$3\times {10}^{-4}$	200	4 8 11
					$V^2$-Net-SCP	1000	$3\times {10}^{-4}$	200	4 8 11
					E-$V^2$-Net-SC	1000	$2\times {10}^{-4}$	200	4 8 11
					E-$V^2$-Net-SCP	1000	$2\times {10}^{-4}$	200	4 8 11
					MSN-SC	1000	$2\times {10}^{-4}$	200	11
					MSN-SCP	1000	$2\times {10}^{-4}$	200	11
					Trans-SC	1000	$4\times {10}^{-4}$	250	4 8 11
					Trans-SCP	1000	$4\times {10}^{-4}$	250	4 8 11

shows training details such as the preprocessing steps and hyperparameters for several backbone predictors and datasets. Our settings are the same as the original settings for backbone predictors without any change. Notably, the recent state-of-the-art methods shown in Table 2 adopt a strategy which only predicts key points and then utilizes linear speed interpolation to complete the whole trajectory. ‘K’ in Table 2 denotes the timesteps of key points.

4.2. Comparisons to State-of-the-Art Methods

We first design extensive experiments to verify the effectiveness of abnormal interactions modeling.

Cross-Validation Improvements with Abnormal Priors on ETH-UCY. As shown in

Table 3. The averaged ADE/FDE results across 5 ETH-UCY subsets in cross-dataset validation. ‘-abn’ indicates our abnormal interaction module; bold shows our improvement.

Models (ETH-UCY)	Year	-abn	${\mathit{ADE}}_{20}$	${\mathit{FDE}}_{20}$
EigenTraj	2023	-	0.21	0.34
TUTR	2023	-	0.21	0.36
LED	2023	-	0.21	0.33
MSN	2023	-	0.21	0.34
EqMotion	2023	-	0.21	0.35
PPT	2024	-	0.20	0.31
MSN-SCP	2025	-	0.215	0.374
		✓	0.208	0.355
$V^2$-Net-SCP	2025	-	0.188	0.312
		✓	0.185	0.302
E-$V^2$-Net-SCP	2025	-	0.189	0.309
		✓	0.185	0.305

, our learned abnormal social priors enhance the performance on ETH-UCY. $V^2$-Net-SCP-abn outperforms PPT by 8% ADE and 2.5% FDE. Although MSN-SCP lags behind $V^2$-Net-SCP/E-$V^2$-Net-SCP, abnormal interaction modeling still improves it by 2.4% ADE and 4.0% FDE.

SDD. As shown in

Table 4. ADE/FDE results on the SDD with abnormal social interaction modeling ‘-abn’. Bold means improvement.

Models (SDD)	ADE/FDE	Models (SDD)	ADE/FDE
LED	8.48/11.36	LB-EBM	8.87/15.61
FlowChain	9.93/17.17	AgentFormer	10.18/16.91
UPDD	6.59/13.50	IMP	8.98/15.54
RAN	10.97/19.95	EigenTraj	7.42/12.49
LG-Traj	7.80/12.79	PPT	7.03/10.65
$V^2$-Net	7.12/11.39	E-$V^2$-Net	6.57/10.49
$V^2$-Net-SC	6.71/10.66	E-$V^2$-Net-SC	6.54/10.36
$V^2$-Net-SC-abn	6.59/10.60	E-$V^2$-Net-SC-abn	6.48/10.32
$V^2$-Net-SCP	6.59/10.39	E-$V^2$-Net-SCP	6.44/10.22
$V^2$-Net-SCP-abn	6.38/10.04	E-$V^2$-Net-SCP-abn	6.38/10.08
MSN	7.69/12.16	Transformer	17.44/33.36
MSN-SC	7.49/12.12	Trans-SC	16.47/32.08
MSN-SC-abn	7.32/11.78	Trans-SC-abn	15.57/30.93
MSN-SCP	7.32/11.76	Trans-SCP	16.11/31.43
MSN-SCP-abn	7.25/11.51	Trans-SCP-abn	15.70/31.16

, $V^2$-Net-SCP-abn surpasses $V^2$-Net-SCP by 3.2% ADE and 3.4% FDE. Although MSN models perform below $V^2$-Net, our abnormal interaction modeling still enhances MSN-SC by 2.3% ADE and 2.8% FDE. Notably, MSN-SC-abn matches the performance of MSN-SCP without physical input (e.g., RGB images). Even with simple backbones, our abnormal interaction plugin boosts a Transformer model by 10.7% ADE and 7.3% FDE.

Performance on Long-Tail Cases. 

Table 5. Cross-dataset validation on ETH-UCY for abnormal interaction modeling ‘-abn’ and rare intention modeling ‘-r’. ‘Dataset’: validation subset; baseline: $V^2$-Net-SCP-abn. Bold means the best ADE/FDE metric of each dataset.

Dataset	$-\mathit{abn}$	$-\mathit{r}$	Top 1%	Top 5%	Top 10%	All
eth	-	-	1.186/2.379	0.796/1.540	0.650/1.240	0.261/0.412
	✓	-	1.189/2.318	0.774/1.496	0.638/1.201	0.255/0.400
	✓	✓	1.070/2.131	0.713/1.407	0.591/1.150	0.258/0.421
hotel	-	-	0.643/1.128	0.411/0.712	0.331/0.563	0.109/0.158
	✓	-	0.644/1.217	0.391/0.724	0.321/0.565	0.108/0.152
	✓	✓	0.592/1.085	0.369/0.646	0.305/0.519	0.108/0.155
univ	-	-	1.617/3.376	0.986/2.053	0.767/1.562	0.251/0.446
	✓	-	1.432/2.938	0.897/1.828	0.708/1.429	0.250/0.444
	✓	✓	1.419/2.874	0.825/1.610	0.657/1.258	0.251/0.446
zara1	-	-	1.234/2.589	0.660/1.355	0.499/0.992	0.180/0.308
	✓	-	1.075/2.160	0.596/1.132	0.462/0.853	0.174/0.282
	✓	✓	1.054/2.070	0.584/1.106	0.453/0.843	0.174/0.289
zara2	-	-	1.267/2.684	0.723/1.478	0.548/1.075	0.137/0.232
	✓	-	1.192/2.506	0.701/1.441	0.535/1.044	0.137/0.233
	✓	✓	1.170/2.387	0.676/1.345	0.515/0.972	0.136/0.233
all	-	-	1.189/2.431	0.715/1.428	0.559/1.086	0.188/0.311
	✓	-	1.106/2.228	0.672/1.324	0.533/1.018	0.185/0.302
	✓	✓	1.061/2.109	0.633/1.223	0.504/0.948	0.185/0.309

demonstrates the effectiveness of our rare intention modeling ‘-r’ through cross-dataset validation on ETH-UCY. As shown in

Table 6. ADE/FDE (m) for top 1–10% of long-tail samples in the SDD. ‘-abn’: abnormal interaction; ‘-r’: rare intention. Bold: the best test set performance for each comparison group. Underline: the best long-tail performance for each comparison group.

Models (SDD)	$-\mathit{abn}$	$-\mathit{r}$	Top 1% ↓	Top 5% ↓	Majority (95%) ↓	Top 10% ↓	Majority (90%) ↓	All ↓
Y-Net	-	-	65.82/134.01	34.72/67.46	6.54/8.96	-	-	7.93/11.88
Y-Net + FEND	-	-	57.58/108.61	31.27/57.98	6.64/9.24	-	-	7.87/11.68
MSN-SCP	-	-	80.43/124.90	41.19/70.43	5.55/8.68	29.95/51.75	4.82/7.33	7.33/11.77
	✓	-	80.80/120.16	40.40/67.57	5.52/8.56	29.30/49.62	4.81/7.28	7.26/11.51
	✓	✓	78.01/105.87	38.14/57.97	5.31/8.04	27.65/43.28	4.65/6.90	6.95/10.54
$V^2$-Net-SCP	-	-	67.00/121.30	35.76/63.99	5.05/7.55	26.27/45.90	4.40/6.42	6.59/10.37
	✓	-	63.60/115.40	33.79/60.40	4.94/7.39	25.03/43.74	4.31/6.30	6.38/10.04
	✓	✓	62.69/110.50	32.96/58.05	4.84/7.16	24.35/42.09	4.24/6.10	6.25/9.70
E-$V^2$-Net-SCP	-	-	69.61/127.92	36.10/65.62	4.88/7.30	26.25/46.50	4.24/6.19	6.44/10.22
	✓	-	63.34/112.78	33.93/60.33	4.93/7.43	25.13/43.88	4.30/6.32	6.38/10.08
	✓	✓	60.47/108.30	32.13/57.22	4.82/7.21	23.89/41.74	4.22/6.15	6.19/9.71
Trans-SCP	-	-	168.55/345.48	95.17/197.17	11.94/22.71	70.12/144.96	10.10/18.82	16.10/31.43
	✓	-	167.92/340.14	93.45/194.35	11.62/22.59	68.86/143.12	9.80/18.74	15.71/31.18
	✓	✓	69.63/115.91	35.83/60.32	5.36/7.80	26.33/44.01	4.72/6.70	6.88/10.43

, while FEND [15] improves the performance of the top 5% of cases with the degradation of 95% majority-class accuracy, our E-$V^2$-Net-SCP-abn-r achieves comparable performance gains in challenging cases and effectively preserves the accuracy of the majority class, establishing a new SOTA (6.19/9.71 ADE/FDE) on the SDD benchmark as well. Despite omitting frequency-domain analysis and sophisticated architectures, our rare intention extraction approach ‘-r’ attains a performance (6.88/10.43) comparable to complex models when implemented on simple Transformer baselines. Further improvements of long-tailed samples on other backbone models are observed.

4.3. Discussions and Ablation Studies

Discussion I: Threshold Analysis of Abnormal Interactions. Our ablation study reveals two key threshold-dependent patterns in

Table 7. Ablation study of the abnormal interaction extraction threshold on the SDD. ∥$N_{abn}^{test}$∥ means the number of abnormal interactions extracted during inference. Bold means the best ADE/FDE metric.

Model	${\mathit{ϵ}}_{\mathit{abn}}$	∥${\mathit{N}}_{\mathit{abn}}^{\mathit{train}}$∥	${\mathit{ϵ}}_{\mathit{abn}2}$	∥${\mathit{N}}_{\mathit{abn}}^{\mathit{test}}$∥	ADE/FDE
$V^2$-Net-SCP-abn	−4	16.6k	−4	2.7k	6.45/10.22
	−2	53.9k	0	5.2k	6.43/10.14
	−2	53.9k	−2	7.1k	6.38/10.04
	0	309.0k	0	87.8k	6.43/10.11
E-$V^2$-Net-SCP-abn	−4	16.6k	−4	2.8k	6.49/10.26
	−2	57.2k	0	5.1k	6.38/10.08
	−2	57.2k	−2	7.8k	6.41/10.14
	0	293.4k	0	35.2k	6.44/10.22

: (1) For offline abnormal interaction detection (${ϵ}_{abn}$), V²-Net-SCP-abn achieves peak performance (1.1% ADE/1.8% FDE gain) at ${ϵ}_{abn}$ = −2, with degradation beyond this threshold (0.8% ADE/0.7% FDE loss in (−2.0)) due to noisy abnormal interaction meta-components. (2) The optimal inference filtering threshold (${ϵ}_{abn2}$) for abnormal interaction extraction exhibits architecture dependence, requiring 7.1k samples for V²-Net-SCP-abn versus 5.1k for E-V²-Net-SCP-abn, while insufficient extraction consistently degrades model performance. The study establishes ${ϵ}_{ab{n}_2}$ = −2 as the optimal choice for SDD.

Discussion II: Settings of the Rare Intention Modeling. In

Table 8. Ablation study of rare intention modeling ‘-r’ on univ. Our baseline model is $V^2$-Net-SCP-abn. $N_{int}$ is the intention GMM components number and ${\theta }^{\prime }$ is the activation threshold for ${\mathcal{L}}_{ProtoNCE}$. Bold means the best ADE/FDE metric.

Plugin	${\mathit{N}}_{\mathit{int}}$	${\mathit{\theta }}^{\prime }$	Top 1%	Top 5%	Top 10%	All
-	-	-	1.62/3.38	0.99/2.05	0.77/1.56	0.25/0.44
-abn	-	-	1.43/2.94	0.90/1.83	0.71/1.43	0.25/0.44
-abn-r	512	0.4	1.57/3.26	0.92/1.87	0.71/1.41	0.25/0.43
	512	0.6	1.42/2.87	0.83/1.61	0.66/1.26	0.25/0.45
	512	0.8	1.51/3.05	0.88/1.75	0.69/1.33	0.25/0.45
	256	0.6	1.52/3.14	0.89/1.76	0.69/1.34	0.25/0.44
	1024	0.6	1.54/3.17	0.90/1.78	0.69/1.34	0.25/0.43

, we analyze the effect of the number of GMM components with rare intention $N_{int}$ and the abnormal interaction activation threshold (${ϵ}_{abn}$ and ${ϵ}_{abn2}$) for ${\mathcal{L}}_{ProtoNCE}$ on the univ dataset. We can see that, for $V^2$-Net-SCP-abn, $N_{int}=512$, and ${\theta }^{\prime }=0.6$ is the best choice. This is because, when ${\theta }^{\prime }$ is too small, more easy samples will disturb the generation of features of the long-tailed class. On the other hand, when ${\theta }^{\prime }$ is too large, fewer long-tailed samples are taken into account. Both factors affect the accuracy of the model. Apart from that, selecting the proper component number of intention GMM is essential. Intuitively, excessive clustering dilutes prototype representativeness, while insufficient clustering overlooks minority-class prototypes.

Discussion III: Selection of K-Nearest Intention GMM Components and ${\mathit{R}}_{\mathit{int}}$ Rare Intention Definition. Next, we verify the effectiveness of the K-nearest intention GMM component selection shown in

Table 9. Ablation study of K-nearest GMM components for rare intention modeling on univ dataset. Baseline: $V^2$-Net-SCP-abn. AI: average inference time (ms). FI: fast inference time (ms). Bold means the best ADE/FDE metric of each dataset.

Dataset	K	${\mathit{R}}_{\mathit{int}}$	Top 1%	Top 5%	Top 10%	All	AI	FI
SDD	-	-	63.60/115.40	33.79/60.40	25.03/43.74	6.38/10.04	43.7	40.7
	1	0.1	56.47/94.44	31.30/54.70	23.79/41.81	6.77/11.36	42.8	40.8
	5	0.05	61.95/111.65	32.93/58.85	24.41/42.62	6.29/9.88	42.6	40.9
	5	0.1	62.69/110.50	32.96/58.05	24.35/42.09	6.25/9.70	43.1	40.6
	5	0.2	62.73/112.54	33.16/59.26	24.54/42.91	6.30/9.88	43.2	40.7
univ	1	0.1	1.43/2.92	0.83/1.61	0.65/1.26	0.25/0.45	-	-
	2	0.1	1.46/2.93	0.85/1.64	0.67/1.27	0.25/0.44	-	-
	5	0.1	1.42/2.87	0.83/1.61	0.66/1.26	0.25/0.45	-	-
	6	0.1	1.50/3.02	0.87/1.69	0.68/1.30	0.25/0.44	-	-
	7	0.1	1.54/3.19	0.92/1.83	0.71/1.38	0.25/0.44	-	-

. On the SDD, discrete waypoint modeling (K = 1) greatly improves long-tail performance but significantly reduces overall precision, whereas our continuous approach (K = 5) generates smoother long-tail cluster centroids to address this limitation. We also discover the effect of the K value on the ADE/FDE metrics. In the context of cross-validation where the univ subset is set as a test set, K = 5 yields the highest model performance for $V^2$-Net-SCP-abn. Additionally, $R_{int}$ = 0.05 gets the best results on the SDD.

Discussion IV: Inference Time. We test the inference time of the SDD, which has the most complex scenarios, on an NVIDIA GeForce RTX 4090 GPU. Note that all hyperparameters are same as those in Table 2. From Table 9, it can be seen that the average inference time and fast inference time do not increase because our rare intention modeling does not introduce any extra layer.

4.4. Qualitative Analysis

Visualized predictions. Figure 4a illustrates backbone predictions with our abnormal social interaction plugin in three SDD scenes: little1, hyang6, and bookstore6. Although different backbone prediction networks could generate varied predictions, all predictions with the abnormal interaction module preserve the quality and diversity of the original backbone predictions. Figure 4b shows a visual comparison of the same scene with and without our abnormal social interaction plugin. In the first row of Figure 4b, ego-biker should take full advantage of abnormal social interaction $i_1$ on the left side of the scene because it contains information about walkable roads. The MSN-SCP predictions on the backbone in the scene have a large gap between the ground truth because $i_1$ is averaged with other interactions in its SocialCircle space, so it cannot be given more attention. Agents tend to mimic the motion style of other agents in the same scene. In Figure 4b row 2, there are neighbor tracks that need to be imitated to walk around trees. The red circle prediction of $V^2$-Net-SCP-abn succeeds in imitating the tendency of neighbor agents and fits the ground truth better than $V^2$-Net-SCP. In general, places with few neighbor agents can be regarded as ‘uninhabited’ areas. People do not always like to go to an uninhabited place alone, including our ego agent. In Figure 4b row 3, all neighbor agents are gathered on the road near the roof while the rest of the scene could appear as ’uninhabited’ areas. Trans-SCP with our plugins narrows the gap to the ground truth by paying more attention to the crowded area.

Stress Testing. We verify the capability of our abnormal interaction modeling by manually adding an abnormal manual neighbor to observe our model reflection. Figure 5 visualizes and compares several toy examples of real-world SDD scenes. In Figure 5a, despite adding a side-by-side walking pedestrian close to the ego pedestrian, the $V^2$-Net-SCP predictions do not show the avoidance tendency in relation to the added neighbor, while our $V^2$-Net-SCP-abn predictions do show one. In Figure 5b,c, a manual interaction with high velocity toward the ego pedestrian guides our predictions. $V^2$-Net-SCP-abn and Trans-SCP-abn appear to be highly effective in preventing possible collisions, while backbone predictions do not appear. In Figure 5d, a high-speed neighbor is added to influence the ego agent’s right turn motion. Our MSN-SCP-abn predictions exhibit a larger spacing d to the abnormal neighbor than the backbone predictions. The results confirm that our approach extracts abnormal interactions and adapts across networks.

Interpretability of Abnormal Interactions. Our abnormal interaction representation demonstrates interpretability through the semantically distinct clusters identified via GMM’s two-stage clustering (Figure 6): (1) static agents (purple cluster, normalized velocity < 0.2), (2) agents approaching ego (green cluster, relative angle ≈ $\pi$), and (3) high-velocity neighbors (sky-blue cluster, normalized velocity > 0.5). This method effectively isolates abnormal interactions with clear behavioral semantics.

As illustrated in Figure 7 and the pie chart below, we select three scenarios involving abnormal social interactions and analyze the contributions of different semantic components to the model’s predictions. A common trend observed across all three scenarios is that the predicted trajectories of surrounding agents consistently avoid the abnormal interactions (highlighted in blue). However, the underlying semantic reasons for this avoidance vary by scenario. In Scenario 1, the “high-velocity” component is the dominant factor, accounting for 71% of the overall contribution. This suggests that the model relies primarily on the speed of the interacting agents to predict avoidance. In Scenario 2, where the target pedestrian’s orientation is nearly perpendicular (approx. $\pi /2$) to the direction of the abnormal interaction, we see a shift in the contributing factors. Compared to a baseline of normal interactions, the contribution of the ’opposite direction’ component increases by 34%. Simultaneously, the ’high-velocity’ component still maintains a high absolute contribution of 45%. This combination implies that, in such an unusual configuration, the model integrates both the high speed and the opposing direction of the agents to forecast their behavior. Scenario 3 presents a different situation: Here, the speed of the abnormal interaction is notably lower than in the first two scenarios. Consequently, the model assigns greater importance to stationary agents, with the contribution of the “static agent” component rising by 34% relative to the baseline. This indicates that, when a dynamic interaction is slow, the presence of static elements in the scene becomes a more critical cue for prediction. Finally, it is worth noting that a substantial portion of the model’s decision-making remains unaccounted for by these interpretable components. In scenario 3, over 57% of the contributory factors are classified as ’other abnormal components’ with no clearly visible semantic meaning. This points to a limitation in the current semantic framework and highlights an area for future investigation.

Generation of Intention GMM Components. The 10% of the intentions with the worst fit in the first stage are retained as long tails and they are subjected to a second GMM fit to obtain the center of the long-tail cluster. These long-tail intention clusters are combined with the clusters obtained from the first stage of clustering into the final intention GMM model to better reflect the distribution of long-tail intention. As shown in Figure 8, the above clustering method (right) gives a large weight to the sparsely distributed long-tail intention endpoints (i.e., the GMM clustering center in the right figure is more dispersed).

4.5. Robustness Evaluation and Comparative Results

Figure 9 is the box plot of the ADE/FDE results of 15 parallel experiments on a univ dataset. The ADEs/FDEs of our method get Q1, Q2, Q3 better than the baseline corresponding to the lower bottom edge, the center line of the box body, and the upper edge. Moreover, all our results do not appear in *, indicating that there are no outliers. Figure 10 shows FDE metrics in a test dataset with training epochs. Our plugin not only reduces testing errors but also makes the training results more stable. For the hotel subset, the best FDE reduces by 0.01 from 0.14 to 0.13. For the univ subset, the FDE metrics of the red curve experiments do not increase with the number of training epochs, which proves a better generalization performance.

5. Outlook

We will explore more frequency-based methods to couple abnormal interaction and rare intention modeling in our future study. This direction holds promise for developing a more holistic understanding of outlier events in multi-agent systems.

6. Conclusions

We unravel the fundamental causes of hard case scenarios in pedestrian trajectory prediction as (1) abnormal social interactions and (2) rare intentions in challenging scenarios. We propose a method to obtain abnormal social interactions. Moreover, an improved PCL algorithm facilitates the learning of rare intentions under continuous pseudo-label settings. A frequency-sensitive goal-driven decoder fuses both factors. Compared to the SOTA, our method outperforms on both the full dataset and the long-tail subset, advancing trajectory prediction.