InfoSTGCAN: An Information-Maximizing Spatial-Temporal Graph Convolutional Attention Network for Heterogeneous Human Trajectory Prediction

Abstract

Predicting the future trajectories of multiple interacting pedestrians within a scene has increasingly gained importance in various fields, e.g., autonomous driving, human–robot interaction, and so on. The complexity of this problem is heightened due to the social dynamics among different pedestrians and their heterogeneous implicit preferences. In this paper, we present Information Maximizing Spatial-Temporal Graph Convolutional Attention Network (InfoSTGCAN), which takes into account both pedestrian interactions and heterogeneous behavior choice modeling. To effectively capture the complex interactions among pedestrians, we integrate spatial-temporal graph convolution and spatial-temporal graph attention. For grasping the heterogeneity in pedestrians’ behavior choices, our model goes a step further by learning to predict an individual-level latent code for each pedestrian. Each latent code represents a distinct pattern of movement choice. Finally, based on the observed historical trajectory and the learned latent code, the proposed method is trained to cover the ground-truth future trajectory of this pedestrian with a bi-variate Gaussian distribution. We evaluate the proposed method through a comprehensive list of experiments and demonstrate that our method outperforms all baseline methods on the commonly used metrics, Average Displacement Error and Final Displacement Error. Notably, visualizations of the generated trajectories reveal our method’s capacity to handle different scenarios.

1. Introduction

It is important to accurately predict pedestrian trajectories [1,2,3]. For example, in situations like crosswalks and crowded public areas, accurately predicting pedestrian trajectories can improve safety and prevent potential accidents [4,5,6,7,8].

In monitoring systems, predicting pedestrian trajectories is pivotal in facilitating the detection of anomalous behaviors [9,10,11]. Additionally, it can help optimize the planning of transportation systems with better insights into pedestrian flow and behavior modeling [12,13,14,15,16].

Forecasting the trajectory of a pedestrian still remains a significant challenge, primarily for two reasons: (1) the complexity of interactions among pedestrians in a given environment and (2) the heterogeneity in individual behavioral preferences. Regarding the first reason, there are multiple factors influencing a pedestrian’s trajectory, e.g., static obstacles like trees and roads and dynamic components including vehicles and other pedestrians. As reported by [17], up to 70% of pedestrians in a crowd move in groups, such as families or friends walking together. Such interactions are mainly driven by “social interactions” [18,19]. Regarding the second reason that this remains challenging, different individuals usually display varied behaviors under similar circumstances [20], which makes it complicated to establish a universal behavioral model that fully represents the entire population [21,22,23]. For instance, pedestrians of different ages may have distinct walking preferences [24]. Additionally, these walking preferences might further change significantly depending on whether individuals are walking alone or within a group [25]. The heterogeneity in behaviors across agents substantially influences their actions across various observational contexts [26].

Figure 1 illustrates scenarios corresponding to the mentioned reasons, where Figure 1a shows how different pedestrians might favor different trajectories, and Figure 1b highlights the interactions of surrounding pedestrians on an individual’s decisions. Nevertheless, in the majority of the prior research, emphasis has been placed on the second aspect, i.e., how to model pedestrian interactions. One of the most classical approaches is called the “social force model” [18,19], which leverages forces such as attractive and repulsive forces to elucidate pedestrian behavior mechanisms in general.

Driven by the recent successes of deep learning techniques, there is a growing interest among researchers in developing deep learning-based methods to model the social interactions between pedestrians, incorporating the social attributes inherent in pedestrian movements. Social LSTM [27] is a deep learning-based method, which predicts pedestrians’ trajectories using recurrent neural networks. Subsequent research is developed based on such methodology, e.g., Peek Into The Future (PITF) [28] and State-Refinement LSTM (SR-LSTM) [29]. Another interesting direction is to use the Generative Adversarial framework [23,30,31,32,33], e.g., Social GAN [34] and Sophie [35].

Despite the existing body of research, there remains limited exploration into modeling pedestrian social interactions and heterogeneous individual behaviors simultaneously. To address the aforementioned problems, we propose InfoSTGCAN in this paper. Within this framework, we represent pedestrians’ trajectories through a spatial-temporal graph, integrating the spatial-temporal convolution and spatial-temporal attention mechanisms. Meanwhile, the behavior pattern of each pedestrian is modeled as a latent distribution based on the observed trajectory. Intuitively speaking, pedestrians with distinct latent codes have different styles of trajectories. As such, our approach not only improves predictive accuracy but also provides insights into the varied behavioral patterns exhibited by individuals.

1.1. Literature Review

1.1.1. Pedestrian Trajectory Prediction

In this section, we review the existing research focusing on the task of pedestrian trajectory prediction, which is the most related field to this paper. Generally, most pedestrian trajectory prediction methods can be typically categorized into two primary types: physics-based models and deep learning-based models.

Physics-based models One classical approach to tackle the challenge of pedestrian trajectory prediction is to utilize physics-based models. The physics-based models are usually characterized by some basic rules or generic functions, taking into account both physical constraints and pedestrians’ social or psychological factors [2,36,37,38,39,40]. One well-known physics-based model is the “cellular automaton model” [41,42]. The cellular automaton model is a discrete model, which is based on the discrete motions of pedestrians traversing a grid of cells, and it is assumed that each cell is in a finite number of states.

The second type of physics-based models is called the “social force model” [18]. The social force model is a microscopic continuous model, which studies the motions of pedestrians by some social forces, such as the destination choice and the need to avoid collisions with other pedestrians. Basically, the original social force model assumes that most of scenarios encountered by pedestrians are standard, such that behavioral strategies acquired through experience can be utilized.

Later, ref. [19] introduced “Nomad”, a generalized version of the social force model, which incorporates behavioral rules, and continuous route choice. This activity-based approach allows pedestrians to adapt their movements based on different traffic conditions, e.g., distance to a destination. In [36], the authors provided a comprehensive review of crowd motion simulation models, explaining their characteristics, applicability, and the underlying crowd movement phenomena. Interested in pedestrian behaviors at signalized crosswalks, ref. [43] adapted the social force model and calibrated its parameters using maximum likelihood estimation.

Another type of physics-based model is represented by the category of “velocity-based models”, which have been widely used in the game industry and robots [44,45,46,47]. Technically speaking, velocity-based models rely on differential equations, and their associated speed functions depend on the relative positions of the neighboring pedestrians and obstacles. For example, the reciprocal velocity obstacle model [45] is able to navigate multiple agents in real time and generates safe and oscillation-free motions. In [48], the authors proposed the optimal reciprocal collision avoidance (ORCA) model, which can provide local collision-free motions for a large number of agents within a time interval.

Deep learning-based trajectory prediction Inspired by the success of deep learning models [22,49,50,51,52,53], numerous research studies have focused on utilizing deep learning models for the task of pedestrian trajectory prediction. Social long short-term memory (Social LSTM) [27] is one of them. The Social LSTM model employs a type of recurrent network to learn the sequential movement of each pedestrian. To predict the trajectory afterwards, the “social pooling” mechanism is utilized to aggregate the output of the RNNs. Specifically, the model pools the neighbor hidden states of a pedestrian within a distance threshold. Later, based on LSTMs and Generative Adversarial Networks [30], Social GAN [34] was proposed. Social GAN designed a novel pooling mechanism that calculates interactions according to the relative distances among pedestrians.

Subsequent works have built upon Social LSTM and Social GAN [28,29,35,54]. For example, State-Refinement LSTM (SR-LSTM) [29] proposed a new pooling mechanism, which leverages the intentions of neighboring pedestrians. This approach iteratively refines the states of all pedestrians using a mechanism known as “social-aware information selection”. Peek Into The Future (PITF) [28] incorporates visual features and proposes modules that take pedestrian–scene interactions into consideration. The Sophie framework [35] is an LSTM-based generative adversarial network. It utilizes convolutional neural networks (CNNs) to extract scene features, followed by a dual attention mechanism. Subsequently, Sophie combines the attention outputs with the scene features.

Given that graph structures are able to explicitly represent the interactions of pedestrians, there has been a growing research interest in graph-based approaches [55,56]. Graph attention networks (GATs) [55] leverage the architecture of Bicycle-GAN and capture the pedestrian interactions with the help of the graph attention mechanism [55]. Recursive Social Behavior Graphs [56] utilize graph convolution networks, combined with additional social information from expert sociologists.

To directly utilize spatial and temporal information together, ref. [57] proposed the Social Spatial-Temporal Graph Convolutional Neural Network (Social-STGCNN). Social-STGCNN models pedestrian trajectories as a spatial-temporal graph, where the spatial edges represent social interactions between pedestrians, weighted by their relative distances. However, the graph kernel function in Social-STGCNN is still based on some predefined rules, e.g., pedestrians with shorter distances have higher weights. By incorporating the spatial-temporal attention mechanism, we are able to move beyond such “predefined rules”. Our proposed model learns to assign varying levels of importances to different neighbor nodes based on their features, while also taking the predefined rules into consideration.

In summary, while most previous works focus on modeling pedestrian interactions, the future trajectory of a pedestrian is usually uncertain and different pedestrians may exhibit distinct preferences regarding their behaviors. Therefore, it is not only crucial to model pedestrian interactions, but also imperative to explicitly quantify the inherent heterogeneity present within pedestrian trajectories. To bridge this gap, we introduce the InfoSTGCAN framework in this study. The proposed framework encapsulates pedestrian trajectories within a spatial-temporal graph, leveraging both convolutional and attention mechanisms across spatial and temporal dimensions. Furthermore, we model the behavioral patterns of each pedestrian as a latent distribution derived from their trajectories. This approach intuitively assigns unique latent codes to pedestrians, corresponding with distinct trajectory styles. Figure 1 illustrates such concept and

Table 1. Existing research on pedestrian trajectory prediction.

	References	Method	Required Features	Probabilistic or Deterministic	Social Interactions Modeling	Heterogeneity Modeling
Physics-based	[18]	Social Force Model	Positions + Velocities + Destinations	Deterministic	Social force fields	Different characteristics to different agents
	[41]	Cellular Automaton Model	Pedestrians (Velocities, Density, …) + Cells + Rules	Probabilistic	Predefined rules	Multiple walking classes
Deep learning-based	[27]	Social LSTM (S-LSTM)	Positions	Deterministic	Social pooling	$-$
	[34]	Social GAN (S-GAN-Pooling)	Positions	Probabilistic	Max-Pooling	Variety loss
	[55]	Graph Attention Network (GAT)	Positions + Images	Probabilistic	Social Attention	$-$
	[35]	LSTM-based Generative Adversarial Network (SoPhie)	Positions + Images	Probabilistic	Social attention	$-$
	[57]	Social Spatio-Temporal Graph Convolutional Neural Network (Social-STGCNN)	Positions	Probabilistic	Social kernels	$-$
	[58]	Spatial Context Attentive Network (SCAN)	Positions	Probabilistic	Spatial attention mechanism	$-$
	This study	InfoSTGCAN	Positions	Probabilistic	Social kernels + social attention	Pedestrian-level latent codes

summarizes the differences between the proposed framework with the previous methods.

1.1.2. Graph Neural Networks

Graph neural networks (GNNs) are well-suited for processing non-Euclidean data [59,60,61]. There are several kinds of GNNs [62,63,64,65,66,67,68,69]: Recurrent graph neural networks, convolutional graph neural networks, graph attention networks and so on.

Convolutional graph neural networks Basically, there are two major types of convolutional graph neural networks: spectral-based approaches and spatial-based approaches. Spectral-based approaches develop convolution operations based on the graph Fourier transform, e.g., GCNs [60],and ChebNet [62]. Spatial-based approaches perform convolution directly on the edges, making them suitable for asymmetric adjacency matrices, e.g., GraphSage [63] and DGCNN [64]. To deal with the spatial-temporal data, ST-GCN [70] expands the spatial GCN into a spatial-temporal version for the task of skeleton-based action recognition, incorporating information from a localized spatial-temporal context.

Graph attention networks Attention has been widely used in a series of tasks, e.g., machine translation [71], entity resolution [72] and so on. Proposed by [73], graph attention networks bring attention mechanisms to graph neural networks, which calculate the relative weights between two connected nodes by the attention scores. Later, [74] proposed GeniePath, which controls the flow of information by some LSTM-style gating mechanisms.

1.2. Contributions

Specifically, the main contributions of this paper are highlighted as follows:

• We formulate the task of pedestrian trajectory prediction as a spatial-temporal graph and propose a novel trajectory prediction model, InfoSTGCAN. This model takes both pedestrian interactions and heterogeneous behavior choice modeling into consideration. Through a comprehensive list of experiments, we demonstrate the superiority of InfoSTGCAN in comparison to existing baseline methods.
• Our proposed method integrates spatial-temporal graph convolution and spatial-temporal graph attention. This fusion enables our method to more effectively model pedestrian interactions by evaluating pedestrian importance using a combination of prior knowledge and data-driven features.
• Based on the technique of variational mutual information maximization, our model generates an individual-level latent code for each pedestrian. These distinct latent codes facilitate the generation of trajectories with heterogeneous behavior choices.

The remainder of this paper is organized as follows: Section 2 presents the preliminaries and major notations and defines the problem of human trajectory prediction. In Section 3, we explicate the proposed method, focusing on the spatial-temporal graph network, the variational mutual information maximization, and the multi-objective loss function. The proposed model is then evaluated in Section 4, and the design is validated through a process that includes performance comparison, results visualization, and ablation studies. Finally, we draw conclusions and discuss future research directions in Section 5.

2. Problem Statement

Suppose there are N pedestrians within a scene. Given their past observed trajectories $t{r}_{obs}^{1:N}$ during a period of time $T_{obs}$, our objective is to predict their future trajectories $t{r}_{pred}^{1:N}$ over the forthcoming time period $T_{pred}$. In this study, we jointly predict the future trajectories of all pedestrians.

To clarify, we begin by defining the observed trajectory of a pedestrian n ($n\in \{1,\dots ,N\}$). Specifically, for pedestrian n, its observed trajectory $t{r}_{obs}^n$ can be formulated as follows:

$$\begin{array}{c}\hfill t{r}_{obs}^n=\left\{{\mathit{p}}_t^n=({\mathit{x}}_t^n,{\mathit{y}}_t^n)\mid t\in \{1,\dots ,T_{obs}\}\right\}.\end{array}$$

(1)

where $({\mathit{x}}_t^n,{\mathit{y}}_t^n)$ are a pair of random variables that indicate the location distribution of the $n^{th}$ pedestrian at time step t. Following a similar formulation in [27,57], the probability distribution of $({\mathit{x}}_t^n,{\mathit{y}}_t^n)$ is modeled by a bi-variate Gaussian distribution $\mathcal{N}({\mu }_t^n,{\Sigma }_t^n)$, where ${\mathsf{\mu }}_t^n$ denotes the mean of the distribution, and ${\Sigma }_t^n$ denotes the covariance matrix.

$$\begin{array}{c}\hfill {\mathsf{\mu }}_t^n=\left[\begin{array}{c}{\mathsf{\mu }}_{t,x}^n\\ {\mathsf{\mu }}_{t,y}^n\end{array}\right]{\Sigma }_t^n=\left[\begin{array}{cc}{\left({\sigma }_{t,x}^n\right)}^2& {\rho }_{\dot{x}\dot{y}}{\sigma }_{\dot{x}}{\sigma }_{\dot{y}}\\ {\rho }_{\dot{x}\dot{y}}{\sigma }_{\dot{x}}{\sigma }_{\dot{y}}& {\sigma }_{\dot{y}}^2\end{array}\right]\end{array}$$

(2)

Next, we define the future trajectory of pedestrian n. For pedestrian n, her or his future trajectory is represented as ${\widehat{tr}}_{pred}^n$, which is formulated as follows:

$$\begin{array}{c}\hfill {\widehat{tr}}_{pred}^n=\left\{{\widehat{\mathit{p}}}_t^n=({\widehat{\mathit{x}}}_t^n,{\widehat{\mathit{y}}}_t^n)\mid t\in \{1,\dots ,T_{pred}\}\right\}.\end{array}$$

(3)

where $({\widehat{\mathit{x}}}_t^n,{\widehat{\mathit{y}}}_t^n)$ are a pair of random variables that indicate the predicted location distribution of the $n^{th}$ pedestrian at time step t. ${\widehat{tr}}_{pred}^n$ follows the estimated distribution $\mathcal{N}({\widehat{\mathsf{\mu }}}_t^n,{\widehat{\Sigma }}_t^n)$, i.e., ${\widehat{tr}}_{pred}^n\sim \mathcal{N}({\widehat{\mathsf{\mu }}}_t^n,{\widehat{\Sigma }}_t^n)$. The corresponding ground truth trajectory is represented as $t{r}_{pred}^n$. Note that $T_{pred}$ can be different from $T_{obs}$, signifying that the observation length can differ from the prediction length.

Before delving into our proposed model, we first specify the major notations that will be utilized in this paper.

3. Methodology

As depicted in Figure 2, there are two primary components at the core of our proposed method: the spatial-temporal graph network (Section 3.1) and variational mutual information maximization (Section 3.2). To begin with, we construct the spatial-temporal graph representation from the pedestrian trajectories (Section 3.1.1). Following this, the spatial-temporal graph convolutional attention network (STGCAN) comes into play. STGCAN serves a significant role in comprehensively modeling pedestrian interactions across both spatial and temporal dimensions. Delving deeper into its composition, this network is composed of two parts: spatial-temporal graph convolution (Section 3.1.2) and spatial-temporal graph attention (Section 3.1.3).

Spatial-temporal graph convolution applies convolution filters or kernels to the graph. On the other hand, spatial-temporal graph attention implicitly computes the relative weights among different nodes in a data-dependent way. For clearer differences and understanding how these two components work together to model pedestrian interactions, readers are directed to the end of Section 3.1.3.

For the second primary component, based on information theory, the technique of variational mutual information maximization (Section 3.2) helps to optimize the latent distribution through the proposed information-theoretic loss. Specifically, the model learns to predict an individual-level latent code for each pedestrian that possesses high mutual information with the future predicted trajectory. As a result, it enables the model to effectively capture the complexity and inherent heterogeneity of the pedestrian trajectories.

To summarize, the proposed approach is not only able to present a holistic view of their interactions but also able to capture the heterogeneity in pedestrian movements. Details regarding the optimization of the proposed method, including the multi-objective loss function, can be found in Section 3.3.

3.1. Spatial-Temporal Graph Convolutional Attention Network

3.1.1. Spatial-Temporal Graph Representation of Pedestrian Trajectories

To begin with, we need to construct the spatial-temporal graph representation from pedestrian trajectories. Essentially, a spatial-temporal graph is an attributed graph where the node attributes evolve through time [59]. The key idea behind the spatial-temporal graph is its capacity to simultaneously account for both spatial and temporal dependencies.

To build a spatial-temporal graph, we commence by formulating a sequence of spatial graphs. For every step t, a spatial graph ${\mathcal{G}}_t$ is constructed to represent the locations of pedestrians within a given scene. Each ${\mathcal{G}}_t$ is composed of three parts: a set of vertices $V_t$, a set of edges $E_t$, and an adjacency matrix $A_t$, i.e., ${\mathcal{G}}_t=(V_t,E_t,A_t)$.

Elaborating further, the vertex set is given by $V_t=\left\{v_t^n\mid \forall n\in \{1,\dots ,N\}\right\}$, where each vertex $v_t^n$ represents a pedestrian. The edge set $E_t$ is composed of edges between two vertices, i.e., $E_t=\left\{e_t^{mn}\mid \forall m,n\in \{1,\dots ,N\}\right\}$. Specifically, the edge $e_t^{mn}$ represents whether the vertex $v_t^m$ and the vertex $v_t^n$ are connected or not. If $v_t^m$ and $v_t^n$ are connected, $e_t^{mn}=1$; if $v_t^m$ and $v_t^n$ are not connected, $e_t^{mn}=0$. The adjacency matrix $A_t\in {\mathbb{R}}^{N\times N}$ is defined as follows:

$$\begin{array}{c}\hfill A_t=\left[\begin{array}{cccc}{a}_t^{11}& a_t^{12}& \dots & a_t^{1N}\\ a_t^{21}& a_t^{22}& \dots & a_t^{2N}\\ ⋮& ⋮& \ddots & ⋮\\ a_t^{N1}& a_t^{N2}& \dots & a_t^{NN}\end{array}\right]\end{array}$$

(4)

Each item $a_t^{mn}$ ($m,n\in \{1,\dots ,N\}$) models the influence between the vertex $v_t^m$ and the vertex $v_t^n$. As such, a spatial-temporal graph ${\mathcal{G}}_{1:T}$ is finally constructed, consisting of a series of spatial graphs ${\mathcal{G}}_{1:T}=\left\{{\mathcal{G}}_1,\dots ,{\mathcal{G}}_T\right\}$.

3.1.2. Spatial-Temporal Graph Convolution

Based on the spatial-temporal graph representation developed in Section 3.1.1, we introduce the spatial-temporal graph convolution (ST-GC) operation in this subsection. Before diving into the complete technical details of ST-GC, we start with a more general version of the convolution operation, which is on a two-dimensional grid or feature map.

In deep learning, neural networks that employ convolution operations are referred to as “convolutional neural networks (CNNs)” [49,53]. Typically, a CNN consists of multiple convolutional layers, and within each layer, multiple learnable filters or kernels are applied to the input feature map. These filters help to capture local patterns or features, enabling CNNs to learn hierarchical representations from the input data. Additionally, CNNs significantly benefit from the idea of “parameter sharing”, which reduces the number of parameters compared to fully connected layers.

For example, suppose the kernel size is equal to k, ${\mathrm{feature}}^{\left(l\right)}$ denotes the feature map at layer l, and ${\mathrm{feature}}^{(l+1)}$ denotes the feature map at layer $l+1$. Through the training process, convolution operations are able to learn to aggregate the information from the neighbors centering around each location in ${\mathrm{feature}}^{\left(l\right)}$. Therefore, the convolution operation on layer l can be summarized as:

$$\begin{array}{c}\hfill {\mathrm{feature}}^{(l+1)}=\sigma \left(\sum _{h=1}^k\sum _{w=1}^k\mathbf{p}({\mathrm{feature}}^{\left(l\right)},h,w)·{\mathbf{w}}^{\left(l\right)}(h,w)\right),\end{array}$$

(5)

where $\mathbf{p}(·)$ denotes the sampling function, ${\mathbf{w}}^{\left(l\right)}$ represents the weight function at layer l, and $\sigma$ denotes an activation function, e.g., ReLU or Sigmoid [53]. Note that the weight function ${\mathit{w}}^{\left(l\right)}$ is shared across different locations in ${\mathrm{feature}}^{\left(l\right)}$.

Transitioning to the context of graph neural networks (GNNs), as different instances might have inconsistent structures, the convolution operation on graphs should be node-order agnostic [60,63,75,76,77]. As a result, some important modifications for convolution operations shown in Equation (5) are required. In consequence, the spatial-temporal graph convolution operation is formulated as follows:

$$\begin{array}{c}\hfill v_t^{n,(l+1)}=\sigma \left(\frac{1}{\Omega }\sum _{v_{t^{\prime }}^{m,\left(l\right)}\in B\left(v_t^{n,\left(l\right)}\right)}\mathbf{p}\left(v_t^{n,\left(l\right)},v_{t^{\prime }}^{m,\left(l\right)}\right)·{\mathbf{w}}^{\left(l\right)}\left(v_t^{n,\left(l\right)},v_{t^{\prime }}^{m,\left(l\right)}\right)\right),\end{array}$$

(6)

where $\Omega$ denotes the normalizing term, and

$$\begin{array}{c}\hfill B\left(v_t^n\right)=\left\{v_{t^{\prime }}^m\mid \mathrm{dist}(v_t^n,v_t^m)\le {\mathrm{Dist}}_{\mathrm{spatial}},|t^{\prime }-t|\le {\mathrm{Dist}}_{\mathrm{temporal}}\right\}\end{array}$$

(7)

which denotes the neighbor set of the vertex $v_t^n$. Specifically, $\mathrm{dist}(v_t^n,v_t^m)$ denotes the minimum path connecting $v_t^n$ and $v_t^m$, ${\mathrm{Dist}}_{\mathrm{spatial}}$ determines the spatial range of nodes to be contained in the neighbor set, and ${\mathrm{Dist}}_{\mathrm{temporal}}$ determines the temporal range of nodes to be contained in the neighbor set.

Based on the discussions above, the convolution operation on graphs can be defined by adapting the aforementioned formulation to scenarios where the input features map resides on a spatial-temporal graph. Figure 3 illustrates this concept, where nodes of different colors represent different pedestrians. In order to model how a certain pedestrian is walking in the future, information is aggregated from his or her past trajectories and the nearby pedestrians. As the feature map’s level increases, predictions utilize a greater number of pedestrians and longer temporal information.

3.1.3. Spatial-Temporal Graph Attention

We begin by introducing a generalized version of the attention mechanism and providing insights into its underlying principles. Following that, we elucidate our approach to investigating attention within the context of a spatial-temporal graph.

Generally, in urban environments, pedestrians tend to know which other pedestrians require their attention to prevent possible collisions. This intuitive awareness allows us humans to determine which pedestrians might pose a higher risk of collision and adjust our own paths accordingly to maintain a smooth and safe walking experience. Therefore, different pedestrians have different relative importance. Inspired by recent significant success [55,71,72,73,78], we leverage this principle by applying the “attention” mechanism to the constructed spatial-temporal graph.

Typically, an attention mechanism consists of three major components: a query, a key, and a value [72]. Intuitively speaking, the query is a representation of the current item or context that the model is trying to process. It serves as a reference for determining essential elements within the input data. The key represents an item in the input sequence, which the model compares with the query to determine their similarity or relevance. The value represents the significant information associated with each element in the input data.

The attention mechanism computes a score for each key–query pair, typically using a dot product, scaled exponential, or some other similarity function [71,72,73,79,80]. These scores are then passed through a softmax function and are converted into a probability distribution, which represents “the attention weights”. Intuitively speaking, when a weight between a query and a key is higher, the corresponding key–value pair is more important; thus, the attention mechanism pays more “attention” to this pair.

Finally, the attention mechanism computes a weighted sum of the values using the obtained attention weights, effectively selecting and aggregating the relevant information from different elements. This aggregated context vector is then used in subsequent layers of the model to make predictions. The process can be summarized into the following equation:

$$\begin{array}{c}\hfill \mathrm{Att}\left(\mathrm{Qry},\mathrm{Key},\mathrm{Val}\right)=\mathrm{Softmax}\left(\frac{\mathrm{Qry}·{\mathrm{Key}}^T}{\sqrt{d_k}}\right)\mathrm{Val},\end{array}$$

(8)

where $d_k$ is the dimension of each query, and the term $1/\sqrt{d_k}$ can enhance the numerical stability of attention mechanisms.

In the context of GNNs, the previously mentioned three components, namely, a query, a key, and a value, are employed to learn meaningful representations of nodes, based on their local features and the structure of the underlying graph. As discussed in Section 3.1.1, each vertex represents a pedestrian, and $v_t^n$ represents pedestrian n at step t. We represent his or her corresponding query vector as ${\mathrm{qry}}_t^n=f_{\mathrm{Qry}}\left(v_t^n\right)$, the key vector as ${\mathrm{key}}_t^n=f_{\mathrm{Key}}\left(v_t^n\right)$, and the value vector as ${\mathrm{val}}_t^n=f_{\mathrm{Val}}\left(v_t^n\right)$.

As illustrated in Figure 4, there are two major kinds of attention mechanisms in our framework: spatial attention and temporal attention. Both of them can be viewed as a way of message passing on a connected graph [73,81].

Spatial attention focuses more on message exchanges among nodes within the same time step. Intuitively speaking, it helps to generate feasible trajectories by aggregating information from other pedestrians. Suppose the message passed from node $v_t^m$ to $v_t^n$ is ${\mathrm{msg}}_t^{m\to n}$, which is defined as:

$$\begin{array}{c}\hfill {\mathrm{msg}}_t^{m\to n}={\mathrm{qry}}_t^n·{\mathrm{key}}_t^m,\end{array}$$

(9)

and based on Equation (8), we may formulate spatial attention as:

$$\begin{array}{c}\hfill {\mathrm{Att}}_{\mathrm{spatial}}\left(v_t^n\right)=\mathrm{Softmax}\left(\frac{{\left[{\mathrm{msg}}_t^{m\to n}\right]}_{n,m=1:N}}{\sqrt{d_k}}\right){\left[{\mathrm{val}}_t^m\right]}_{m=1}^N\end{array}$$

(10)

On the other hand, temporal attention focuses more on the process of temporal message passing by aggregating information through the relevant time steps. Intuitively speaking, it helps to generate feasible trajectories by incorporating the temporal significant features of pedestrians.

Here, we show how the temporal message of $v_t^n$ passed from $t^{\prime }$ to t:

$$\begin{array}{c}\hfill {\mathrm{msg}}_{t^{\prime }\to t}^n={\mathrm{qry}}_t^n·{\mathrm{key}}_{t^{\prime }}^n,\end{array}$$

(11)

and based on Equation (8), we may formulate temporal attention as follows:

$$\begin{array}{c}\hfill {\mathrm{Att}}_{\mathrm{temporal}}\left(v_t^n\right)=\mathrm{Softmax}\left(\frac{{\left[{\mathrm{msg}}_{t^{\prime }\to t}^n\right]}_{t,t^{\prime }=1:T}}{\sqrt{d_k}}\right){\left[{\mathrm{val}}_{t^{\prime }}^n\right]}_{t^{\prime }=1}^T\end{array}$$

(12)

Difference Between ST-GC and ST-GAT Spatial-temporal graph convolution (ST-GC) and spatial-temporal graph attention (ST-GAT) are two commonly used techniques in GNNs. Both of them can learn meaningful representations of pedestrian trajectories through the spatial and temporal dimensions. Their major difference lies in how they aggregate information from neighboring nodes.

• In ST-GC, information from neighboring nodes is communicated by applying convolution filters or kernels on the graph, which typically involves a weighted sum of features across neighboring nodes. Usually, those weights can be identical (e.g., GraphSAGE [63]), predetermined, or learnable ([60,70]). Therefore, the weights are considered to be “explicitly” assigned to the neighborhoods of the focused node during the aggregation process [59].
• However, in ST-GAT, the weights between two connected nodes are considered to be “implicitly” computed. Specifically, those weights are learned based on the similarity of their feature representations, which takes into account the relative importance for different node pairs [59,73]. Typically, more important nodes tend to have higher similarity scores, resulting in them being assigned larger weights.

Social Interaction Modeling In this subsection, based on the discussed ST-GC and ST-GAT, we aim to clarify how they capture the modeling of pedestrian social interactions.

As introduced in Section 3.1.1, the adjacency matrix $A_t$ can be considered as a representation of the graph edge attributes. In the spatial-temporal graph convolution part, we incorporate our prior knowledge about the social relations among different pedestrians into a kernel function, e.g., pedestrians closer in distance tend to be more important. The kernel function maps node attributes at $v_t^n$ and $v_t^m$ to the attribute value $a_t^{mn}$, which is defined as follows:

$$\begin{array}{c}\hfill a_t^{mn}=\left\{\begin{array}{cc}1/{∥v_t^m-v_t^n∥}_2\hfill & ,{∥v_t^m-v_t^n∥}_2\ne 0\hfill \\ 0\hfill & ,\mathrm{Otherwise}.\hfill \end{array}\right.\end{array}$$

(13)

Equation (13) is consistent with the intuition that pedestrians are more likely to be influenced by each other if they are closer. Additionally, the kernel function is symmetric:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {∥v_t^m-v_t^n∥}_2={∥v_t^n-v_t^m∥}_2\ne 0& ,a_t^{mn}=a_t^{nm}\hfill \\ \hfill {∥v_t^m-v_t^n∥}_2={∥v_t^n-v_t^m∥}_2=0& ,a_t^{mn}=a_t^{nm}=0.\hfill \end{array}\end{array}$$

(14)

However, since the form of the kernel function is predetermined, and some pedestrian interactions are asymmetric, the interaction modeling in a purely spatial-temporal graph convolution-based model might be insufficient. Therefore, we integrate the spatial-temporal graph attention mechanism into our model. As discussed in Section 3.1.3, the relative importance of pedestrian m to pedestrian n is data-dependent, which is determined by the value of ${\mathrm{msg}}_t^{m\to n}={\mathrm{qry}}_t^n·{\mathrm{key}}_t^m$. Therefore, there is not any predetermined kernel in the the spatial-temporal graph attention mechanism. Generally, the spatial-temporal graph attention mechanism is able to capture the asymmetric importance:

$$\begin{array}{cc}\hfill {\mathrm{msg}}_t^{m\to n}& ={\mathrm{qry}}_t^n·{\mathrm{key}}_t^m\hfill \end{array}$$

(15)

$$\begin{array}{cc}\hfill {\mathrm{msg}}_t^{n\to m}& ={\mathrm{qry}}_t^m·{\mathrm{key}}_t^n\hfill \end{array}$$

(16)

$$\begin{array}{cc}\hfill {\mathrm{msg}}_t^{m\to n}& \ne {\mathrm{msg}}_t^{n\to m}.\hfill \end{array}$$

(17)

which stems from the fact that, in general, ${\mathrm{qry}}_t^m\ne {\mathrm{qry}}_t^n$ and ${\mathrm{key}}_t^m\ne {\mathrm{key}}_t^n$.

3.2. Variational Mutual Information Maximization

In real-world scenarios, when a pedestrian encounters other pedestrians, their reactions can vary from person to person. This variance can be influenced by factors like age, with different age groups having distinct walking behaviors [24]. Furthermore, an individual’s walking preference might have notable changes depending on whether walking alone or in a group [22,25]. Although few frameworks have been employed to produce such diverse trajectory styles (e.g., the variety loss [34]), there is still a need to understand how to effectively capture the intrinsic heterogeneity within the pedestrian trajectories.

In this subsection, to solve the mentioned issue, we introduce the technique of variational mutual information maximization. We begin by considering the principles of information theory [82,83,84,85]. Suppose X and Y are random variables. If we want to measure the “amount of information” learned about Y by providing the knowledge of X or vice versa, mutual information $I(X;Y)$ is utilized. The mutual information $I(X;Y)$ can be expressed as the difference between the self-entropy of Y and the conditional entropy of Y given X:

$$\begin{array}{c}\hfill I(X;Y)=H\left(X\right)-H(X\mid Y)=H\left(Y\right)-H(Y\mid X),\end{array}$$

(18)

where $H\left(X\right)$ denotes the self-entropy of X, and $H\left(Y\right)$ denotes the self-entropy of Y. $H(X\mid Y)$ denotes the conditional entropy of X given Y, and $H(Y\mid X)$ denotes the conditional entropy of Y given X.

The conditional mutual information is defined as below:

$$\begin{array}{cc}\hfill I(X;Y\mid Z)& =H(X\mid Z)-H(X\mid Y,Z)\hfill \end{array}$$

(19)

Intuitively speaking, $I(X;Y\mid Z)$ can be treated as how much uncertainty is reduced in X when Y is observed, given Z. If X and Y are independent, the knowledge of X does not provide any information about Y and vice versa. As a result, the mutual information between X and Y is zero. However, given Z, if the knowledge of X provides extensive information about Y, then $I(X;Y\mid Z)$ can be extremely high.

This interpretation helps to formulate the idea: given the past trajectory $t{r}_{obs}^n$, in order to learn meaningful representations for the future pedestrian trajectory, there should be high conditional mutual information between the latent code $c^n$ and the generated trajectory $G(t{r}_{obs}^n,c^n)$. In other words, $I(c^n;G(t{r}_{obs}^n,c^n)\mid t{r}_{obs}^n)$ should be high. As such, we propose an information-theoretic loss:

$$\begin{array}{c}\hfill I(c^n;G(t{r}_{obs}^n,c^n)\mid t{r}_{obs}^n)\end{array}$$

(20)

Based on Equations (19) and (20), we are able to derive the following equation:

$$\begin{array}{cc}\hfill I(c^n;G(t{r}_{obs}^n,c^n)\mid t{r}_{obs}^n)& =H(c^n\mid t{r}_{obs}^n)-H(c^n\mid G(t{r}_{obs}^n,c^n),t{r}_{obs}^n)\hfill \end{array}$$

(21)

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill & I(c^n;G)\hfill \\ \hfill =& -H\left(c^n\mid G\right)+H\left(c^n\right)\hfill \\ \hfill =& {\mathbb{E}}_{t{r}_{pred}^n\sim G}\left[{\mathbb{E}}_{c^{\prime }\sim P(c\mid t{r}_{obs}^n,t{r}_{pred}^n)}\left[logP\left(c^{\prime }\mid t{r}_{obs}^n,t{r}_{pred}^n\right)\right]\right]+H\left(c^n\right)\hfill \\ \hfill =& {\mathbb{E}}_{t{r}_{pred}^n\sim G}\left[D_{KL}P(·\mid t{r}_{obs}^n,t{r}_{pred}^n)\parallel Q_{\varphi }(·\mid t{r}_{obs}^n,t{r}_{pred}^n)+{\mathbb{E}}_{c^{\prime }\sim P(c\mid t{r}_{obs}^n,t{r}_{pred}^n)}\left[log{Q}_{\varphi }(c^{\prime }\mid t{r}_{obs}^n,t{r}_{pred}^n)\right]\right]+H\left(c^n\right)\hfill \\ \hfill \ge & {\mathbb{E}}_{t{r}_{pred}^n\sim G}\left[{\mathbb{E}}_{c^{\prime }\sim P(c\mid t{r}_{obs}^n,t{r}_{pred}^n)}\left[log{Q}_{\varphi }(c^{\prime }\mid t{r}_{obs}^n,t{r}_{pred}^n)\right]\right]+H\left(c^n\right)\hfill \end{array}\end{array}$$

(22)

However, in practice, directly maximizing the mutual information $I(c^n;G(t{r}_{obs}^n,c^n)\mid t{r}_{obs}^n)$ is extremely challenging, as it requires the truth unknown posterior $P(c^n\mid t{r}_{obs}^n,t{r}_{pred}^n)$. Therefore, we utilize a common technique in statistics and machine learning to address this problem, i.e., variational inference [85,86,87,88]. By defining an approximate posterior $Q_{\varphi }(c^n\mid t{r}_{obs}^n,t{r}_{pred}^n)$ over the original unknown posterior $P(c^n\mid t{r}_{obs}^n,t{r}_{pred}^n)$, we are able to construct a lower bound over the original quantity $-H(c^n\mid G(t{r}_{obs}^n,c^n),t{r}_{obs}^n)$:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill & -H\left(c^n\mid G,t{r}_{obs}^n\right)\hfill \\ \hfill & ={\mathbb{E}}_{t{r}_{pred}^n\sim G}\left[{\mathbb{E}}_{c^n\sim P(c^n\mid t{r}_{obs}^n,t{r}_{pred}^n)}\left[logP\left(c^n\mid t{r}_{obs}^n,t{r}_{pred}^n\right)\right]\right]\hfill \\ \hfill & ={\mathbb{E}}_{t{r}_{pred}^n\sim G}\left[\underset{\ge 0}{\underbrace{D_{KL}P(·\mid t{r}_{obs}^n,t{r}_{pred}^n)\parallel Q_{\varphi }(·\mid t{r}_{obs}^n,t{r}_{pred}^n)}}+{\mathbb{E}}_{c^n\sim P(c^n\mid t{r}_{obs}^n,t{r}_{pred}^n)}\left[log{Q}_{\varphi }(c^n\mid t{r}_{obs}^n,t{r}_{pred}^n)\right]\right]\hfill \\ \hfill & \ge {\mathbb{E}}_{t{r}_{pred}^n\sim G}\left[{\mathbb{E}}_{c^n\sim P(c^n\mid t{r}_{obs}^n,t{r}_{pred}^n)}\left[log{Q}_{\varphi }(c^n\mid t{r}_{obs}^n,t{r}_{pred}^n)\right]\right]\hfill \end{array}\end{array}$$

(23)

where G is the abbreviation for $G(t{r}_{obs}^n,c^n)$, $D_{KL}(·\parallel ·)$ stands for the Kullback–Leibler (KL) divergence, and the last step holds true because KL divergence is always always non-negative [82,84]. Therefore, we may construct a lower bound $L_I$ over the original objective Equation (21):

$$\begin{array}{c}\hfill I(c^n;G(t{r}_{obs}^n,c^n)\mid t{r}_{obs}^n)\ge \underset{L_I}{\underbrace{{\mathbb{E}}_{t{r}_{pred}^n\sim G}\left[{\mathbb{E}}_{c^n\sim P(c^n\mid t{r}_{obs}^n,t{r}_{pred}^n)}\left[log{Q}_{\varphi }(c^n\mid t{r}_{obs}^n,t{r}_{pred}^n)\right]\right]+H(c^n\mid t{r}_{obs}^n)}}\end{array}$$

(24)

As the approximate posterior $Q_{\varphi }(c^n\mid t{r}_{obs}^n,t{r}_{pred}^n)$ approaches the true posterior distribution $P(c^n\mid t{r}_{obs}^n,t{r}_{pred}^n)$, $D_{KL}P(·\mid t{r}_{obs}^n,t{r}_{pred}^n)\parallel Q_{\varphi }(·\mid t{r}_{obs}^n,t{r}_{pred}^n)$ approaches zero. Therefore, the lower bound $L_I$ approaches the mutual information $I(c^n;G(t{r}_{obs}^n,c^n)\mid t{r}_{obs}^n)$ and becomes tighter. It is worth mentioning that we also optimize the conditional entropy of the latent code $H(c^n\mid t{r}_{obs}^n)$, so that we the latent variable distribution and the predictor are learned simultaneously.

To summarize, the final objective of the variational mutual information maximization part can be written as:

$$\begin{array}{cc}\hfill L_{\mathrm{info}}& =-L_I\hfill \end{array}$$

(25)

$$\begin{array}{cc}\hfill & =-\left({\mathbb{E}}_{t{r}_{pred}^n}\left[{\mathbb{E}}_{c^n}\left[log{Q}_{\varphi }(c^n\mid t{r}_{obs}^n,t{r}_{pred}^n)\right]\right]+H(c^n\mid t{r}_{obs}^n)\right)\hfill \end{array}$$

(26)

$$\begin{array}{cc}\hfill & =-\left({\mathbb{E}}_{c^n\sim P_{\theta }(c^n\mid t{r}_{obs}^n),t{r}_{pred}^n\sim G}\left[log{Q}_{\varphi }(c^n\mid t{r}_{obs}^n,t{r}_{pred}^n)\right]+H(c^n\mid t{r}_{obs}^n)\right)\hfill \end{array}$$

(27)

where $P_{\theta }(c^n\mid t{r}_{obs}^n)$ is the conditional prior distribution for the latent code $c^n$.

The primary differences between Equation (20) and the mutual information-inspired objective in [85] are:

• In ref. [85], there is only one latent code for each training example. However, in this paper, there are multiple latent codes for each training example. Different pedestrians may have distinct preferences and walking styles. It is generally infeasible to assume all pedestrians follow the same preference or walking style. Therefore, for each pedestrian n, he or she has its own latent code $c^n$, and different pedestrians generally have different latent codes, allowing the proposed framework to effectively model the latent patterns in pedestrian trajectories.
• In this paper, the proposed information-theoretic loss is based on the conditional mutual information. However, in ref. [85], the loss is based on the mutual information.
• Different from the previous research taken in [85], where the prior latent code distribution is assumed to be fixed, we opt to optimize the prior distribution $P_{\theta }(c^n\mid t{r}_{obs}^n)$ as well.

$$\begin{array}{cc}\hfill L_I& ={\mathbb{E}}_{t{r}_{pred}^n}\left[{\mathbb{E}}_{c^n}\left[log{Q}_{\varphi }(c^n\mid t{r}_{obs}^n,t{r}_{pred}^n)\right]\right]+H(c^n\mid t{r}_{obs}^n)\hfill \end{array}$$

(28)

$$\begin{array}{cc}\hfill & ={\mathbb{E}}_{c^n\sim P(c^n\mid t{r}_{obs}^n),t{r}_{pred}^n\sim G}\left[log{Q}_{\varphi }(c^n\mid t{r}_{obs}^n,t{r}_{pred}^n)\right]+H(c^n\mid t{r}_{obs}^n)\hfill \end{array}$$

(29)

3.3. Multi-Objective Loss Function

To optimize the proposed method, we utilize the multi-objective loss defined below:

$$\begin{array}{c}\hfill L_{\mathrm{total}}={\lambda }_1{L}_{\mathrm{pred}}+{\lambda }_2{L}_{\mathrm{GAN}}+{\lambda }_3{L}_{\mathrm{info}}\end{array}$$

(30)

where $L_{\mathrm{pred}}$ denotes the prediction loss, $L_{\mathrm{GAN}}$ denotes the generative adversarial loss, and $L_{\mathrm{info}}$ denotes the information loss. ${\lambda }_1$, ${\lambda }_2$ and ${\lambda }_3$ control the relative importance of each loss.

$$\begin{array}{cc}\hfill L_{\mathrm{pred}}& =\sum _n{\mathbb{E}}_{c^n}\left[-log\left(P\left(t{r}_{pred}^n\mid G(t{r}_{obs}^n,c^n)\right)\right)\right].\hfill \end{array}$$

(31)

• $L_{\mathrm{pred}}$: The prediction loss relies on negative log-likelihood, which is defined as:

  $$\begin{array}{cc}\hfill L_{\mathrm{pred}}& =-\sum _{n}log\left(P\left(t{r}_{pred}^n\mid G(t{r}_{obs}^n,c^n)\right)\right).\hfill \end{array}$$

  (32)

  where $c^n\sim P_{\theta }(c^n\mid t{r}_{obs}^n)$. Intuitively speaking, when $L_{\mathrm{pred}}$ is decreasing, the log-likelihood of $t{r}_{pred}^n$ is increasing. The model $G(t{r}_{obs}^n,c^n)$ and the posterior distribution $P_{\theta }(c^n\mid t{r}_{obs}^n)$ together are encouraged to accurately predict the ground-truth future trajectory.
• $L_{\mathrm{GAN}}$: The generative adversarial loss relies on the generator G and the discriminator D, in which two models are jointly trained. The generator G captures the distribution for the future trajectory, and the discriminator distinguishes whether a sample comes from the training data or the generator G.

  $$\begin{array}{cc}\hfill L_{\mathrm{GAN}}& =\mathbb{E}\left[log\left(D(t{r}_{obs}^n,t{r}_{pred}^n)\right)\right]+\mathbb{E}\left[log\left(1-D(t{r}_{obs}^n,G(t{r}_{obs}^n,c^n))\right)\right]\hfill \end{array}$$

  (33)
• $L_{\mathrm{info}}$: The information-theoretic loss relies on the conditional prior distribution $P_{\theta }(c^n\mid t{r}_{obs}^n)$, the model $G(t{r}_{obs}^n,c^n)$, and the approximate posterior $Q_{\varphi }(c^n\mid t{r}_{obs}^n,t{r}_{pred}^n)$, which has been discussed in Section 3.2.

  $$\begin{array}{cc}\hfill L_{\mathrm{info}}& =-\underset{L_I}{\underbrace{\left({\mathbb{E}}_{c^n\sim P_{\theta }(c^n\mid t{r}_{obs}^n),t{r}_{pred}^n\sim G(t{r}_{obs}^n,c^n)}\left[log{Q}_{\varphi }(c^n\mid t{r}_{obs}^n,t{r}_{pred}^n)\right]+H(c^n\mid t{r}_{obs}^n)\right)}}\hfill \end{array}$$

  (34)

Notably, in order to enable effective backpropagation through the discrete latent code, we utilize the Gumbel–Softmax reparameterization trick [89,90]. Basically, the sampling process from a categorical distribution is usually non-differentiable. The Gumbel-Softmax reparameterization allows us to relax the discrete sampling process into a continuous and differentiable one, thereby enabling gradient-based optimization.

To summarize, the pipeline depicted in Figure 5 outlines the primary structure of our proposed framework, and Algorithm 1 describes the training procedure. Initially, a spatial-temporal graph is constructed from the pedestrian trajectories. Following this, spatial-temporal graph convolution (ST-GC) and spatial-temporal graph attention (ST-GAT) are deployed to extract meaningful information from pedestrian interactions in both spatial and temporal dimensions. In the meantime, latent codes will be inferred for all pedestrians, enabling the model to capture the intrinsic heterogeneity within their trajectories, and forecast their future paths. The optimization process integrates the prediction loss, the generative adversarial loss, and the information loss, which helps to achieve a more accurate and reliable result.

Algorithm 1: Training Procedure of InfoSTGCAN

1:  Input: Observed trajectories $t{r}_{obs}^{1:N}$ for pedestrians $1,2,\dots ,N$ within the scene 2:  for iteration $epoch=0,1,2,\dots$ do 3:     Utilize Pθ to generate latent codes c1, c2, …, cN for pedestrians 1, 2, …, N, respectively 4:     With c1, c2, …, cN, spatial-temporal graph convolutional attention network generates the distributions of the predicted trajectories ${\widehat{tr}}_{pred}^{1:N}$ 5:     Update the parameters of G, D, Pθ, and Qϕ based on Ltotal 6:  end for 7:  return The learned generator, discriminator, prior distribution, and posterior distribution

4. Experiments and Results

In this section, we first outline the experimental settings, including the datasets and evaluation metrics. Subsequently, we delve into the implementation details, present the experiment results, provide a comprehensive analysis of these results, and conclude with ablation studies.

4.1. Datasets and Evaluation Metrics

Datasets The proposed method is evaluated on multiple publicly accessible datasets: ETH [91] and UCY [92]. The ETH dataset contains approximately 750 different pedestrians and is divided into two scenarios: ETH and HOTEL. The UCY dataset contains approximately 786 unique pedestrians and is divided into three scenarios: ZARA1, ZARA2, and UNIV. These datasets consist of real-world pedestrian trajectories with complex human interactions. Specifically, lots of challenging pedestrian behaviors are covered in the datasets, such as pedestrians crossing each other, walking together, avoiding collision, and groups assembling and disbanding [91]. In accordance with a similar strategy utilized in previous studies [27,34], all trajectories are sampled every 0.4 s.

Evaluation Metrics In accordance with prior work [27,34,57,93], we choose to use the following evaluation metrics:

• Average Displacement Error (ADE): The average $L_2$ distance between the predicted trajectory and the ground truth trajectory across all time steps, which is defined as follows:

  $$\begin{array}{c}\hfill \mathrm{ADE}=\frac{1}{N{T}_{pred}}\sum _{n=1}^N\sum _{t=1}^{T_{pred}}\sqrt{{(x_t^n-{\widehat{x}}_t^n)}^2+{(y_t^n-{\widehat{y}}_t^n)}^2},\end{array}$$

  (35)

  where $(x_t^n,y_t^n)$ are the real locations, and $({\widehat{x}}_t^n,{\widehat{y}}_t^n)$ are the predicted locations.
• Final Displacement Error (FDE): The $L_2$ distance between the predicted final destination and the true final destination at the end of the prediction period $T_{pred}$, which is defined as follows:

  $$\begin{array}{c}\hfill \mathrm{FDE}=\frac{1}{N}\sum _{n=1}^N\sqrt{{(x_{T_{pred}}^n-{\widehat{x}}_{T_{pred}}^n)}^2+{(y_{T_{pred}}^n-{\widehat{y}}_{T_{pred}}^n)}^2}.\end{array}$$

  (36)

Intuitively speaking, different metrics serve as different purposes. ADE evaluates the average prediction error across the whole trajectory, whereas FDE focuses solely on the prediction error at the destination.

4.2. Implementation Details

In this section, we provide important details on appropriately implementing the proposed model. To facilitate the learning process [60,70], we normalize the adjacency matrix $A_t$ at each time step t as follows:

$$\begin{array}{c}\hfill A_t={\Lambda }_t^{-\frac{1}{2}}(A_t+I){\Lambda }_t^{-\frac{1}{2}},\end{array}$$

(37)

where I is an identity matrix, which serves to add self-connections to all nodes. ${\Lambda }_t$ is the diagonal node degree matrix of $(A_t+I)$. We use $\mathit{A}$ to denote the stack of all adjacency matrices $A_1+I,\dots ,A_T+I$, and $\Lambda$ to denote the stack of matrices ${\Lambda }_1,\dots ,{\Lambda }_T$. Suppose the vertices values at the layer l as $V^{\left(l\right)}$, which is a stack of vertices values across all steps $1,\dots ,T$. We can now employ the matrices defined to implement the ST-GC layers:

$$\begin{array}{c}\hfill f(V^{\left(l\right)},\mathit{A})=\sigma \left({\Lambda }^{-\frac{1}{2}}\mathit{A}{\Lambda }^{-\frac{1}{2}}{V}^{\left(l\right)}{\mathbf{W}}^{\left(l\right)}\right),\end{array}$$

(38)

where ${\mathbf{W}}^{\left(l\right)}$ represents the learnable parameters at the l-th layer. The above Equation (38) follows similar ideas in [57,70].

Model Architecture and Training Setup The proposed model consists of a series of ST-GC and ST-GAT layers, which helps to extract spatial-temporal node embeddings from the input data. Later, those node embeddings are concatenated with latent codes, and then, several convolutional layers are followed such that the output time dimension is manipulated to match the length of predicted horizon $T_{pred}$.

Unless noted otherwise, we choose to use PReLU [94] as the activation function through our model. During training, we used a batch size of 128 and the default optimizers were chosen to use Stochastic Gradient Descent (SGD). The initial learning rate was 0.01, and it was decreased based on exponential scheduling with a decay factor 0.97. To prevent overfitting to the training data, we randomly dropped out the features at a probability of 0.5.

4.3. Results Analysis

In this subsection, we begin by comparing our results with baseline models. Subsequently, we provide a comprehensive qualitative analysis of how our proposed method models pedestrian interactions and takes heterogeneous behavior choices into account. We illustrate cases where InfoSTGCAN successfully predicts collision-free trajectories for scenarios such as pedestrians walking in the same direction, approaching from opposing directions, or merging at angles. Moreover, our model is able to generate socially acceptable trajectories based on the predicted personalized latent codes.

4.3.1. Comparison with Baseline Models

Baselines We compare the proposed method with the following baselines:

• Linear: A linear regression model characterized by minimizing the least square error.
• Social LSTM (S-LSTM) [27]: An LSTM approach that incorporates the “social pooling” mechanism for hidden states.
• S-GAN-Pooling [34]: A GAN-based approach that utilizes global pooling for pedestrian interactions.
• SR-LSTM-2 [29]: An LSTM-based method that leverages a state refinement technique.
• GAT [55]: A graph attention network leveraging the sequence-to-sequence architecture.
• Sophie [35]: A GAN-based method that takes both scene and social factors into account through a dual attention mechanism.
• SCAN [58]: An LSTM-based encoder–decoder framework that incorporates a novel spatial attention mechanism to predict trajectories for all pedestrians.
• Social-STGCNN [57]: A spatial-temporal graph-based approach that employs a spatial-temporal graph convolutional network to handle complex social interactions.

The performance of the proposed method is evaluated against other benchmark models on ADE/FDE metrics, as presented in

Table 2. ADE/FDE metrics for several baselines and our method on all the datasets. All methods are evaluated with an observation length of 8 frames (3.2 s) and a prediction horizon of 12 frames (4.8 s). The model with lower values of metrics has a better performance.

Algorithm	Performance (ADE/FDE)
	ETH	HOTEL	UNIV	ZARA1	ZARA2	AVG
Linear	1.33/2.94	0.39/0.72	0.82/1.59	0.62/1.21	0.77/1.48	0.79/1.59
S-LSTM	1.09/2.35	0.79/1.76	0.67/1.40	0.47/1.00	0.56/1.17	0.72/1.54
S-GAN-Pooling	0.87/1.62	0.67/1.37	0.76/1.52	0.35/0.68	0.42/0.84	0.61/1.21
SR-LSTM-2	0.63/1.25	0.37/0.74	0.51/1.10	0.41/0.90	0.32/0.70	0.45/0.94
GAT	0.68/1.29	0.68/1.40	0.57/1.29	0.29/0.60	0.37/0.75	0.52/1.07
Sophie	0.70/1.43	0.76/1.67	0.54/1.24	0.30/0.63	0.38/0.78	0.54/1.15
SCAN	0.84/1.58	0.44/0.90	0.63/1.33	0.31/0.85	0.37/0.76	0.51/1.08
Social-STGCNN	0.64/1.11	0.49/0.85	0.44/0.79	0.34/0.53	0.30/0.48	0.44/0.75
InfoSTGCAN	0.61/0.82	0.48/0.71	0.40/0.64	0.33/0.51	0.30/0.44	0.42/0.62

. In general, our method outperforms all baseline methods on the two metrics. Our proposed model achieves an error of 0.62 on the average FDE metric, representing an approximate 20% improvement over the previous best performance (0.75). For the average ADE metric, the proposed model is better than the previous best performance by 5%. Interestingly, although our model does not need the vision signal containing scene context information, it can still outperform methods that utilize such information, such as SR-LSTM and Sophie.

4.3.2. Results Visualization

Figure 6 illustrates the validation losses over successive epochs for the considered datasets, where the x-axis corresponds to the training epoch index, and the y-axis represents the value of the validation loss. Each line in the figure represents the progression of the validation loss for a specific dataset. As observed from Figure 6, the proposed method exhibits convergence across all datasets.

In Figure 7, we present multiple qualitative examples and the trajectory distributions generated by our proposed method. In Figure 7a,b, pedestrians are moving straight in different directions. Our model successfully captures these trends, and accurately covers the ground truth future trajectory. In Figure 7c, the pedestrian makes an initial right turn before proceeding straight. As the observed trajectory in (c) is more curved compared to the trajectories in (a,b), we observe a broader predicted distribution. This broader prediction is intuitive, since a turn in the observed trajectory increases uncertainty, making the accurate prediction of the future trajectory more challenging.

Additionally, we present the trajectory distributions for the scenarios of multi-pedestrian interactions in Figure 8 and Figure 9. Figure 8 depicts a scenario where two pedestrians are walking together in parallel from the same direction. Typically, when pedestrians are walking together in parallel, their connections are usually tight and their momentum should be preserved in the future. Predictions generated by InfoSTGCAN support this observation, indicating that both pedestrians are likely to continue walking in parallel without colliding. Additionally, the predicted density demonstrates a close alignment with the ground truth trajectory.

On the other hand, Figure 9 depicts a scenario where two pedestrians are approaching each other from different directions. If both pedestrians maintain their original directions and momentum, a collision could happen. Accordingly, such a scenario requires the proposed model to generate plausible collision-free trajectories. The finding suggests that the proposed model achieves it effectively by capturing the social dynamics among different pedestrians. Furthermore, Figure 10 illustrates a scenario involving four pedestrians meeting from different directions.

In order to gain a comprehensive understanding of the quality of samples generated by the proposed method, we visualize the trajectories sampled from the predicted bi-variate Gaussian distributions. As shown in Figure 11, there are two scenarios. In the first scenario (Figure 11a), two pedestrians merge at an angle, and the result demonstrates our model’s capability to properly predict the trajectories for this situation. In the second scenario (Figure 11b), the pedestrian in orange attempts to avoid potential collisions with other two pedestrians who are walking together. The results indicate that the model is able to correctly predict the trajectories for all three pedestrians involved in this scenario. This analysis shows that the samples generated by the proposed method can encapsulate different social behaviors of pedestrians.

4.3.3. Interpretable Latent Representation

As illustrated in Figure 12, we compare the generated trajectories from different latent codes. When the spatial-temporal network needs to generate trajectories, pedestrians’ latent codes have to be provided. Therefore, different latent codes may generate different trajectories. Both code 0 and code 1 are sampled from the learned conditional prior distribution $P_{\theta }$, with code 0 exhibiting a higher likelihood than code 1. As depicted in Figure 12, the trajectory generated when applying code 0 typically follows a straight path, whereas that induced by code 1 tends to turn left.

In the context of pedestrian interaction, the trajectory generated by code 1 may result in collisions with the other pedestrian (in gray). The trajectory generated by code 0 is better aligned with the ground truth trajectory without potential collisions. As such, code 0 is more desirable and has a higher predictive likelihood than code 1. This evidence indicates that our framework is able to generate satisfied trajectories through learning a personalized latent code for each individual pedestrian.

4.4. Ablation Study

We conduct a series of comparative experiments to validate the efficacy of our proposed method by examining different values of $\lambda$ in the multi-objective loss function $L_{\mathrm{total}}$. As discussed in Section 3.3, the multi-objective loss function is defined as $L_{\mathrm{total}}={\lambda }_1{L}_{\mathrm{pred}}+{\lambda }_2{L}_{\mathrm{GAN}}+{\lambda }_3{L}_{\mathrm{info}}$. Through these experiments, two primary goals are addressed:

• Demonstrate the crucial role of the GAN loss part.
• Highlight the significance of maintaining a balanced weight between $L_{\mathrm{pred}}$ and $L_{\mathrm{info}}$.

According to the results presented in

Table 3. The ablation study on $\lambda$. Multiple different values are tested to show the model performance on the HOTEL dataset.

InfoSTGCAN	Performance (ADE/FDE)
${\lambda }_1=1.0,{\lambda }_2=1.0,{\lambda }_3=0.06$	0.48/0.71
${\lambda }_1=1.0,{\lambda }_2=0.0,{\lambda }_3=0.06$	1.11/1.90
${\lambda }_1=1.0,{\lambda }_2=1.0,{\lambda }_3=1.0$	0.57/0.92

, omitting the GAN loss by setting ${\lambda }_2=0$ leads to a decline in performance. More specifically, when we exclude the GAN loss, it effectively means removing the discriminator from the architecture. This underscores the importance of the GAN loss component within the multi-objective loss function.

Additionally, as shown in Table 3, when we overemphasize $L_{\mathrm{info}}$ or downplay $L_{\mathrm{pred}}$ by setting the weight ratio ${\lambda }_1:{\lambda }_3=1:1$, the model potentially underestimates the importance of accurately predicting the ground-truth future trajectory. Therefore, a balanced weight on $L_{\mathrm{info}}$ resulted in an appropriate focus on the final prediction accuracy, which is desirable.

These findings validate the importance of the GAN loss component and the significance of maintaining a balanced weight between the prediction loss $L_{\mathrm{pred}}$ and the information loss $L_{\mathrm{info}}$ for optimal performance.

5. Conclusions and Future Research

In this paper, we formulate the task of pedestrian trajectory prediction as a spatial-temporal graph and develop a novel pedestrian trajectory prediction model, InfoSTGCAN. The proposed model takes into account both pedestrian interactions and heterogeneous behavior choices. Specifically, to better model pedestrian interactions, our proposed model consists of two parts, spatial-temporal graph convolution and spatial-temporal graph attention, enabling the analysis of interactions through a combination of prior knowledge and data-driven methods. To address the heterogeneity within the pedestrian behavior choices, we utilize the variational mutual information maximization technique, which is primarily composed of a conditional prior distribution and an approximate posterior distribution.

The proposed method outperforms baseline models across several publicly accessible datasets. Visualization of the generated trajectories reveals our method’s capacity to handle various scenarios, including pedestrians going straight from different directions or making a right turn first and then going straight. We also conduct a qualitative analysis of the proposed method in different situations, such as collision avoidance, parallel walking, and pedestrians merging. In these situations, InfoSTGCAN tends to generate realistic collision-free trajectories. Additionally, we show that our framework is able to generate satisfactory trajectories through learning a personalized pedestrian-level latent code.

Nevertheless, we identify several promising future directions that are worth exploring further. The first aspect involves exploring more metrics related to probabilistic trajectory prediction beyond the standard ADE/FDE for training and evaluation, e.g., Mahalanobis distance [95]. Secondly, our methodology currently models pedestrian social interactions through ST-GC and ST-GAT; an exciting direction is to integrate more socially aware or physics-based methods [96]. Lastly, the third aspect refers to an integrative approach that combines heuristic optimization [97], causal inference [22,23,26] or clustering techniques [98].