Recognition and Prediction of Pedestrian Hazardous Crossing Intentions in Visual Field Obstruction Areas Based on IPVO-LSTM

Abstract

Pedestrians who suddenly cross the street from within the blind spot of a vehicle’s field of view can pose a significant threat to traffic safety. The dangerous pedestrian crossing intentions in view-obscured scenarios have not received as much attention as the prediction of pedestrian crossing intentions. In this paper, we present a method for recognizing and predicting the dangerous crossing intention of pedestrians in a view-obscured region based on the interference, pose, velocity observation–long short-term memory (IPVO-LSTM) algorithm from a road-based view. In the first step, the road-based camera captures the pedestrian’s image. Then, we construct a pedestrian interference state feature module, pedestrian three-dimensional pose feature module, pedestrian velocity feature module, and pedestrian blind observation state feature module and extract the corresponding features of the studied pedestrians. Finally, the pedestrian hazard crossing intention prediction module based on a feature-fused LSTM (ff-LSTM) and attention mechanism is used to fuse and process the above features in a cell state process to recognize and predict the pedestrian hazard crossing intention in the blind visual area. Experiments are compared with current common algorithms in terms of the input parameter selection, intention recognition algorithm, and intention prediction time range, and the experimental results validate our state-of-the-art method.

1. Introduction

Accidental crossing of pedestrians is one of the causes of pedestrian-related traffic accidents [1]. In an area with an open field of view, pedestrians, drivers, or autonomous vehicles can effectively sense each other’s movements and thus make effective decisions to avoid accidents. In contrast, in an area with an obstructed field of view, avoiding accidents caused by the sudden appearance of pedestrians in a vehicle’s driving area relies mainly on the defensive driving awareness of the driver or autonomous driving system, as well as the pedestrian’s specific observation pose, e.g., pedestrians lean out at the edge of the visual occlusion to observe the road traffic.

Pedestrian negligence in observing traffic conditions in their blind spots can lead to hazardous crossing intentions, which in turn can lead to intrusion into the vehicle’s driving area, posing a considerable challenge for drivers or automated driving systems to perform reasonable and practical avoidance actions in a brief period.

As shown in Figure 1, the views of both the pedestrian marked in gold and the vehicle marked in red are blocked by other vehicles parked at the edge of the road (marked in blue). In our study, we define such a crossing intention of a pedestrian who does not sufficiently observe the traffic conditions in the view-obscured area and ventures to cross the road as a hazardous crossing intention of the pedestrian in the view-obscured area. This issue poses a significant challenge for improving vehicles’ active safety and largely limits the comprehensive commercial application of self-driving vehicles in complex urban road environments [2]. Although there has been a large amount of research on predicting pedestrian crossing intentions or trajectories, there has been little research on pedestrian crossing intentions in view-obscured scenarios. Although this problem occurs in a more specific traffic situation, the solution is significant for improving active vehicle safety and fully promoting the commercial application of L5-level autonomous driving systems. In this paper, based on a depth camera, pose estimation technique, and spatial–temporal graph convolutional networks (ST-GCN), we first obtain pedestrian three-dimensional (3D) pose feature points and pedestrian action types, and we then construct a pedestrian interference state feature module, pedestrian 3D pose feature module, pedestrian speed feature module, and pedestrian blind observation state feature module. We extract the pedestrian features based on these modules. These features are concatenated in the cell state process of our constructed ff-LSTM, and the pedestrian hazard crossing intention in the blind area of the visual field is finally recognized and predicted by the pedestrian hazardous crossing intention prediction module.

With the popularization and development of infrastructure-to-vehicle (I2V) communication, the model proposed in this paper can compensate for the lack of perception ability of autonomous driving systems and conventional vehicles in view-obscured environments to improve the decision reliability and safety of autonomous driving systems and drivers. Our contributions are summarized as follows:

• For the issue of recognizing and predicting pedestrians’ hazardous crossing intentions in view-obscured areas, we present the interference, pose, velocity observation–long short-term memory (IPVO-LSTM) algorithm based on the road-based view.
• When selecting the input parameters for the pedestrian hazardous crossing intention recognition and prediction, we added interference features between pedestrians and blind observation degree features to the pedestrian pose features and historical trajectory features and constructed an ff-LSTM for fusing the above features.
• When performing interference judgment of pedestrians, we added the pedestrian action type and body pose features to the pedestrian spatial location features.
• In extracting the pedestrian blind observation degree features, we proposed a method to determine the pedestrian blind observation status based on the pedestrian head pose angle, torso forward angle, shoulder pose angle, and K-Means++ algorithm.

The remainder of this paper is organized as follows. Section 2 reviews the related work. Section 3 presents our proposed method to recognize pedestrians’ hazardous intentions and introduces the experimental procedure. Section 4 details the experiments and results. Section 5 summarizes the current work and possible work.

2. Related Work

Since this study utilized cameras as sensors, and pedestrian crossing intention recognition or prediction is usually performed in the same environment as pedestrian trajectory prediction [3], we collated related studies on pedestrian crossing intention prediction (recognition) based on vision sensors and trajectory prediction, as discussed below.

2.1. Observational Perspective for Pedestrian Crossing Intentions Prediction

By watching a large number of road traffic accident videos, we found that when pedestrians’ vision is obstructed, pedestrians do not always effectively observe the road conditions in the blind spot before crossing the street, often looking in the wrong direction. Failure to observe the road conditions can lead to pedestrians directly entering the vehicle driving area, causing accidents. As shown in

Table 1. Literature on pedestrian intention prediction.

Title	Method	Input Data	Observe Orientation	View Obscured
Fang et al. [4], 2019	CNN + RF	2D pose estimation	On Board	No
Xue et al. [5], 2018	CNN + LSTM	Trajectory, relative position, scene image features	Bird’s-eye view	No
Zhang et al. [6], 2021	SVM, RF, GBM, XGBT	2D Pose estimation, temperature, traffic light duration, and others	Bird’s-eye view	No
Quan et al. [7], 2021	Holistic LSTM	Trajectory, speed, and intent assessment	On board	No
Alahi et al. [8], 2016	CNN + LSTM	Trajectory	On board	No
Abughalieh et al. [9], 2020	CNN	Pedestrian pose 3D key points	On board	No
Wu et al. [10], 2021	HSTA	Trajectory	Bird’s-eye view	No
Kothari et al. [11], 2021	LSTM	Pedestrian trajectory and body pose	Bird’s-eye view	No
Razali et al. [12], 2021	CNN	Trajectory	On board	No
Chen et al. [13] 2021	GCN + LSTM	Pose estimation and object visual features	On board	No
Yau et al. [14], 2021	LSTM	Coordinates and directions	On board	No
Sun et al. [15], 2020	BiLSTM + CNN + GCN	Trajectory	Bird’s-eye view	No
Ma et al. [16], 2017	Multi-player game model	Age attributes, speed, and body direction	Bird’s-eye view	No
Saleh et al. [17], 2019	Spatio-temporal DenseNet	Sequential frame images	On board	No
Goldhammer et al. [18], 2019	KF, ML	Trajectory, motion state	Bird’s-eye view	No

CNN stands for convolutional neural network, HSTA stands for hierarchical spatio-temporal attention, SVM stands for support vector machine, GBM stands for gradient boosting machine, XGBT stands for extreme gradient boosting machine, GCN stands for graph convolutional network, KF stands for Kalman filter, ML stands for machine learning, LSTM stands for long short-term memory, Bi-LSTM stands for bi-directional long short-term memory, RF stands for random forest.

, the previous research on vision-based pedestrian intention or trajectory prediction was mainly from two observation perspectives: on-board and road-based vision sensors.

Most studies based on on-board vision sensors have focused on pedestrians directly in front of the sensor’s field of view. For example, Quan et al. [7] predicted the pedestrian trajectory ahead of the vehicle based on the image information captured by on-board vision sensors and an improved LSTM model. Fang et al. [4] implemented cyclist and pedestrian intention prediction in front of the field of view based on image information captured by on-board vision sensors and a convolutional neural network (CNN) + RF model. However, it is clear that the on-board sensor-based approach cannot overcome the critical problem of field of view occlusion and cannot meet the needs of the pedestrian intention or trajectory prediction in an area where the field of view is occluded.

In fact, self-vehicle-sensor field-of-view occlusion is also an important cause of difficulties for autonomous driving systems to make safe decisions and produce human-like driving behavior in dynamic and uncertain environments [19].

Compared to on-board vision sensors, a road-based sensor-based view can provide a complete field of view. Zhang et al. [6] captured videos of pedestrians crossing a road through a road-based camera and analyzed the intentions of pedestrians running red lights to cross the road at traffic light intersections based on information such as pedestrian two-dimensional (2D) skeletal points. Zhao et al. [20] developed a human-in-the-loop system based on lidar mounted on the roadside to bring pedestrians into the connected environment and predict pedestrian crossing intentions using an improved Naive Bayes method. However, regrettably, as shown in Table 1, although there have been many studies on pedestrian crossing intention or trajectory prediction using road-based approaches, the problem of pedestrian crossing intention or trajectory prediction in view-obscured scenarios has not received sufficient attention. This particular scenario is of great significance for improving active vehicle safety and advancing commercial applications for autonomous driving.

2.2. Common Research Methods for Pedestrian Crossing Intentions

As shown in Table 1, there are three main types of models: parametric models (e.g., Kalman filter (KF)) and various custom models, machine learning models (e.g., support vector machine (SVM)), and deep learning models (e.g., LSTM and CNN models). Deep-learning-based approaches have become more popular in recent years compared to other models [6]. Among the many studies based on deep learning, methods based on LSTM or its variant Bi-LSTM are more widely used. For example, Alexandre et al. [8] proposed the Social LSTM method to learn general human movements and predict their future trajectories. Xue et al. [5] employed LSTM to encode pedestrian trajectory information, grid map information, and image convolution information and then decoded the encoded results with an LSTM to obtain the predicted trajectories of pedestrians. Chen et al. [13] used CNN and graph convolutional network (GCN) modules to encode pedestrian information and used LSTM to decode it to finally obtain pedestrian crossing intention predictions. Given the good results already achieved by LSTM methods for pedestrian intention or trajectory prediction, the same LSTM-based approach was used in our study.

2.3. Common Input Parameters for Pedestrian Crossing Intention Prediction

When predicting pedestrian crossing intentions, traditional methods mainly use pedestrian trajectories to predict the future state of pedestrians [21]. However, subsequent studies have found that it is not sufficient to predict the future state of pedestrians by using their locations or trajectories alone [22]. Therefore, as shown in Table 1, the pedestrian’s pose also becomes an indispensable factor in pedestrian intention or trajectory prediction [4]. Moreover, in recent studies, pedestrian pose is increasingly used as an input factor for predicting pedestrian intentions or trajectories [23].

Furthermore, through the observation and analysis of a large number of pedestrian collision videos, we found that when the pedestrian’s field of view is obscured, the pedestrian’s observation level (including the observation angle and observation time) of the traffic conditions in the obscured area also affects a pedestrian’s intention to cross. This is because the wrong observation direction or an insufficient observation time can cause pedestrians to have hazardous crossing intentions in view-obscured conditions.

Additionally, interference behaviors between pedestrians (e.g., talking and chasing) can distract pedestrians crossing the street, which can affect their final intention. The distance between pedestrians has traditionally been used as a primary criterion for evaluating pedestrian interference. For example, Alexandre et al. [8] analyzed the inter-pedestrian interference problem using pedestrian spacing as an influencing factor. They performed pedestrian trajectory prediction on the ETH dataset by constructing a Social-LSTM network. However, using only the pedestrian-to-pedestrian distance as the interference judgment condition is not realistic [11] because the relationship between shorter pedestrian distances and the behavior interference between pedestrians is not a sufficient but a necessary relationship.

2.4. Pedestrian Crossing Intention Prediction Dataset

In

Table 2. Pedestrian intention prediction datasets.

Dataset	3D Info.	View
L-CAS [24]	Yes	On-board
PIE [25]	No	On-board
JAAD [26]	No	On-board
KITTI [27]	Yes	On-board
ATC [28]	No	Bird’s-eye
Ko-PER [29]	Yes	Bird’s-eye
Stanford drone [30]	No	Bird’s-eye
Cityscapes [31]	Yes	On-board
ActEV/ VIRAT [32]	No	Bird’s-eye
ApolloScape [33]	Yes	On-board
inD [34]	No	Bird’s-eye
Argoverse [35]	Yes	On-board
nuScenes [36]	Yes	On-board
Lyft Level 5 [37]	Yes	On-board
Waymo [38]	Yes	On-board
H3D [39]	Yes	On-board

Marked in bold: contains both 3D information and bird’s-eye view.

, we compiled the datasets used for pedestrian intent or trajectory prediction in the last decade containing 3D information for pedestrians based on different camera perspectives. We found that only the Ko-PER dataset contained both 3D and bird’s-eye view information from a roadside camera. Because the data were obtained for a limited duration, with only 38 pedestrians and a few scenes involving field-of-view occlusion, this dataset did not meet the needs of this study. Therefore, we constructed a dataset for this study by acquiring road videos.

3. Methodology

3.1. Overview and Preliminary Information

3.1.1. Model Overview

To predict pedestrian hazardous crossing intentions in areas with obscured fields of view, we constructed the IPVO-LSTM algorithm, as shown in Figure 2.

First, 2D pose estimates of pedestrians were obtained by collecting videos of pedestrians crossing the street through a depth camera mounted on the roadside, and 3D pose estimates and action classifications of pedestrians were obtained by combining an image depth estimation technique and a pedestrian pose estimation technique, respectively.

Second, the different 3D pose coordinates of pedestrians and the results of the action classification were then used separately in the pedestrian interference state feature module, pedestrian 3D pose feature module, pedestrian velocity feature module, and pedestrian observation state feature module to obtain different features of the pedestrians: the interference state features $Inte{r}_t^i$ between pedestrians closer to the camera and other pedestrians, the 3D pose features $Pos{e}_t^i$ of pedestrians closer to the camera, the velocity features $Velocit{y}_t^i$ of pedestrians closer to the camera, and the observation state features $Observ{e}_t^i$ for describing the traffic conditions in the blind zone of pedestrians closer to the camera.

Finally, the above features were fused in our constructed ff-LSTM, and the pedestrian hazard crossing intention prediction module was used to determine whether the traveler had a hazardous crossing intention.

3.1.2. Long Short-Term Memory (LSTM) Basics

In the basic LSTM network, the final output state can be obtained using the following equation based on a given sequence $\left\{F_1,F_2,\cdots ,F_t\right\}$:

$$h_t=\mathrm{LSTM}(h_{t-1},F_t;W)$$

(1)

where $W$ is the weight matrix, $b_t$ is the bias vector, and $h_t$ are the hidden states. The hidden states in the LSTM cell are mainly determined by the input gate $i_t$, forget gate $f_t$, output gate $o_t$, and cell state $c_t$, which can be expressed as follows:

$$i_t=\sigma (W_i\left[F_t,h_{t-1}\right]+b_i)$$

(2)

$$f_t=\sigma (W_f\left[F_t,h_{t-1}\right]+b_f)$$

(3)

$$c_t=f_t\odot c_{t-1}+i_t\odot u_t$$

(4)

$$u_t=\tanh (W_c\left[F_t,h_{t-1}\right]+b_c)$$

(5)

$$o_t=\sigma (W_o\left[F_t,h_{t-1}\right]+b_o)$$

(6)

$$h_t=o_t\tanh (c_t)$$

(7)

where $\odot$ denotes the element-wise product, $u_t$ denotes the cell state update value, $W_i$, $W_f$, $W_c$, and $W_o$ denote the different weight matrices, $\sigma (\cdots )$ denotes the sigmoid activation function, and $b_i$, $b_f$, $b_c$, and $b_o$ denote the different bias vectors [40].

3.2. Data Preprocessing

In the study of pedestrian crossing intention or trajectory recognition and prediction based on an on-board view, the 2D pose point coordinate values of pedestrians can often be obtained by calculating the body pose angles. However, some body pose angles, e.g., the forward tilt angle of the pedestrian torso, cannot be obtained if only 2D pose coordinate values of pedestrians are used in the pedestrian pose calculation based on a road-based view. To solve this problem, we obtain the depth information of pedestrian skeletal points by acquiring the image depth to determine the 3D coordinates of the pedestrian pose points. In the actual data acquisition process, a Kinect camera can obtain the scene depth of field through infrared and using its computing platform to collect the real-time 3D coordinates of skeletal points. However, the process of outdoor data acquisition will be disturbed by lighting and other conditions, resulting in poor recovery of the depth of field. Moreover, since binocular stereo vision is less affected by lighting but is more computationally intensive in the later data processing, we adopt a combination of both methods for the acquisition of data (for more detailed information, refer to Experimental Procedure (2)).

The 3D feature point calculation of the pedestrian pose based on a ZED binocular RGB camera is described as follows. First, we obtained the coordinates of any skeletal 2D feature point of the pedestrian in the left and right-side cameras based on OpenPose, and obtained the following:

$$A_l=\left(\begin{array}{ccc}{x}_0^l& \cdot \cdot \cdot & x_4^l\\ ⋮& \ddots & ⋮\\ x_{20}^l& \cdot \cdot \cdot & x_{24}^l\end{array}\right)$$

(8)

$$A_r=\left(\begin{array}{ccc}{x}_0^r& \cdot \cdot \cdot & x_4^r\\ ⋮& \ddots & ⋮\\ x_{20}^r& \cdot \cdot \cdot & x_{24}^r\end{array}\right)$$

(9)

where $x_i^l(i=0,1,\cdots ,24)$ and $x_j^r(j=0,1,\cdots ,24)$ denote the horizontal coordinate values of the 25 skeletal points in OpenPose in the left and right cameras, respectively (for more detailed information. Then, the parallax matrix D is obtained:

$$\begin{array}{ll}D& =A_l-A_r\\ & =\left(\begin{array}{ccc}{x}_0^l-x_0^r& \cdots & x_4^l-x_4^r\\ ⋮& \ddots & ⋮\\ x_{20}^l-x_{20}^r& \cdots & x_{24}^l-x_{24}^r\end{array}\right)\end{array}$$

(10)

With the camera focal length f and baseline b, the depth matrix Z of each skeletal feature point can be obtained:

$$Z=\frac{b\cdot f\cdot D^{\ast }}{\left|D\right|}=\left(\begin{array}{ccc}{z}_0& \cdots & z_4\\ ⋮& \ddots & ⋮\\ z_{20}& \cdots & z_{24}\end{array}\right)$$

(11)

where $z_k(k=0,1,\cdots ,24)$ denotes the depth values of each of the 25 skeletal points in OpenPose, i.e., the distance of each skeletal point from the camera. From this, we can obtain the 3D feature point coordinate matrix $Pos{e}_t^i(x_t^j,y_t^j,z_t^j)$ of any skeleton of pedestrian j at moment t.

3.3. Pedestrian Feature Modules

3.3.1. Pedestrian Interference State Feature Module

As is well known, interactions between pedestrians, e.g., talking and chasing, can affect pedestrians’ observations of road conditions and lead to incorrect crossing decisions. Although this problem has received particular attention in pedestrian intention recognition or prediction, in general, existing studies have mainly focused on the interactions between pedestrians by using the distance between them [15], which cannot meet the requirements of all realistic situations, especially when two pedestrians are moving side-by-side in a narrow view-obscured area. Even though they have a small distance between them, an interactive relationship between them cannot be guaranteed. In fact, not only does the distance between pedestrians have an impact on the evaluation of pedestrian interactions but also other factors, such as the azimuth angle between pedestrians, the body pose angle of pedestrians, and the type of pedestrian movement have significant impacts on the determination of pedestrian interactions. There are four aspects to be considered when modeling the pedestrians’ interactions, which are described as follows.

• Distance and azimuth between pedestrians

The azimuthal angle between pedestrians also affects the judgment of the interactions between pedestrians. For example, when the angle between A and B is 0°, and both of them walk at a slower speed, there is often no interaction between them. Let the mid-hip coordinates of the studied pedestrian be $\left(x_8,y_8,z_8\right)$ and the mid-hip coordinates of the other pedestrian be $\left(x_8^{\prime },y_8^{\prime },z_8^{\prime }\right)$. The distance $d_t^i$ and the angle ${\theta }_t^i$ between the other pedestrians and the studied pedestrian i at moment t are expressed as follows:

$$d_t^i=\sqrt{{\left(x_8-x_8^{\prime }\right)}^2+{\left(z_8-z_8^{\prime }\right)}^2}$$

(12)

$${\theta }_t^i=\arcsin \left(\sqrt{{\left(x_8-x_8^{\prime }\right)}^2}/d_t^i\right)$$

(13)

The orientation relationship between the two is shown in Figure 3.

2.

Pedestrian head observation angle

As is well known, humans tend to point their eyes at each other during conversation or other interactions to ensure that the interaction is carried out properly. This is also true for pedestrians. Thus, we take the pedestrian head steering angle α and shoulder steering angle β into account when performing pedestrian interaction evaluation. Figure 4 shows the definitions of these two angles and the pedestrian torso forward angle γ used later.

The head steering angle ${\alpha }_t^i$ and shoulder steering angle ${\beta }_t^i$ for any pedestrian i at moment t are calculated by

$${\alpha }_t^i=\arcsin \left(\sqrt{z_{\alpha \_temp}}/2\left|z_{15}^i-z_{16}^i\right|\right)$$

(14)

$${\beta }_t^i=\arcsin \left(\sqrt{{\left(z_2^i-z_5^i\right)}^2}/\sqrt{z_{\beta \_temp}}\right)$$

(15)

where $z_{\alpha \_temp}={\left(z_{15}^i-z_{16}^i\right)}^2+{\left(x_{15}^i-x_{16}^i\right)}^2$, and $z_{\beta \_temp}={\left(z_2^i-z_5^i\right)}^2+{\left(x_2^i-x_5^i\right)}^2$. $x_{15}^i$, $x_{16}^i$, $x_3^i$, $x_6^i$ are the horizontal coordinates of the projection of the right eye, left eye, right shoulder, and left shoulder skeletal points of pedestrian i in the XOZ plane, respectively; and $z_{15}^i,z_{16}^i,z_3^i,z_6^i$ are the corresponding vertical coordinates.

• Action classification

During particular interaction behaviors, such as chasing, the pedestrians’ heads are not facing the same direction, and the pedestrian spacing may increase over time. We classify the behavior of the resulting skeletal points based on an ST-GCN and improve the reliability of the pedestrian interaction judgment by combining the resulting pedestrian action type features.

As shown in Figure 5, first, parameters such as the 2D coordinates of the pedestrian skeletal points obtained from OpenPose are normalized, and then the temporal and spatial dimensions are transformed by the ST-GCN unit, alternately using a GCN and a temporal convolutional network (TCN). Finally, average pooling, a fully connected layer, and a softmax layer are used to achieve the action classification of the pedestrian actions. In addition, we use the attention mechanism (ATT) module used in the Yan model [25], considering the different levels of importance of different torso positions in judging the actions of pedestrians when crossing the street [41]. To facilitate subsequent LSTM calculations, we use one-hot encoding to label the behavior types of pedestrians crossing the street at time period t, where pedestrian i shows crossing behavior $ac{t}_t^i$ during time period t [42]. Of these, eight common behaviors were selected in our study: talking on phone, playing with a cell phone, greeting, walking, running, holding hands, eating/drinking, and smoking.

2.

Classification of pedestrian interaction features

This part finally uses a fusion layer $\phi (\cdot )$ by combining the above features $d_t^i,{\theta }_t^i,{\alpha }_t^i,{\beta }_t^i,ac{t}_t^i$, the pedestrian interaction feature $X_t$ can be obtained as follows:

$$\begin{array}{ll}{X}_t& =\phi (d_t^i,{\theta }_t^i,{\alpha }_t^i,{\beta }_t^i,ac{t}_t^i)\\ & =d_t^i\oplus {\theta }_t^i\oplus {\alpha }_t^i\oplus {\beta }_t^i\oplus ac{t}_t^i\end{array}$$

(16)

where the tandem state $h_t$ consists of the pedestrian distance and azimuth angle of pedestrian i, pedestrian head observation angle, pedestrian action type classification information, and time point t. The final interference state (0 or 1) is then obtained by taking the pedestrian interaction feature $X_t$ as the input and passing through an LSTM layer, fully connected layer, and softmax layer, as follows:

$${\stackrel{\wedge }{h}}_t=\mathrm{LSTM}({\stackrel{\wedge }{h}}_{{}_{t-1}}^i,X_t;W_1)$$

(17)

$$z_i=W_2{X}_t+b_0$$

(18)

$$\begin{array}{ll}{\mathrm{inter}}_t^i& =\mathrm{softmax}(z_i)\\ & =\exp (z_i)/{\displaystyle \sum _{k=1}^K\exp (z_k)}\end{array}$$

(19)

where $W_1$ and $W_2$ denote different weight matrices, and $b_0$ denotes the bias vector.

In addition, when there is only one pedestrian in the image at time t, although the pedestrian is in a non-interference state with other pedestrians, if the pedestrian exhibits a behavior such as playing with a cell phone with their head down, they may still have a dangerous intention to cross the street due to the distraction. In order to avoid the impact of this problem, we keep the behavioral feature $ac{t}_t^i$ and fuse it with the pedestrian interference state feature ${\mathrm{inter}}_t^i$ calculated by the pedestrian interference state feature module, as shown by the dashed line in Figure 2. Then, we obtain the pedestrian interference feature $Inte{r}_t^i$.

$$Inte{r}_t^i=\left[{\mathrm{inter}}_t^i,ac{t}_t^i\right]$$

(20)

3.3.2. Pedestrian Velocity Feature Module

Although the behavior in the pedestrian interference state feature module is able to determine whether the traveler is running or walking, there is also a difference in the degree of observation of the road conditions between slow-running (walking) and fast-running (walking) pedestrians, so we keep the pedestrian speed feature.

When setting up the pedestrian speed calculation module in Figure 2, we first select pedestrian i, who is closer to the center of the road, as the object of focus. Then, by using the Z-coordinate values of the midpoints of the left and right hip lines (i.e., point 8 in Figure 6a) at moments t and t + 1, we can obtain the velocity feature $Velocit{y}_t^i$ of pedestrian i as follows:

$$Velocit{y}_t^i=(Z_{t+1}-Z_t)/\Delta t$$

(21)

3.3.3. Pedestrian Observation State Feature Module

When the pedestrian view is obscured, the pedestrian’s observation level (including the observation direction and duration) of the traffic condition in the blind area directly affects the pedestrian’s final crossing intention decision. In order to observe the environment in the obstructed field of view, pedestrians not only need to turn their heads and shoulders but also sometimes need to add a forward tilt of the limb trunk to complete the observation. Thus, we also added the forward tilt angle γ of the limb trunk, as shown in Figure 4b.

Let the point $Midhip{(x_8,y_8,z_8)}_t$ be the coordinates of the midpoint of the line connecting the left hip and the right hip at the moment t (mid-hip point). Based on the side view of the human skeletal points in Figure 4b, the forward tilt angle γt of the trunk at the moment t is

$${\gamma }_t=\arcsin \left(\frac{\sqrt{{\left(z_1-z_8\right)}^2}}{\sqrt{{\left(z_1-z_8\right)}^2+{\left(x_1-x_8\right)}^2}}\right)$$

(22)

Let the pedestrian blind zone observation vector at time t be $O({\alpha }_t,{\beta }_t,{\gamma }_t)$, where αt, βt, and γt are the head steering angle, shoulder steering angle, and torso forward angle of the pedestrian at time t. After obtaining the pedestrian blind zone observation vector, in order to be able to judge whether the pedestrian is observing the road condition in the blind zone at moment t based on the pedestrian blind zone observation vector at moment t, we designed a pedestrian blind zone observation experiment (for more detailed information, refer to Experimental Procedure (1)).

First, by collecting the head steering angle, shoulder steering angle, and torso forward angle from different experimenters while observing the road in a blind area, we constructed a pedestrian blind area observation degree body pose dataset, the PO Dataset, $\left\{\left({\alpha }_1,{\beta }_1,{\gamma }_1\right),\cdot \cdot \cdot \right\}$.

Then, the K-means++ algorithm was used to obtain the pose clustering centroids $C_{unaware}({\alpha }_{uw},{\beta }_{uw},{\gamma }_{uw})$ when the pedestrian was not observing the road conditions and the pose clustering centroids $C_{aware\_l}({\alpha }_{wl},{\beta }_{wl},{\gamma }_{wl})$ and $C_{aware\_r}({\alpha }_{wr},{\beta }_{wr},{\gamma }_{wr})$ when the pedestrian was observing the left and right sides within the blind area of the visual field, respectively [43].

Finally, for a new image, the observation state was determined based on the cluster that it was closest to. This was determined by the minimum Euclidean distance, $\min \left({\rho }_{un},{\rho }_{wr},{\rho }_{wl}\right)$, where ${\rho }_{un}=\sqrt{{({\alpha }_t-{\alpha }_{uw})}^2+{({\beta }_t-{\beta }_{uw})}^2+{({\gamma }_t-{\gamma }_{uw})}^2}$. ${\rho }_{wr}$ and ${\rho }_{wl}$ were calculated similarly.

The observation result was set as $Observ{e}_t^i$, which took values of 0, 1, or 2, corresponding to correctly observed, unobserved (forward observation), and incorrectly observed (according to Chinese traffic regulations, vehicles drive on the right side of the road and the direction of incoming traffic is to the left of the pedestrian; thus, a pedestrian who only observes to the right is defined as incorrectly observed), respectively.

3.3.4. Pedestrian 3D Pose Feature Module

The Pedestrian 3D pose feature obtained after 3D pose estimation had dimensions of $25\times 3$, refer to Figure 6a, which would reduce the effectiveness of the LSTM network if it were used as direct input.

To address this problem, Shahroudy et al. [44] grouped the body joints into five groups: torso, two hands, and two legs, and used each of them as input features. Inspired by this, combined with the Openpose-Body25 model we used, we grouped the body joints into six part groups: torso ($F_p^3$), two arms ($F_p^2$ and $F_p^4$), two legs ($F_p^5$ and $F_p^6$), and head ($F_p^1$), as shown in Figure 6, where points 19 to 24 were not included because these points were the toes and heels, which were not used in our study.

In addition, considering that there may be more than one pedestrian in the image at moment t, we choose pedestrian A, whose position is close to the center of the road (e.g., the pedestrian marked in gold in Figure 1). Pedestrian B, who is farther from the center of the road (e.g., the pedestrian in the green box in Figure 1), is able to rely on the pose and response of pedestrian A, who is close to the center of the road when making a crossing intention decision. Therefore, we construct the pedestrian 3D pose selection method shown in Figure 2 by comparing the z-coordinates of different pedestrian mid-hip points at the same moment, approximating the distance of the pedestrian from the camera, and selecting the pose value $Pos{e}_t^i(x_t^i,y_t^i,z_t^i)$ of pedestrian i with the smaller mid-hip z-value as the input to the pedestrian hazardous crossing intention prediction module. To facilitate the batch processing of data, the 2D coordinates captured in the left camera, combined with the depth values of skeletal points, are used to construct 3D feature point coordinates of pedestrian skeletons.

3.4. Pedestrian Dangerous Crossing Intention Prediction Module

3.4.1. Feature-Fusion LSTM

As shown in Figure 7a, a naive way to combine the $Pos{e}_t^i$, $Inte{r}_t^i$, $Observ{e}_t^i$, and $Velocit{y}_t^i$ features is to concatenate them in the feature level and feed them to the corresponding LSTM unit as the network’s input data. However, the dimension of the $Inte{r}_t^i$, $Observ{e}_t^i$, and $Velocit{y}_t^i$ features are very low intrinsically, while the $Pos{e}_t^i$ features often have relatively higher dimensions. As a result of this inconsistency, concatenating these two types of features in the input stage of the network would degrade the overall performance. In [45], different body parts were fed into the Part-Aware LSTM separately and then assembled inside the LSTM unit. Inspired by this work, we propose to fuse the ($Inte{r}_t^i$, $Observ{e}_t^i$, $Velocit{y}_t^i$) and ($Pos{e}_t^i$) features inside the feature-fusion LSTM unit, rather than simply concatenating them at the input level, as shown in Figure 7b.

Since the dimensionalities of the $Inte{r}_t^i$, $Observ{e}_t^i$, and $Velocit{y}_t^i$ features were low, they could be spliced in the network input stage to obtain the ($Inte{r}_t^i$, $Observ{e}_t^i$, $Velocit{y}_t^i$) vectors, which were combined with the six vectors $F_p^1$, $F_p^2$, $F_p^3$, $F_p^4$, $F_p^5$ and $F_p^6$ of $Pos{e}_t^i$, respectively, by Equations (2), (3), (5), (6), and (4) to obtain the corresponding input gate values ($i_t$,$i_t^1$, …, $i_t^6$), forget gate values ($f_t$,$f_t^1$, …, $f_t^6$), output gates ($o_t$,$o_t^1$, …, $o_t^6$), cell state values ($c_t$, $c_t^1$, …, $c_t^6$), and cell state update values ($c_t^{″}$).

Then, the pedestrian crossing intention hidden state $H_t$ was obtained:

$$H_t=\tan \mathrm{h}(c_t^{″})\odot o_t^{″}$$

(23)

$$\begin{array}{ll}{c}_t^{″}=& c_t+c_t^1+c_t^2+c_t^3\\ & +c_t^4+c_t^5+c_t^6\end{array}$$

(24)

$$o_t^{″}=\sigma \left(W\left[\begin{array}{l}Pos{e}_t^i\\ Inte{r}_t^i\\ Observ{e}_t^i\\ Velocit{y}_t^i\\ h_{t-1}\end{array}\right]+B\right)$$

(25)

where $W$ denotes the weight matrices, and $B$ denotes the bias vector.

The intermediate hidden state vector $\left[{\stackrel{\wedge }{H}}_t^1,{\stackrel{\wedge }{H}}_t^2,\cdot \cdot \cdot ,{\stackrel{\wedge }{H}}_t^l\right]$ was obtained after the feature-fusion LSTM layer, where l is the number of hidden units (set to 100 in our network).

3.4.2. Pedestrian Dangerous Crossing Intention Prediction

To prevent overfitting, we perform dropout processing before performing the full connection process, setting $r^{(l)}$ to follow a Bernoulli distribution, i.e., $r^{(l)}\sim \mathrm{Bernoulli}(p)$:

$${\stackrel{\sim }{H}}_t^{(l)}=r^{(l)}\cdot {\stackrel{\wedge }{H}}_t$$

(26)

where $\mathrm{Bernoulli}(p)$ is the Bernoulli function, which is used to generate a vector of random numbers between 0 and 1 with probability $p$. In our study, $p=0.5$. The result of this step is passed through the fully connected attention layer:

$$x_n=W_4{\stackrel{\sim }{H}}_t^{(l)}+b_1$$

(27)

$$\begin{array}{ll}{C}_t& =\mathrm{Attention}(x_n,{\epsilon }_k)\\ & ={\displaystyle \sum _{k=1}^K{\epsilon }_k\cdot x_n}\end{array}$$

(28)

where ${\epsilon }_k=\exp (S(x_n,q))/{\displaystyle \sum _{j=1}^J\exp (S(x_j,q))}$ and $S(x_j,q)$ are the attention scoring function, and

$$S(x_j,q)=\tanh (W{x}_j+q)$$

(29)

The attention distribution result $C_t$ is fully connected, and a softmax function is used to finally obtain the pedestrian’s crossing intention prediction $Intentio{n}_t$. Our final intention result is 0 or 1, indicating crossing and not crossing, respectively:

$$Intentio{n}_t=\mathrm{softmax}(W_5{C}_t+b_2)$$

(30)

where $W_3$, $W_4$, and $W_5$ denote the different weight matrices, $b_0$ indicates the bias vector, and q is a learnable parameter.

3.5. Experimental Procedure

In our study, we designed two corresponding data collection methods to construct the PO dataset to study the degree of pedestrian blindness observations and a dataset to study pedestrian hazard crossing intention recognition, prediction, and pedestrian interference status. The analysis and validation methods for the two were slightly different.

3.5.1. PO Dataset Acquisition

In order to obtain the pose clustering centroids $C_{unaware}$ when pedestrians were not observing the road condition and the pose clustering centroids $C_{aware\_l}$ and $C_{aware\_r}$ within the blind areas of pedestrian’s left and right fields of view, respectively, we first conducted an experiment to obtain the PO Dataset with the pedestrian blind observation features shown in Figure 8.

Since the head steering angle, shoulder steering angle, and torso forward angle of the pedestrians were not related to the height of the camera and the distance between them, we chose an area where the field of view was blocked by buildings, as shown in red in Figure 8a, to conduct the experiment. First, the experimenters (15 people) were asked to stand in the field of view occlusion area. Furthermore, different pedestrians may have different body poses due to their different observation habits when observing traffic conditions in the blind zone. Therefore, in order to restore the observation habits of different types of pedestrians and obtain the corresponding body pose with a small number of experimenters, we randomly placed information panels showing the numbers 1–10 on the left, middle, and right parts of the front windshield of the yellow test vehicle, as shown in Figure 8b. We asked each experimenter to read the corresponding position in the field of view blocked by the wall and to maintain a pose such that they could read the panel for 3 s for later data extraction. Meanwhile, the corresponding pose information was captured and recorded by the depth camera, as shown in Figure 8c.

3.5.2. Other Data Acquisition

The camera shown in Figure 9a used infrared to acquire depth information without calculations during the data processing. It exported the data to the supporting software. Therefore, the software parameters are not noted separately here.

The internal parameters of the depth camera used in Figure 9b are shown in

Table 3. Parameters of the binocular camera.

Name	Value
Left camera/fx	1059.0900 pixels
Right camera/fy	1060.2300 pixels
Baseline between two sensors/b	119.7410 mm

, where fx and fy are the focal lengths of the left and right-side cameras, respectively, and b is the baseline length between the left and right-side cameras.

In addition, during the actual data collection process, we needed to alert pedestrians to avoid accidents when vehicles were in their blind spots and they had hazardous crossing intentions. It was assumed that the experimental equipment would distract the pedestrians, resulting in changes in the crossing pose. Thus, we watched videos of traffic accidents caused by obstructed vision, and the experimenters imitated the behaviors of pedestrians in the videos when the road conditions were safe and the corresponding data were collected.

4. Experiments

4.1. Evaluation Indicators

Considering that the pedestrian interference and blindness observation degree involved in the study were different from those of the traditional methods, the experimental investigation was divided into three experiments: a pedestrian interference experiment, pedestrian blindness observation degree experiment, and pedestrian crossing intention experiment. The evaluation criteria used in the pedestrian interference experiment and the pedestrian crossing intention experiment were Precision, Recall, F1-score, and Accuracy, which were calculated as follows:

$$Precision=TP/\left(TP+FP\right)$$

(31)

$$Recall=TP/\left(TP+FN\right)$$

(32)

$$F1-score=\frac{2\ast Precision\ast Recall}{Precision+Recall}$$

(33)

$$Accuracy=\frac{TP+TN}{TP+FP+FN+TN}$$

(34)

Precision refers to the proportion of correctly classified positive samples among all the classified positive samples, Recall refers to the number of correctly predicted data among the true positive samples, F1-score is the weighted average of the Precision and Recall, and Accuracy is the proportion of correctly classified samples among all the samples, TP and FP refer to the correctly classified positive samples and incorrectly classified negative samples, respectively, and FN and TN refer to incorrectly classified positive samples and correctly classified negative samples, respectively.

4.2. Pedestrian Interference Recognition Experiment

To illustrate the effectiveness of selecting the pedestrian distance, azimuth angle, head observation angle, and action type classification as the input variables for interference analysis, we used the distance between pedestrians, as presented by Alahi et al., to evaluate the pedestrian interference as a comparison input variable [8]. We also selected the distance and angle between the pedestrians, as proposed by Xue et al., to evaluate the pedestrian interference as a comparison [5]. The calculations were performed with the network used in the pedestrian interaction evaluation feature module described above. To account for the influence of the dropout layer on the final results, we repeated the training and testing 50 times during each experiment and finally recorded the maximum, minimum, and average values of the Precision, Recall, F1-score, and Accuracy. The results are shown in

Table 4. Comparison results of input parameters for pedestrian interference recognition. Recognition Accuracy (Acc.), Recall (Rec.), F1-score (F1.), Precision (Pre.). Min, Max, and Average (Ave.) results.

Variable		Acc.	Rec.	F1.	Pre.
Distance [8]	Min	0.653	0.631	0.501	0.619
	Max	0.706	0.757	0.667	0.774
	Ave.	0.684	0.701	0.558	0.677
Distance, Angle [5]	Min	0.943	0.963	0.845	0.916
	Max	0.970	0.987	0.941	0.970
	Ave.	0.957	0.976	0.897	0.945
Ours	Min	0.980	0.986	0.973	0.948
	Max	0.986	0.993	0.980	0.961
	Ave.	0.984	0.991	0.976	0.954

Marked in bold: the best model.

.

The recognition results with only the distance between pedestrians as the input variable had the lowest values of all four evaluation indicators. In comparison, the recognition results with the pedestrian distances plus the angle between the two significantly improved the evaluation indicators. Compared to the above methods, the recognition results for our proposed method were significantly better than the results of the other two methods. Some of the experimental results are shown in Figure 10.

4.3. Blind Observation State Classification Experiment

We used the K-means++ algorithm to cluster the PO data. The results are shown in Figure 11, from which we obtained the coordinates of the center point of the gesture cluster c1 when the pedestrian was not observing the road condition (i.e., looking forward) and the coordinates of the center point of the gesture clusters c2 and c3 when the pedestrian was observing the left and right sides within the blind area of the visual field, respectively. We defined the pose of each pedestrian using a triplet of angles (i.e., α, β, γ), where α was positive when the pedestrian’s head turned to the left, β was positive when the shoulder turned to the left, and γ was positive when the body leaned forward.

In addition, we constructed a test set from the pedestrian pose data collected under other view-obscuring scenarios to verify the classification effect of the obtained clustering centroids. The results showed that the above-mentioned clustering centroids provided a good classification benchmark for the pedestrian blindness observation degree states. Some of the experimental classification results are shown in Figure 12.

4.4. Pedestrian Hazardous Crossing Intention Recognition and Prediction Experiment

First, in terms of the types of pedestrian crossing intention recognition or prediction input variables, as shown in

Table 5. Comparison of results for various input parameters when predicting pedestrian hazardous crossing intention. Recognition Accuracy (Acc.), Recall (Rec.), F1-score (F1.), Precision (Pre.). Min, Max, and Average (Ave.) results.

Variable		Acc.	Rec.	F1.	Pre.
Position [8]	Min	0.662	0.629	0.747	0.774
	Max	0.788	0.825	0.798	0.920
	Ave.	0.723	0.713	0.774	0.864
Position,2D pose [4]	Min	0.806	0.847	0.807	0.745
	Max	0.895	0.954	0.905	0.915
	Ave.	0.861	0.886	0.869	0.856
Position,3D pose [45]	Min	0.954	0.960	0.957	0.949
	Max	0.972	0.982	0.974	0.983
	Ave.	0.962	0.968	0.965	0.963
Ours	Min	0.963	0.966	0.960	0.965
	Max	0.984	0.994	0.988	0.985
	Ave.	0.974	0.975	0.978	0.976

Marked in bold: the best model.

, we chose three control groups to be classified using our network separately to illustrate the validity of our selected input parameters. For the first control group, we compared the results for the proposed input variables with the results obtained with the input used by Goldhammer et al.—only the historical location of the pedestrian—for performing pedestrian trajectory prediction [8]. For the second control group, we compared results for the proposed input variables with the results obtained with the inputs used by Fang et al.—the historical location of the pedestrian and the 2D pose estimate—for performing pedestrian intention recognition [4]. For the third control group, we compared the results for the proposed input variables with the results obtained with the inputs used by Zhong et al.—the trajectory prediction using the pedestrian’s historical position and the 3D pose estimate—as input variables for comparison [46].

Finally, we selected the 3D pose features, interference state features, pedestrian blind observation state features, and pedestrian speed features as input variables for comparison. The computational results with our selected input parameters were improved for all evaluation metrics compared to the input variables used for the control groups. Second, in terms of the pedestrian crossing intention recognition computational methods, considering that the pedestrian hazardous crossing intention prediction module was obtained based on an LSTM improvement in this paper, we choose an LSTM and its improved Bi-LSTM as two algorithms for comparison. In addition, Zhang et al. [6] used the SVM, RF, gradient boosting machine (GBM), and extreme gradient boosting machine (XGBM) for pedestrian crossing intention recognition in their study, and the accuracy of the RF algorithm was higher than those of the other algorithms. Therefore, we chose the RF algorithm as another algorithm control group. As shown in

Table 6. Comparison of results for various pedestrian hazardous crossing intention recognition algorithms. Recognition Accuracy (Acc.), Recall (Rec.), F1-score (F1.), Precision (Pre.). Min, Max, and Average (Ave.) results.

Model		Acc.	Rec.	F1.	Pre.
LSTM	Min	0.947	0.945	0.937	0.951
	Max	0.978	0.994	0.994	0.980
	Ave.	0.967	0.969	0.971	0.970
Bi-LSTM	Min	0.944	0.959	0.948	0.926
	Max	0.975	0.993	0.977	0.971
	Ave.	0.959	0.970	0.962	0.955
RF	Min	0.865	0.812	0.825	0.838
	Max	0.916	0.899	0.882	0.903
	Ave.	0.890	0.859	0.859	0.865
Ours	Min	0.963	0.966	0.960	0.965
	Max	0.984	0.994	0.988	0.985
	Ave.	0.974	0.975	0.978	0.976

Marked in bold: the best model.

, our constructed IPVO-LSTM algorithm outperformed the other two algorithms overall in all evaluation metrics for pedestrian hazardous crossing intention recognition in view-obscured areas (in our study, the parameters of the RF were chosen as follows: the number of classifiers was 10, and the maximum depth of the decision tree was 15; the parameters of the LSTM and Bi-LSTM were selected as follows: the number of hidden units was set to 100, and the dropout rate was set to 0.5).

Second, in terms of the pedestrian crossing intention recognition computational methods, considering that the pedestrian hazardous crossing intention prediction module was obtained based on an LSTM improvement in this paper, we choose an LSTM and its improved Bi-LSTM as two algorithms for comparison. In addition, Zhang et al. [6] used the SVM, RF, gradient boosting machine (GBM), and extreme gradient boosting machine (XGBM) for pedestrian crossing intention recognition in their study, and the accuracy of the RF algorithm was higher than those of the other algorithms. Therefore, we choose the RF algorithm as another algorithm control group. As shown in Table 6, our constructed IPVO-LSTM algorithm outperformed the other two algorithms overall in all evaluation metrics for pedestrian hazardous crossing intention recognition in view-obscured areas (in our study, the parameters of the RF were chosen as follows: the number of classifiers was 10, and the maximum depth of the decision tree was 15; the parameters of the LSTM and Bi-LSTM were selected as follows: the number of hidden units was set to 100, and the dropout rate was set to 0.5).

Finally, in terms of intention prediction time range, according to vehicle dynamics, if the hazardous crossing intentions of pedestrians could be predicted appropriately in advance, a greater reaction time could be provided to the driver or the automated driving system. According to a study by Mercedes-Benz, Germany, it was found that if the driver could be warned 0.5 s earlier, 60% of traffic accidents could be effectively avoided [47]. Therefore, to ensure safety, we advanced the prediction time to 1 s to verify the intention prediction effectiveness of our method at 0.1, 0.5, and 1 s before the pedestrians crossed the street (this was achieved by adopting Zhang’s method of independent variable advancement [6]). The calculation results are shown in

Table 7. Prediction results for different time horizons. Prediction Accuracy (Acc.), Recall (Rec.), F1-score (F1.), Precision (Pre.). Min, Max, and Average (Ave.) results.

Prediction Horizon		Acc.	Rec.	F1.	Pre.
0.1 s	Min	0.963	0.966	0.960	0.965
	Max	0.984	0.994	0.988	0.985
	Ave.	0.974	0.975	0.978	0.976
0.5 s	Min	0.960	0.940	0.963	0.960
	Max	0.975	0.977	0.977	0.988
	Ave.	0.968	0.967	0.971	0.975
1 s	Min	0.828	0.857	0.828	0.762
	Max	0.944	0.929	0.950	0.977
	Ave.	0.889	0.888	0.898	0.911

. With the increase in the time range, although the evaluation indices all decreased, even in the case of 1 s advance, the indices were still able to be maintained above 0.88. In addition, some of the experimental results are shown in Figure 13.

5. Conclusions

In this study, we used pedestrian interaction state features, pedestrian 3D pose features, pedestrian velocity features, and pedestrian blind observation state features as input data and constructed an ff-LSTM to fuse the above features, providing a method for recognizing and predicting pedestrian crossing intentions when views are obstructed. We also verified the effectiveness of our given pedestrian interference judgment method by comparison experiments. In the experimental section, the advancement of our method was verified from three perspectives: input parameter selection, intention recognition algorithm, and intention prediction time range. In the input parameter selection and intention recognition algorithm experiments, the mean values of our method’s four evaluation metrics (Recognition Accuracy, Recall rate, F1-score, and Precision) were 0.974, 0.975, 0.978, and 0.976, which are higher than other commonly used methods. In the intention prediction time range experiment, our method achieved 0.889, 0.888, 0.898, and 0.911 for the four evaluation metrics, even though 1 s advanced the independent variable. The limitations of our study are mainly the following:

• As the system relies on vision sensors, we may not be able to make accurate decisions at night, in bad weather, fog, dust, or when the pedestrian is far from the camera.
• Only the case between two pedestrians is considered in the pedestrian interaction module, and the case of three or more people is not covered.
• We did not consider hazardous crossing intentions caused by the irrational behavior of pedestrians, e.g., when pedestrians intentionally cross the street with the intention of blackmailing or killing themselves after spotting vehicles in the blind spot.

To address the above problems, subsequent studies can consider adding sensors such as radar, improving the model accordingly to reduce the impact of ambient lighting and camera hardware limitations on the recognition and prediction, and adding the group activity recognition (GAR) algorithm to achieve pedestrian behavior recognition for more than two people [48].

As for the hazardous crossing problem caused by the irrational behavior of pedestrians, we can consider introducing the quantum-mechanics-based irrational driving behavior decision method in our previous study [49].