A Safety-Assured Semantic Map for an Unstructured Terrain Environment towards Autonomous Engineering Vehicles

Abstract

Accurate obstacle detection plays a crucial role in the creation of high-precision maps within unstructured terrain environments, as it supplies vital decision-making information for unmanned engineering vehicles. Existing works primarily focus on the semantic segmentation of terrain environments, overlooking the safety aspect of vehicle driving. This paper presents a hazardous obstacle detection framework in addition to driving safety-assured semantic information in the generated high-precision map of unstructured scenarios. The framework encompasses the following key steps. Firstly, a continuous terrain point cloud model is obtained, and a pre-processing algorithm is designed to filter noise and fill holes in the point cloud dataset. The Sobel-G operator is then utilized to establish a digital gradient model, facilitating the labeling of hazardous obstacles. Secondly, a bidirectional long short-term memory (Bi-LSTM) neural network is trained on obstacle categories. Finally, by considering the geometric driving state of the vehicle, obstacles that pose safety risks to the vehicle are accurately extracted. The proposed algorithm is validated through experiments conducted on existing datasets as well as real, unstructured terrain point clouds reconstructed by drones. The experimental results affirm the accuracy and feasibility of the proposed algorithm for obstacle information extraction in unstructured scenes.

1. Introduction

Autonomous engineering vehicles have demonstrated significant potential in various industries, including construction, mining, agriculture, and forestry [1,2]. A fundamental aspect of autonomous driving systems is the utilization of High-Definition (HD) maps, which incorporate semantic information to ensure safe and effective navigation. Semantic maps encompass both geometric mapping and semantic data, playing a crucial role in path planning and navigation tasks [3,4]. In the context of structured road environments, HD maps are generated by collecting point cloud data and images using advanced onboard sensors. These maps contain valuable prior information about complex road conditions, such as road elements, lane lines, traffic signs, and objects of interest [5]. While semantic map construction techniques have been widely employed in urban road scenarios, unstructured environments in which engineering vehicles operate and are characterized by irregular terrain patterns pose distinct challenges. In such environments, clear road boundaries and traffic signs are often absent, and the semantic environment is represented by labeled obstacles and access areas. As a result, the accuracy of semantic segmentation in complex domains significantly deteriorates. Additionally, the harsh vibration and dust present in these environments severely affect the precision and reliability of onboard sensors. Consequently, developing reliable semantic maps for unstructured terrain environments, specifically for autonomous engineering vehicles, becomes even more challenging.

The methods employed for mapping unstructured terrain can be broadly categorized as online real-time scanning and offline data modeling. Real-time terrain scanning, utilizing LiDAR or visual approaches, is a crucial component of Simultaneous Localization and Mapping (SLAM) technology, widely adopted in mobile robotics. For instance, Ebadi et al. implemented and tested a LiDAR-based multi-robot SLAM system for large-scale autonomous mapping in subterranean environments with degraded perception [6]. Pan et al. proposed an online elevation mapping system that rapidly generates dense local elevation maps for fast, responsive local planning while maintaining a globally consistent, dense map for path routing [7]. In comparison to LiDAR mapping, visual modeling facilitates data acquisition and online postprocessing, showcasing its superiority in small-sized field robots and planetary exploration. Matthies et al. discussed the progress in computer vision techniques for planetary rovers and landers, where visual terrain maps provide reliable environmental information for navigation planning and motion control of Mars rovers [8,9]. Ma et al. achieved autonomous 24 hour operation of a dynamic quadruped in complex field terrains through a combination of stereo cameras, an IMU, and leg odometry [10]. Bernuy et al. designed an outdoor semantic mapping method based on visual information and topological maps, enriching the off-road scene with semantic descriptions [11]. Sock et al. estimated the traversability of unstructured terrains and built a probabilistic grid map online using 3D LiDAR and cameras, exploiting the complementary nature of LiDAR and camera data [12].

Real-time terrain modeling aims to achieve efficient data acquisition and mapping in dynamic environments, whereas offline data modeling primarily emphasizes mapping accuracy, particularly for centimeter-scale global maps. Large-scale terrain data is typically collected through manual measurements, LiDAR scanning, and UAV tilt photography. Manual surveying and mapping involve the use of GPS, theodolite, and rangefinder but are mostly suitable for simple, structured scenes due to the manual measurement requirement [13,14]. High-precision LiDAR mapping relies on vehicle or UAV platforms to perform continuous scanning and registration of terrain point clouds, although its high cost restricts its application in autonomous driving systems [15,16,17]. UAV tilt photography, utilizing high-precision cameras, enables the rapid capture of a large number of multi-view images along a designated route. Through camera calibration, matching, and triangulation of image datasets, a 3D terrain model can be generated. This approach offers the comprehensive advantages of low cost, fast modeling, and high accuracy in large-scale, complex outdoor environments [18,19]. Liu et al. produced a high-precision true digital orthophoto map of buildings based on UAV images, validating its accuracy using randomly distributed checkpoints [20]. Liu et al. achieved accurate mapping in UAV photogrammetry without ground control points in the map projection frame, demonstrating elevation accuracy in unstructured terrain environments [21].

While urban road vehicles experience driving instability primarily due to high-speed dynamic operations, lower-speed engineering vehicles in off-road scenarios are more prone to unsafe rollover accidents due to variable vehicle structures and complex terrain environments. This risk is particularly significant when facing occluded visual fields or inaccurate traversability estimation [22,23,24]. Effectively predicting and estimating the risk motion state through a coupled vehicle-terrain dynamic model in unstructured environments enables early risk warning and motion trajectory planning for autonomous engineering vehicles [25]. However, real-time solutions of the coupled model increase the burden on the vehicle control unit and do not provide sufficient time margin for control decision-making in autonomous engineering vehicles. Therefore, integrating safety-assured semantic information into global maps can efficiently provide real-time risk motion states, enabling safe driving operations for off-road engineering vehicles.

In summary, the existing unstructured scenarios predominantly delineate three principal classifications. These classifications encompass natural landscapes inclusive of mountainous terrain and grassland expanses, urban thoroughfares encompassing thoroughfares, architectural constructs, and pedestrian domains, and interior environments characterized by unstructured layouts, furnishings, and ornamental embellishments. Most existing work focuses on semantic segmentation of the terrain environment without considering vehicle dynamics, classifying obstacles of interest as pits, bulges, stones, etc. [26,27]. Additionally, most of the ways to process data require extremely high computational power and deep network structures. This paper proposes a hazardous obstacle detection framework tailored for unmanned engineering vehicles engaged in operations within natural landscapes. The key novelty of this work is the combination of point cloud segmentation techniques with vehicle geometry models. The pre-processing and Bi-LSTM networks employed achieve competitive results without excessive computational power. Furthermore, it provides more effective driving safety semantic information for high-precision maps of unstructured scenes. The main contributions of this paper can be summarized as follows:

• Point cloud filtering algorithms that employ geometric and color spaces to filter outliers and refine unstructured scene point cloud models;
• Introducing the Sobel-G operator to build a digital gradient model. The reverse hole-filling method is designed to address point cloud model defects, and the OTSU segmentation algorithm is employed for further marking of hazardous obstacles;
• Innovative application of a bidirectional long short-term memory (Bi-LSTM) neural network to train point cloud features, facilitating faster and more accurate semantic segmentation of hazardous obstacles in similar unstructured environments;
• Accurate extraction of hazardous obstacles that impact the safe driving of vehicles by leveraging the vehicle-terrain model.

2. Methods

The proposed method is illustrated in Figure 1 and consists of three main stages: pre-processing, deep learning semantic segmentation, and obstacle detection based on the vehicle’s geometric model. The aim is to identify perilous impediments affecting the secure mobility of engineering vehicles and subsequently integrate this data into a high-precision map representation. The hazardous impediments, as posited within this research, specifically refer to convex obstacles encountered within unstructured roadway contexts. In the pre-processing stage, the image point cloud map model undergoes filtering and hole-filling to address the interference caused by outlier points in the point cloud model. Segmentation and filling algorithms are then applied to segment and label the obstacles, providing training data for the deep learning network. Subsequently, the deep learning network quickly extracts obstacles of the same type from the point cloud. Finally, hazardous obstacles that affect the vehicle’s safe driving are segmented in conjunction with the vehicle’s geometric model.

2.1. Experimental Equipment and Site

Terrain-based 3D point clouds play a pivotal role in extracting semantic representations. To ascertain the viability of the suggested approach, we utilize UAV tilt photogrammetry for the generation of an image point cloud model dataset specifically tailored for unstructured terrain settings. As depicted in Figure 2, an irregular topographic region spanning an approximate area of 2000 square meters is chosen as the designated site for surveying and modeling endeavors. The planning of aerial routes encompasses five distinct operational directions: a singular orthophoto route and four tilt photography routes directed towards the east, south, west, and north orientations. A comprehensive summary of the key parameters associated with the UAV employed for the experimental activities is furnished in

Table 1. UAV basic parameters.

DJI Phantom 4 Pro
Photosensitive device area	1 inch CMOS	Angle of view	84°
Effective pixels	20 MP	Aperture	f/2.8–f/11
Focal length	35 mm equivalent	Shutter speed	8–1/8000 s
Focusing distance	1 m–infinity	IOS	100–12,800
Weight	1375 g	Flight duration	30 min
Vertical retention accuracy	± 0.1 m	Horizontal preservation accuracy	± 0.3 m
Gimbal pitch angle	30°–90°	Gimbal angle jitter	± 0.02°

.

2.2. Terrain Point Cloud Filtering and Hole Completion

Throughout the image acquisition procedure, the terrain-derived point cloud sourced from the drone’s tilting maneuver often exhibits instances of noise, outliers, and perturbations stemming from carrier frame oscillations, as well as effects introduced by factors like smoke, dust shading, and variations in illumination. Moreover, as illustrated in Figure 3, the presence of weeds within the unstructured environment might not directly compromise the operational security of unmanned engineering vehicles. Nevertheless, these elements can be erroneously interpreted as obstacles within the point cloud data, thereby impinging upon the decision-making process. To faithfully represent the underlying topographical contours, the original point cloud undergoes a preliminary pre-processing stage rooted in statistical filtering, encompassing both geometric and color space domains. This procedural facet encompasses the exclusion of outliers and weed-associated point clouds.

• Geostatistical filtering on outliers

Assuming that the average geometric distance between any point in the point cloud and the surrounding points follows a normal distribution $\mathrm{N}\left(\mathsf{\mu },{\mathsf{\sigma }}^2\right)$, a filtering threshold of $\mathsf{\mu }+2\mathsf{\sigma }$ is adopted in this paper. Points with an average distance greater than the threshold are considered outliers and filtered out.

First, the average distance $\overline{{\mathrm{D}}_{\mathrm{i}}}$ between points in the neighborhood space is calculated using the following formula:

$$\overline{{\mathrm{D}}_{\mathrm{i}}}=\frac{{{\displaystyle \sum }}_{\mathrm{n}=1}^{\mathrm{N}}\sqrt{{({\mathrm{x}}_{\mathrm{i}}-{\mathrm{x}}_{\mathrm{n}})}^2+{({\mathrm{y}}_{\mathrm{i}}-{\mathrm{y}}_{\mathrm{n}})}^2+{({\mathrm{z}}_{\mathrm{i}}-{\mathrm{z}}_{\mathrm{n}})}^2}}{\mathrm{N}},$$

(1)

where $\mathrm{P}\left({\mathrm{x}}_{\mathrm{i}},{\mathrm{y}}_{\mathrm{i}},{\mathrm{and}\mathrm{z}}_{\mathrm{i}}\right)$ represents any point in the point cloud, $K\left(x_n,y_n,and z_n\right)$ represents a point within the range of point P, and N is the number of points in the neighborhood. The larger the value of $\overline{\mathrm{D}}$, the further the point is from its surrounding points.

A field window of 0.04 square meters has been selected for this paper. Therefore, point K satisfies the following condition:

$$K\in \text{{}\left(x_i,y,z_i\right)\left||x_i-x_n\right|<10,|y_i-y_n|<10\}.$$

(2)

After that, the expectation (mean) and standard deviation of the point distances from the mean value $\overline{\mathrm{D}}$ in the neighborhood space are calculated for all points using the following formulas:

$$\mu ={{\displaystyle \sum }}_{i=1}^{NUM}\frac{\overline{D_i}}{NUM},$$

(3)

$$\sigma =\sqrt{\frac{{{\displaystyle \sum }}_{i=1}^{NUM}{\left(\overline{D_i}-\mu \right)}^2}{NUM}},$$

(4)

where NUM is the total number of points in the point cloud.

Figure 4 illustrates the effect of the geostatistical filtering algorithm on a group of original terrain point cloud data in a $3 \mathrm{m}\times 3 \mathrm{m}$ area. The original point cloud data consists of 36,686 spatial point coordinates. After applying the statistical filtering with the filtering threshold set to $\mathsf{\mu }+2\mathsf{\sigma }$, the point cloud data is reduced to 33,147 points. The statistical filtering algorithm effectively removes noise and outlier data points, as demonstrated in the figure.

• Color space statistical filtering on surface weed

Given the inclusion of comprehensive color characteristics within point cloud models derived from unmanned aerial photography and the notable dissimilarity in feature information between weeds and other entities within unstructured terrains, this study presents a method for weed surface filtration utilizing color space statistical filtering.

The point cloud is segmented using a statistical method where the average distance ${\overline{D}}_{\mathrm{RGB}}$ of the RGB color space of neighboring points is used as the evaluation metric. The formula for calculating ${\overline{D}}_{RG{B}_i}$ is as follows:

$${\overline{D}}_{RG{B}_i}=\frac{{{\displaystyle \sum }}_{n=1}^N\sqrt{{(R_i-R_n)}^2+{(G_i-G_n)}^2+{(B_i-B_n)}^2}}{N},$$

(5)

in the formula, R, G, and B represent the coordinates of the three axes in the RGB color space; $\left(R_i,G_i,and B_i\right)$ are the color space coordinates corresponding to any point $\mathrm{P}\left({\mathrm{x}}_{\mathrm{i}},{\mathrm{y}}_{\mathrm{i}},{\mathrm{and}\mathrm{z}}_{\mathrm{i}}\right)$ in the point cloud; and $\left(R_n,G_n,and B_n\right)$ are the color space coordinates corresponding to point $\mathrm{K}\left({\mathrm{x}}_{\mathrm{n}},{\mathrm{y}}_{\mathrm{n}},{\mathrm{and}\mathrm{z}}_{\mathrm{n}}\right)$ in the neighborhood of point P. N is the number of points in the neighborhood.

Figure 5 illustrates the effect of color space statistical filtering using the filtering threshold $\mathsf{\mu }+2\mathsf{\sigma }$ The filtering algorithm described in this paper effectively extracts and separates the surface weed point cloud.

• Hole-filling of filtered point clouds

Following the removal of weed-associated point clouds, certain lacunae may manifest within the terrain point cloud. Remedying these voids is imperative to achieving an intact terrain point cloud representation. To this end, the study leverages the region-growing technique to address the task of point cloud cavity completion. Commencing from the periphery of the cavity and progressing inward, the expansion of points is orchestrated on a rasterized grid.

To ensure algorithm efficiency and match the downsampling resolution of the subsequent digital gradient model, the grid side length of the network is selected as 4 cm. The elevation value of the growing point P (the Z coordinate) is used to complete the hole. The formula for calculating $Z_P$ is as follows:

$$Z_P=\frac{{{\displaystyle \sum }}_{n=1}^N{z}_i}{{\mathrm{N}}_{\mathrm{f}}},K\in \{\left(x_i,y_i,z_i\right)\left|\left|x_i-x_n\right|<0.5l,|y_i-y_n\right|<0.5l\},$$

(6)

in the formula, ${\mathrm{N}}_{\mathrm{f}}$ is the total number of points in the neighborhood point cloud, and l is the side length of the neighborhood window.

The hole-filling point cloud growing algorithm runs inward from the hole boundary in a single iteration. Through multiple iterations, the point cloud holes can be filled, as depicted in Figure 6.

2.3. Segmentation of Primary Obstacles Based on a Digital Gradient Model

• Digital Gradient Model Construction and Obstacle Semantic Labeling Based on the Sobel-G Operator

In order to depict the morphological attributes of the terrestrial surface relief, the augmented point cloud map is transformed into a digital elevation model. To harness the elevation data effectively, a computation of gradients is executed for every individual point, thereby yielding a digital gradient model. Within this study, a tailored adaptation of the Sobel operator, denoted as the Sobel-G operator, has been devised to approximate the absolute gradient magnitude.

The coefficient matrix of the Sobel-G operator is expressed as a discrete gradient calculation formula:

$$\begin{gathered} g_x=\frac{f\left(x+1,y\right)}{4l}+\frac{f\left(x+1,y+1\right)+f\left(x+1,y-1\right)}{8l}, \\ -\frac{f\left(x-1,y\right)}{4l}+\frac{f\left(x-1,y+1\right)+f\left(x-1,y-1\right)}{8l}, \end{gathered}$$

(7)

$$\begin{gathered} g_y=\frac{f\left(x,y+1\right)}{4l}+\frac{f\left(x-1,y+1\right)+f\left(x+1,y+1\right)}{8l}, \\ -\frac{f\left(x,y-1\right)}{4l}+\frac{f\left(x-1,y-1\right)+f\left(x+1,y-1\right)}{8l}, \end{gathered}$$

(8)

in the formula, $x$ and $y$ are the point coordinates of the elevation information, and$\mathrm{l}$ is the side length of the operator grid.

The digital gradient model is constructed by applying the Sobel-G operator to the digital elevation model. The gradient values in the digital gradient model are then converted to grayscale for visualization and further processing. Figure 7 shows the terrain obstacles’ contours clearly observed in the digital gradient model.

Subsequent to the derivation of the digital gradient model through the utilization of the Sobel-G operator, a subsequent phase entails the partitioning of terrain impediments along their boundaries. Within the scope of this research, the OTSU segmentation algorithm is deployed to effectuate the partitioning of concave-convex barriers based on the gradient model. As illustrated in Figure 8, the segmentation outcomes reveal the accurate identification of three obstacles within the non-structured terrain through the employment of gradient model-based segmentation. In contrast, only two larger obstacles are discernible from the elevation model. Furthermore, the boundaries demarcating the segmented obstacles are characterized by enhanced precision when employing the gradient model-based segmentation approach.

• Indirect hole filling based on background region culling

The obstacle boundaries obtained from the gradient segmentation often contain hollow regions, as shown in Figure 9. Distinguishing between the background area and the hollow area can be challenging based solely on algorithm logic. Direct completion of these regions may lead to incorrect filling. To address this issue, this paper proposes an indirect hole filling Algorithm 1 based on background area culling. The algorithm indirectly obtains the complete obstacle target area by identifying the complete background area. The algorithmic steps are as follows:

Algorithm 1 Indirect hole filling for obstacles in boundary models

Input: Obstacle_Boundary_Model(OB) Output: Obstacle_Model(O) Start: 1. define   Seed_Point $⊳$ Seeds for regional growth 2. define   Background_Point_Cloud $⊳$ Background area obtained from regional growth 3. define   Raster_Grid $⊳$ Targeted raster network 4. while(1) do: 5. for Point in Raster_Grid: 6. if Point is located in the Seed_Point neighborhood and Point is not in OB: 7. Point add to Seed_Point 8. Point add to Background_Point_Cloud 9. end if 10. end for 11. if Seed_Point not growth: 12. Break 13. end if 14. for Point in Raster_Grid: 15. if Point is not in Background_Point_Cloud: 16. Point add to O 17. end if 18. end for 19. return O End

The algorithm starts by obtaining the obstacle boundary based on the segmentation of the digital gradient model. By setting area growth seeds, growing them to cover the background area, and subsequently removing the background area, the complete semantic information of the target obstacle is indirectly obtained. In cases where the background area A is large or discontinuous, multiple regional growth seeds can be set to ensure the timeliness and integrity of the background area extraction.

2.4. Point Cloud Segmentation Based on Bi-LSTM Neural Network

• The deep learning network framework for point cloud tuning

After completing the aforementioned steps, the semantic information of the geometric features of the target obstacle’s point cloud in the unstructured environment is fully extracted. To efficiently extract obstacles of the same type from the point cloud, we manually labeled the extracted obstacle information and employed a bidirectional long short-term memory (Bi-LSTM) deep learning method based on a bidirectional recurrent neural network to analyze the target features. The structure of the network is illustrated in Figure 10. The key components include a sequence input layer, a bidirectional long- and short-term memory layer with 300 hidden units, a fully connected layer, a classification layer, and a softmax layer.

During the network training process, the forget gate is responsible for discarding redundant information, the input gate updates the relevant point cloud feature information, and the output gate performs long- and short-term memory computations. The cyclic arrangement of these three gates enables the LSTM to learn how to balance the proportion of long- and short-term memory. The choice of a bidirectional long short-term memory neural network is driven by our intention to express the relationship between each point and its neighboring points in a sequential logic format, which better represents the geometry around the point cloud and the interactions between feature vectors. For each point, we identify its neighbors and input them into the deep learning algorithm as a sequence. To ensure training and semantic segmentation accuracy, we incorporate the scattered coordinates of each point in addition to considering the geometric features of the points within the point cloud sequence.

In comparison to traditional recurrent neural network models and LSTM models, which only propagate information forward, the Bi-LSTM model allows for the inclusion of context information more comprehensively at each time step. By leveraging a bidirectional recurrent neural network composed of LSTM neuron models, the Bi-LSTM model can capture contextual information with greater precision.

• Implementation process for applicable point clouds

As mentioned earlier, the implementation of the network is governed by three key gates: forget, input, and output. These gates are responsible for reading, selecting, and forgetting input information. Each gate functions as a fully connected layer, where the input is a vector and the output is a vector of real numbers between 0 and 1. The gate can be expressed as follows:

$$g\left(x\right)=\mathsf{\sigma }\left(Wx+b\right),$$

(9)

the $\mathsf{\sigma }$ (sigmoid) function maps the output values to the range (0–1), making the gate act as a switch that can control the flow of feature information. A gate output of 0 means that the input feature information cannot pass, while an output of 1 allows the feature information to pass through.

The network utilizes two gates to control the content of the unit’s cell state: the forget gate and the input gate. The forget gate determines how much of the previous moment’s unit states are retained in the current moment, while the input gate determines how much of the network’s input is stored in the cell state at the current moment. Finally, the output gate controls the output of the cell state to produce the current output value of the Bi-LSTM.

The specific representation is as follows:

Forget gate:

$$f_t=\mathsf{\sigma }\left(W_f·\left[h_{t-1},x_t\right]+b_f\right),$$

(10)

in the above formula $W_f$ is the weight matrix of the forget gate, and $\left[h_{t-1},x_t\right]$means to connect two vectors into a vector that receives longer information. $b_f$ is the bias term of the forget gate. $\mathsf{\sigma }$ is the sigmoid function. If the dimension of the input is ${\mathrm{d}}_{\mathrm{x}}$, the dimension of the hidden layer is ${\mathrm{d}}_{\mathrm{h}}$, and the dimension of the unit state is ${\mathrm{d}}_{\mathrm{c}}$, the dimension of the weight matrix $W_f$ of the forget gate is ${\mathrm{d}}_{\mathrm{c}}·({\mathrm{d}}_{\mathrm{h}}+{\mathrm{d}}_{\mathrm{x}})$. In order to allow more feature information from the point cloud to be acquired by the network, the weight matrix $W_f$ of this network concatenates the two matrices together. One is $W_{fh}$, which corresponds to the input item $h_{t-1}$, and its dimension is ${\mathrm{d}}_{\mathrm{c}}·{\mathrm{d}}_{\mathrm{h}}$. The other is $W_{fx}$, which corresponds to the input item $x_t$, and its dimension is ${\mathrm{d}}_{\mathrm{c}}·{\mathrm{d}}_{\mathrm{x}}$. Therefore, the $W_f$ of this paper can be written as:

$$\left[W_f\right]\left[\begin{array}{c}{h}_{t-1}\\ x_t\end{array}\right]=\left[W_{fh} W_{fx}\right]\left[\begin{array}{c}{h}_{t-1}\\ x_t\end{array}\right]=W_{fh}{h}_{t-1}+W_{fx}{x}_t,$$

(11)

input gate:

$$i_t=\mathsf{\sigma }\left(W_i·\left[h_{t-1},x_t\right]+b_i\right),$$

(12)

$${\tilde{c}}_t=\tan \mathrm{h}\left(W_c·\left[h_{t-1},x_t\right]+b_c\right),$$

(13)

in the above formula $W_i$ is the weight matrix of the input gate, and $b_i$ is the bias term of the input gate. ${\tilde{c}}_t$ is the cell state of the current input, which is calculated based on the previous output and the current input.

After obtaining the current input cell state ${\tilde{c}}_t$, the cell state $c_t$ at the current moment can be calculated. It is generated by multiplying the last unit state elementwise by the forget gate, multiplying the current input unit state elementwise by the input gate, and adding the two products. As shown in the following formula:

$$c_t=f_t\ast c_{t-1}+i_t\ast {\tilde{c}}_t,$$

(14)

the symbol $\ast$ indicates element-wise multiplication.

In this way, we combine the LSTM’s current memory and long-term memory to form a new cell state. For the control of the forget gate, it can save the feature information of the long sequence point cloud for a long time. At the same time, the control of the input gate can prevent redundant interference information from entering the memory.

Output gate:

$$o_t=\mathsf{\sigma }\left(W_o·\left[h_{t-1},x_t\right]+b_o\right),$$

(15)

where $W_o$ is the weight matrix of the output gate and $b_o$ is the bias term of the output gate.

The predicted output of the final point cloud feature, determined by the output gate and the cell state:

$$h_t=o_t\ast \tan \mathrm{h}\left(c_t\right).$$

(16)

2.5. Semantic Segmentation of Hazardous Obstacles Based on Vehicle-Terrain Model

To further identify hazardous obstacles within the primary obstacles, the static rollover motion of articulated engineering vehicles is taken into consideration. This part builds upon our previous work and has been further optimized. The articulated engineering vehicle in this paper is a scaled prototype of a real articulated wheeled loader, as depicted in Figure 11a. It possesses a variable structure, a variable mass center, and variable load characteristics. The vehicle’s kinematic model is established with the steering hinge point as the coordinate origin, as shown in Figure 11b.

The vehicle’s total mass is primarily distributed among different body parts: front body mass G1, front wheel mass G2, front swing arm mass G3, rear body mass G4, and rear wheel mass G5. The center of gravity of the vehicle (x0, y0, z0) is determined by the following equation:

$$\mathrm{G}=\frac{{{\displaystyle \sum }}_{\mathrm{n}=1}^5{\mathrm{M}}_{\mathrm{n}}{\mathrm{G}}_{\mathrm{n}}}{{{\displaystyle \sum }}_{\mathrm{n}=1}^5{\mathrm{M}}_{\mathrm{n}}},$$

(17)

here ${\mathrm{M}}_{\mathrm{n}}$ represents the mass of each vehicle body part, and ${\mathrm{G}}_{\mathrm{n}}$ denotes the coordinates of each vehicle body part.

In order to determine the grounding points of the tires, their coordinates must be determined. These include the left front wheel P1l(x1l, y1l, z1l), right front wheel P1r(x1r, y1r, z1r), left rear wheel P2l(x2l, y2l, z2l), and right rear wheel P2r(x2r, y2r, z2r), expressed as:

$$\mathrm{G}=\frac{{{\displaystyle \sum }}_{\mathrm{n}=1}^5{\mathrm{M}}_{\mathrm{n}}{\mathrm{G}}_{\mathrm{n}}}{{{\displaystyle \sum }}_{\mathrm{n}=1}^5{\mathrm{M}}_{\mathrm{n}}},$$

(18)

$$P_{1l}{}^T=R_1{\left(W,L_1,-H\right)}^T,P_{1r}{}^T=R_1{\left(-W,L_1,-H\right)}^T,$$

(19)

$$P_{2l}{}^T=R_2{\left(W,-L_2,-H\right)}^T,P_{2r}{}^T=R_2{\left(-W,-L_2,-H\right)}^T,$$

(20)

$$R_1=\left[\begin{array}{ccc}\cos \theta & -\sin \theta & 0\\ \sin \theta & \cos \theta & 0\\ 0& 0& 1\end{array}\right],R_2=\left[\begin{array}{ccc}\cos \alpha & 0& \sin \alpha \\ 0& 1& 0\\ -\sin \alpha & 0& \cos \alpha \end{array}\right],$$

(21)

in the above equations, $W$ represents half the track width. $L_1$ and $L_2$ represent the distances from the front and rear axles to the center of gravity, respectively; H denotes the height of the steering hinge point; $\theta$ represents the articulated steering angle; and $\alpha$ represents the rear axle swing angle.

To account for the influence of obstacle elevation on vehicle trafficability, invalid obstacles that do not directly affect the driving stability of engineering vehicles are filtered out by calculating the roll angle threshold as the vehicle crosses obstacles. For this purpose, a critical height threshold is defined to assess the height of hazardous obstacles obtained from the digital elevation model. The static rollover process of articulated engineering vehicles is illustrated in Figure 12.

As the vehicle’s wheels maneuver over an obstacle, the vehicle body will roll around the P1rP2 axis when the roll angle is less than the restraint angle of the rear oscillating axle. The roll angle $\phi$ generated at low speed is defined as:

$$\phi =\arcsin \left(\frac{h}{D}\right),D=2W\frac{L_1+L_2}{\sqrt{W^2+{\left(L_1+L_2\right)}^2}},$$

(22)

here $h$ represents the obstacle height, and $D$ is the projection distance from P1l to the P1rP2 axis.

Upon surpassing the prescribed rotational threshold of the rear oscillating axle, the vehicle’s orientation experiences a roll maneuver about the P1rP2r axis, where the parameter D equates to twice the vehicle width (D = 2W). The vehicle transits between these dual states throughout the rollover sequence, with the upper limit of the permissible roll angle computable either through vehicle geometric attributes or empirical data obtained from rollover test benches. Consequently, a designated safety factor δ is introduced to deduce the critical obstacle height, as outlined by the following expression:

$$h_0=\frac{\left(2W\sin {\phi }_{\max }\right)}{\delta },$$

(23)

by computing the limit roll angle of the scaled prototype vehicle, the invalid obstacles at low speed are filtered out using the elevation model, and the hazardous obstacles are further extracted in the terrain mapping. Figure 13 illustrates the semantic information of hazardous obstacles in a site with dense obstacles, effectively expanding the vehicle’s passable area.

3. Experiment and Result Analysis

3.1. Datasets and Evaluation Metrics

To evaluate the effectiveness of the algorithm framework proposed in this paper, experiments were conducted on self-collected and calibrated datasets as well as two authoritative datasets for further validation, as depicted in Figure 14. The first dataset used for verification is the ISPRS 3D semantic labeling contest dataset, where the extraction effectiveness of the proposed Bi-LSTM neural network is tested. This dataset consists of point cloud data acquired using a Leica ALS50 airborne laser scanner over Vaihingen. The points in the dataset have not been rasterized or post-processed. The dataset provides point labels for nine classes, including powerlines, low vegetation, impervious surfaces, cars, fences/hedges, roofs, facades, shrubs, and trees.

The second dataset used for verification is the Canadian planetary emulation terrain 3D mapping dataset, which aims to assess the obstacle detection effect of the segmentation network combined with vehicle kinematic modeling technology. This dataset comprises 95 laser scans obtained at the University of Toronto Institute for Aerospace Studies (UTIAS) indoor rover test facility in Toronto, Ontario, Canada. The test facility includes a large dome structure that covers a circular workspace area with a diameter of 40 m. Within this workspace, gravel was distributed to emulate scaled planetary hills and ridges, providing a characteristic natural and unstructured terrain.

For the tests conducted on these datasets, three evaluation metrics were employed to quantify the detection effectiveness of the proposed method: Recall, Precision, and F1-measure. Recall measures the true positive rate, representing the percentage of correctly identified positives in the ground truth. Precision measures the positive predictive value, representing the percentage of correctly identified positives out of the total identified positives. F1-measure is a combined performance metric that considers both Recall and Precision. The specific expressions of these metrics are as follows:

$$Recall=\frac{TP}{TP+FN},$$

(24)

$$Precision=\frac{TP}{TP+FP},$$

(25)

$$F1 measure=\frac{2\times Recall\times Precision}{Recall+Precision}$$

(26)

here TP represents the number of true positives, FN represents the number of false negatives, and FP represents the number of false positives in the point cloud data. These metrics provide a comprehensive assessment of the detection performance of the proposed method.

3.2. Obstacle Detection Results of the Image Point Cloud

The results of our proposed approach for risky obstacle detection on the validation set of the drone-generated image point cloud model are presented in Figure 15 and Figure 16. These figures depict the predicted obstacle detection outcomes. The performance of our method is comprehensively evaluated using three metrics: Precision, Recall, and F1-score. The performance values are summarized in

Table 2. Hazardous obstacle prediction result for a medium scale unstructured scene.

	Hazardous Obstacle	Ground	Precision	Recall	F1
Hazardous obstacle	11,639	1416	93.99%	89.15%	91.51%
Ground	743	27,160	95.04%	97.34%	96.18%

.

Among these indicators, for the detection of dangerous obstacles, our method achieved a Precision of 93.99%, Recall of 89.15%, and an F1-score of 91.51%. These results demonstrate that our method exhibits highly accurate and reliable performance on the annotated dataset.

The visual representations in Figure 15 and Figure 16, combined with the quantitative evaluation in Table 2, highlight the effectiveness and robustness of our proposed approach for obstacle detection in the image point cloud. The achieved performance metrics indicate that our method is capable of accurately identifying and segmenting risky obstacles, providing a solid foundation for safe driving decision-making in unstructured terrain environments.

3.3. Classification Results of the ISPRS 3D Semantic Labeling Contest Dataset

To assess the holistic efficacy of the proposed Bi-LSTM neural network, a series of assessments were carried out on the ISPRS 3D semantic labeling contest dataset. The outcomes of the classification process are meticulously documented, with the corresponding confusion matrix presented in

Table 3. Confusion matrix of our method based on the ISPRS benchmark testing data. Precision, Recall, and F1 scores are shown for each category.

	Power	Low_veg	Imp_Surf	Car	Fence	Roof	Facade	Shrub	Tree
Power	230	1	0	0	0	195	10	1	163
Low_veg	0	73,154	9471	190	497	6114	212	6002	3050
Imp_Surf	0	6542	94,725	130	8	307	14	207	53
Car	0	421	27	1899	276	174	23	857	31
Fence	0	911	82	48	1398	176	240	3427	1140
Roof	64	2441	53	3	127	93,402	1123	793	11,042
Façade	31	478	73	68	727	1447	4083	1276	3041
Shrub	0	3981	202	227	742	1879	307	12,386	5094
Tree	4	1038	26	34	532	3379	938	4490	43,785
Precision (%)	69.91	82.23	90.51	73.07	32.46	87.23	58.75	42.07	64.96
Recall (%)	38.33	74.13	92.88	51.21	18.84	85.65	36.38	49.91	80.75
F1 (%)	49.51	77.97	91.68	60.22	23.84	86.43	44.93	45.66	72.00

and Figure 17. Furthermore, a visual depiction of the classification outcomes is furnished in Figure 18.

Remarkably, our method accomplished accurate predictions for 67% of the test points, encompassing clearly defined categories such as Imp_Surf and Roof. This notable achievement can be attributed to the synergistic amalgamation of distinctive features and the temporal sequence training embedded within our methodology.

Table 4. Quantitative comparisons on the ISPRS 3D semantic labeling contest dataset. The numbers in the first nine columns present the F1 scores for different categories. The average F1 score (Avg. F1) is listed in the last column.

Method	Power	Low_veg	Imp_Surf	Car	Fence	Roof	Facade	Shrub	Tree	Avg. F1
PointNet [28]	0.526	0.700	0.832	0.112	0.075	0.748	0.078	0.246	0.454	0.419
DT+GMM [29]	0.018	0.653	0.746	0.333	0.144	0.818	0.287	0.348	0.732	0.453
DGCNN [30]	0.446	0.712	0.810	0.420	0.118	0.938	0.643	0.464	0.817	0.597
alsNet [31]	0.701	0.805	0.902	0.457	0.076	0.931	0.473	0.347	0.745	0.604
HA-GCN [32]	0.303	0.866	0.903	0.765	0.370	0.957	0.377	0.582	0.827	0.694
RandLA-Net [33]	0.690	0.801	0.914	0.731	0.362	0.937	0.584	0.458	0.827	0.701
Ours	0.495	0.780	0.917	0.602	0.238	0.864	0.449	0.457	0.720	0.614

presents a quantitative comparative analysis between our proposed approach and several contemporary, comprehensive classification methodologies utilizing the ISPRS dataset. The examined methodologies encompass the machine learning-based DT+GMM method alongside deep learning-based strategies such as DGCNN, alsNet, HA-GCN, and RandLA-Net.

In contrast to the machine learning paradigm represented by DT+GMM, our deep learning approach yields a notably elevated average F1-score of 0.614, attesting to its superior efficacy in the domain of point cloud processing and classification. Furthermore, in contrast to the foundational deep learning framework of PointNet, our proposed approach demonstrates commendable competitive attributes, manifesting superior classification performance.

Compared to DGCNN, which prioritizes local feature acquisition between data points, and alsNet, characterized by an encoding-decoding architecture, our deeper network structure yields discernible computational efficiency enhancements. By leveraging Bi-LSTM networks to train classifier features as a recognition sequence, our approach attains superior average F1-scores relative to DGCNN and alsNet, surpassing them by margins of 0.017 and 0.01, respectively.

While a noticeable disparity persists between our approach and the advanced methodologies of HA-GCN and RandLA-Net, our technique remains a viable contender, particularly within categories characterized by well-defined features like Imp_Surf. This observed gap can be attributed to the adoption of extensive, multi-tiered network architectures in recent algorithms, benefiting from potent GPUs to enhance point cloud processing. Conversely, our proposed methodology presents a compact model size and abbreviated training time, rendering it efficacious and cost-efficient for practical implementation. The training process consumes approximately 10 min, and the model’s dimensions are confined to a mere 5 MB, negating the necessity for GPU-based training.

In summation, the classification outcomes gleaned from the ISPRS dataset affirm both the effectiveness and efficiency of our proposed Bi-LSTM neural network approach in semantic segmentation endeavors. This substantiates its prospective utility in facilitating comprehensive point cloud analysis and classification, with implications extending across diverse application domains.

3.4. Classification Results of the Canadian Planetary Emulation Terrain 3D Mapping Dataset

Figure 19 showcases the obstacle detection results obtained from the Canadian Planetary Emulation Terrain 3D Mapping dataset. A comparison is made between Chen’s latest method and our proposed method for obstacle detection. Chen’s method is depicted in Figure 19b. As can be observed, his obstacle detection method, which relies on a clustering algorithm, fails to accurately detect obstacles due to the black point clouds in the dataset not being completely separated.

On the other hand, Figure 19c presents the prediction results of our neural network-based obstacle detection method. By leveraging deep learning, our approach overcomes the limitations of traditional clustering algorithms that can only detect specific datasets and classified objects. Our method demonstrates the ability to detect more comprehensive obstacles, showcasing the algorithm’s generalizability to different data.

Furthermore, Figure 19d displays the prediction results of hazardous obstacles that impact driving safety, combining the dynamics of the vehicle’s geometry. In contrast to the global obstacle detection method, our approach, which incorporates dynamic hazardous obstacle prediction, provides the optimal path for vehicles to navigate through, ensuring their safety during the driving process.

Overall, the results obtained from the Canadian Planetary Emulation Terrain 3D Mapping dataset demonstrate the effectiveness of our proposed method in detecting obstacles and predicting hazardous obstacles. The combination of deep learning-based obstacle detection and vehicle geometry dynamics contributes to improved obstacle detection accuracy and enhanced safety in driving scenarios.

3.5. Engineering Applications of the Hazardous Obstacle Detection Framework

Figure 20 demonstrates the practical application of our hazardous obstacle detection framework in the rapid construction of high-precision maps for unmanned engineering vehicles in unstructured environments. This framework effectively tackles the challenges faced in unstructured environments by utilizing UAV oblique photography techniques to construct terrain point clouds. By employing UAVs instead of vehicle-mounted platforms, we overcome the difficulties associated with driving on unstructured terrain and the high vibration that can impact modeling accuracy. This approach also meets the demand for rapid and periodic modeling of dynamically changing unstructured surfaces.

The framework incorporates various steps to enhance the accuracy and usefulness of the constructed maps. Initially, statistical filtering is applied in both geometric and color spaces to remove outliers and ground-weed point clouds. This pre-processing step optimizes the appearance of the terrain point cloud and establishes a formatted data model that facilitates indexing and further analysis.

Semantic segmentation of obstacles is achieved through deep learning-based techniques. This step provides rich semantic information regarding the regional driving safety level. By constructing a vehicle-terrain contact coupling model, static stability indicators are calculated, enabling the segmentation of driving safety levels. This additional semantic information enhances the design of autonomous driving decision-making and control systems for engineering vehicles.

In engineering scenarios, our framework enables the rapid and accurate construction of high-precision maps in unstructured environments. It addresses the specific challenges associated with such environments, improves the quality of modeling, and provides valuable semantic information for the design of autonomous driving decision-making and control systems for engineering vehicles. By incorporating advanced technologies, our framework enhances the safety and efficiency of unmanned engineering vehicle operations in complex terrain.

4. Conclusions

In this paper, we have presented a comprehensive framework for hazardous obstacle detection in unstructured environments using drone-based modeling. It is a combined method of innovation where the synergistic integration of various techniques contributes to a novel and effective approach. By leveraging the advantages of drones, such as low cost, speed, and periodicity, we have successfully addressed the challenges associated with modeling large-scale unstructured scenarios. Through point cloud filtering and the application of deep learning algorithms for obstacle identification, we have significantly improved the availability of obstacle semantic information. Furthermore, we have incorporated a vehicle-terrain dynamics model specific to articulated engineering vehicles to extract hazardous obstacles that pose a threat to vehicle safety. By integrating this information into high-precision maps, our framework provides valuable semantic and vehicle safety information for autonomous engineering vehicles. This enhances their ability to navigate and make informed decisions in complex environments. Moving forward, our future research will focus on developing path tracking algorithms based on high-precision maps with semantic information about hazardous obstacles. By further advancing the capabilities of our framework, we aim to improve the path planning and decision-making processes for engineering vehicles operating in unstructured environments.