Real-Time Semantic Segmentation of 3D LiDAR Point Clouds for Aircraft Engine Detection in Autonomous Jetbridge Operations

Abstract

This paper presents a study on aircraft engine identification using real-time 3D LiDAR point cloud segmentation technology, a key element for the development of automated docking systems in airport boarding facilities, known as jetbridges. To achieve this, 3D LiDAR sensors utilizing a spinning method were employed to gather surrounding environmental 3D point cloud data. The raw 3D environmental data were then filtered using the 3D RANSAC technique, excluding ground data and irrelevant apron areas. Segmentation was subsequently conducted based on the filtered data, focusing on aircraft sections. For the segmented aircraft engine parts, the centroid of the grouped data was computed to determine the 3D position of the aircraft engine. Additionally, PointNet was applied to identify aircraft engines from the segmented data. Dynamic tests were conducted in various weather and environmental conditions, evaluating the detection performance across different jetbridge movement speeds and object-to-object distances. The study achieved a mean intersection over union (mIoU) of 81.25% in detecting aircraft engines, despite experiencing challenging conditions such as low-frequency vibrations and changes in the field of view during jetbridge maneuvers. This research provides a strong foundation for enhancing the robustness of jetbridge autonomous docking systems by reducing the sensor noise and distortion in real-time applications. Our future research will focus on optimizing sensor configurations, especially in environments where sea fog, snow, and rain are frequent, by combining RGB image data with 3D LiDAR information. The ultimate goal is to further improve the system’s reliability and efficiency, not only in jetbridge operations but also in broader autonomous vehicle and robotics applications, where precision and reliability are critical. The methodologies and findings of this study hold the potential to significantly advance the development of autonomous technologies across various industrial sectors.

1. Introduction

The safety and efficiency of an aircraft heavily depend on the state of its engine [1]. A decline in engine performance or a defect can significantly reduce the performance of the aircraft, and in severe cases can lead to safety accidents. Therefore, accurately monitoring the state of an aircraft engine is crucial. However, accurately recognizing and monitoring an aircraft engine is not easy, as such engines have complex structures and operate under various environmental conditions [2].

To solve this problem, 3D LiDAR (light detection and ranging) sensors have recently been receiving attention. Such 3D LiDAR sensors measure the distance to an object using the speed of light and provide accurate 3D information in real time. Because of these advantages, 3D LiDAR sensors are widely used in various applications such as autonomous vehicles, robotics, and geography. In particular, 3D LiDAR sensors can be very useful tools for recognizing aircraft engines [3]. However, 3D LiDAR sensors generate a 3D point cloud composed of many points. Processing this 3D point cloud is very difficult [4].

To solve this, a deep learning model called PointNet was proposed. PointNet extracts features by directly utilizing the information from each point to process the 3D point cloud [5]. The extracted features are processed through 3D convolution, and finally 3D object detection is performed.

In this paper, we propose a method to recognize aircraft engines in real time from 3D LiDAR data using PointNet. Through this, we expect to be able to monitor the status of an aircraft engine in real time and improve the safety and efficiency of the aircraft based on this [6].

This research will make an important contribution to improving the safety and efficiency of aircraft. In particular, the results of this paper will provide an important application area for jetbridge autonomous driving technology. The jetbridge is a moving passage that allows passengers to board or disembark from an aircraft. The jetbridge’s autonomous driving technology needs to accurately recognize the location of the aircraft engine. For this, the method of recognizing aircraft engines using PointNet proposed in this paper will be very useful. Through this, we expect to improve the safety and efficiency of jetbridge autonomous driving technology.

2. Related Works

In this paper, we aim to discuss previous research cases for the study of object classification techniques of semantic LiDAR data. Various 3D data have been researched for object detection using deep learning techniques, and in this paper, we will discuss three representative techniques.

2.1. PointSeg: Real-Time Semantic Segmentation Based on 3D LiDAR Point Clouds

PointSeg (real-time semantic segmentation based on 3D LiDAR point clouds) proposes a real-time end-to-end semantic segmentation method for road objects based on spherical images transformed from 3D LiDAR point clouds. This model is based on the lightweight network, SqueezeNet, and includes several improvements for enhanced accuracy [7]. This model is highly effective for real-time environmental perception. However, PointSeg may lose some positional information during the process of converting 3D LiDAR point clouds into spherical images. Additionally, the process of converting 3D LiDAR point clouds into spherical images is necessary, which could incur additional computational costs. This research makes a significant contribution to semantic segmentation using 3D LiDAR point clouds [8].

2.2. Semantic Segmentation of 3D Point Clouds Based on High-Precision Range Search Network

A deep hierarchical structure called the high-precision range search (HPRS) network was proposed, which can learn local features at increasing contextual scales. This structure was proposed to address the problem of limiting the refined pattern recognition capabilities of the recently proposed deep neural network, PointNet, which only utilizes the global features of point clouds and does not include any local features [9].

The HPRS network has developed an adaptive ball query algorithm that designs a comprehensive grouping strategy that can collect detailed local feature points compared to the common ball query algorithm. This algorithm is particularly effective when there are not enough feature points within the ball range [10].

Moreover, the HPRS network solves the disorder of points and minimizes the information loss of features by combining both max pooling and average pooling to aggregate the point features in the local area in the hierarchy, compared to the sole use of either. This network has demonstrated excellent performance on the S3DIS dataset, with a decrease in mIoU by 0.26% compared to the state-of-the-art DPFA network [11].

2.3. Learning Semantic Segmentation of Large-Scale Point Clouds with Random Sampling

The idea of learning the semantic segmentation of large-scale point clouds with random sampling addresses the problem of efficiently learning the semantic segmentation of large-scale 3D point clouds. In this research, an efficient and lightweight neural architecture called RandLA-Net is proposed, which can directly infer point-wise semantics for large-scale point clouds [12].

The core of this research is the use of random point sampling. This method is computationally and memory efficient but it has the problem of potentially discarding key features by chance. To overcome this, researchers introduced a new local feature aggregation module that effectively preserves geometric details by gradually increasing the receptive field for each 3D point. However, this method does not always work perfectly in all cases. In particular, this method may not always yield optimal results in data with complex or detailed structures [13].

Comparative experiments show that RandLA-Net can process 1 million points at once up to 200 times faster than existing approaches. Additionally, extensive experiments on five large-scale point cloud datasets, including Semantic3D, Semantic KITTI, Toronto3D, NPM3D, and S3DIS, have demonstrated the state-of-the-art semantic segmentation performance of RandLA-Net. However, the limitations of the random sampling method and performance degradation in data with complex or detailed structures remain as additional research topics [14].

2.4. AM-SegNet for Additive Manufacturing In Situ X-Ray Image Segmentation and Feature Quantification

In additive manufacturing (AM), in situ monitoring plays a crucial role in ensuring the quality and reliability of printed components. A previous study introduced AM-SegNet, a novel deep learning model designed to perform real-time segmentation and for the feature quantification of X-ray images captured during the manufacturing process. This model addresses key challenges in AM, such as detecting defects and material inconsistencies.

The core innovation of AM-SegNet lies in its segmentation capabilities, where it effectively distinguishes between different regions within X-ray images, enabling precise defect detection. Additionally, the model includes a feature quantification mechanism, which allows for the measurement of the defect size, shape, and distribution, offering a comprehensive analysis of the AM process.

By incorporating AM-SegNet, the study highlights the importance of real-time decision-making during manufacturing and demonstrates the model’s ability to process high-dimensional data quickly, thereby minimizing potential production defects.

This study demonstrates that improving segmentation techniques in complex industrial environments and real-time image processing can significantly enhance the operational outcomes [15].

3. Proposed Method

This study was conducted to implement autonomous driving technology for jetbridges by developing an algorithm for detecting and tracking aircraft engines for jetbridge path planning. Unlike with cars and ships, the jetbridge autonomous driving technology performs docking and undocking autonomously once a specific destination, the aircraft door, is determined. Therefore, accurately detecting surrounding obstacles and specific objects for docking purposes is crucial [16]. To achieve this, we based our research on 3D LiDAR data, which provide clear shape and location information, rather than image data, which contain numerous environmental features and noise [17].

3.1. System Configuration

When an aircraft is parked at the apron, numerous obstacles such as workers, vehicles, and equipment for operational tasks are present. In an environment where there are many obstacles and the location is not standardized, the autonomous operation of the boarding bridge must function properly. Therefore, it is crucial to minimize shadow areas in the 3D LiDAR data and collect high-accuracy data in real time.

As shown in Figure 1, the 3D LiDAR system was installed along the movement path and direction of the jetbridge, and the sensor and processor configurations were set up as illustrated in Figure 2. The 3D LiDAR system used a high-resolution sensor capable of detecting workers and small obstacles on the ground. A 64-channel LiDAR system was employed, and the basic specifications of the sensor are listed in

Table 1. Optical performance of 3D LiDAR systems [19].

Range (80%, 1024 @ 10 Hz mode)	170 m @ > 90% detection probability, 100 klx sunlight
Range (10%, 1024 @ 10 Hz mode)	90 m @ > 90% detection probability, 100 klx sunlight
Min~Max Range	0.5 m~150 m
Vertical Resolution	64 channels
Horizontal Resolution	1024 or 2048 (configurable)
Rotation Rate	10 or 20 Hz (configurable)
Field of View	Vertical (+22.5° to −22.5°), Horizontal 360°
Angular Accuracy	Vertical: ±0.01°, Horizontal: ±0.01°
Range Resolution	0.1 cm

.

3.2. Data Analysis

According to Table 1, the 3D LiDAR system uses a light source to measure distances, representing the distance to reflective objects as points. The point data are displayed as a 3D point cloud, with the light source rotating 360 degrees. These point cloud data, which contain specific characteristics of objects, can be used for object detection algorithms. Although data distortion may occur in foggy weather with high humidity, in this study, we applied object detection algorithms that are not affected by the time of day or night or weather conditions when the target objects are within 20 m [19].

However, the collected data capture the entire surrounding environment, including the ground of the aircraft apron, people, cars, and other obstacles, as shown in Figure 3. Therefore, unnecessary information must be filtered. In this study, ground data filtering was performed to specify the regions required for the algorithm. Ground and distorted data filtering were conducted using the 3D RANSAC technique. By adjusting the field of view of the sensor data, could we processed only the regions requiring specific object recognition without processing the entire scanning dataset, ensuring computational speed. Additionally, the spatial position between the jetbridge and aircraft was fixed, and cylindrical objects were identified as aircraft engines, ensuring a high detection rate compared to other studies due to the specific nature of the data training [20].

In this study, the 3D RANSAC (random sample consensus) algorithm was used to filter ground data during the preprocessing of 3D LiDAR data. The 3D RANSAC method is a robust technique for efficiently estimating models in noisy data. It effectively extracts ground data and removes unnecessary points, enhancing the accuracy of the semantic segmentation process. The basic operation of RANSAC is as follows:

• Random sampling: Select a random subset of samples from the data set.
• Model estimation: Estimate a model based on the selected samples.
• Consensus set formation: Identify data points that fit the estimated model to form a consensus set.
• Model evaluation: Evaluate the size of the consensus set and select the model that includes the most data points.
• Iteration: Repeat the above process for a predefined number of iterations to find the optimal model.

Particularly, 3D RANSAC performs these steps in a 3D space to estimate various models such as planes, spheres, and cylinders. However, the shapes must be well-defined, and it can be challenging to distinguish and classify small objects or those with non-standard features. This study utilized 3D RANSAC to extract ground data in the aircraft apron environment. As shown in Figure 3, ground data are collected with noise from 3D LiDAR scan data, and accurately extracting these ground points is crucial. The process of filtering ground data using 3D RANSAC in this study was as follows: [21].

• Initial data collection: Collect raw point cloud data scanned with a 3D LiDAR sensor. There data comprise points from various objects, including the ground.
• Plane model estimation: Randomly select samples from the collected point cloud and estimate a plane model. The plane model, representing the ground, is estimated using the 3D RANSAC algorithm.
• Ground point identification: Identify points that fit the estimated plane model to form a consensus set. During this process, a distance threshold is set to distinguish ground points from non-ground points.
• Ground data filtering: Filter out points that do not belong to the consensus set, thus obtaining a clean point cloud containing only ground points.
• Post-processing: Perform additional preprocessing tasks using the filtered ground data. For instance, normalize the height information for the ground or separate obstacle data on the ground.

The RANSAC algorithm does not re-predict the model from the sum of the current errors but instead creates an approximate model directly from the predicted model. This enables high-speed processing because it reduces the time required to compare the errors related to the model predicted from a small number of observation points. However, the algorithm is sensitive to noise because the errors are not compensated. Unlike the least squares method, which uses the entire dataset, the RANSAC method estimates the model parameters by applying different observation values to each subset of the data. For the subsampled data, the region of each 3D LiDAR channel is divided by ${\lambda }_k$, and ${\lambda }_0$ = 0.2 m is defined. Furthermore, ${\lambda }_k$ can be calculated using Equations (1)–(3):

$$a_0={\tan }^{-1}({\displaystyle \frac{{\lambda }_k}{(h_{offset}+400 \mathrm{mm})}})$$

(1)

$${\lambda }_k=h\cdot \tan (a_0+k\cdot \eta \cdot \Delta a), \left\{k:1,\dots \dots ,N\right\}$$

(2)

$$N={\displaystyle \frac{a_N-a_0}{\eta \cdot \Delta a}}, \eta =2, \Delta a=0.2$$

(3)

In conclusion, the 3D RANSAC algorithm is a powerful tool for filtering ground noise in 3D LiDAR data, as shown in Figure 4. This technique significantly enhances the path planning and environmental awareness capabilities of autonomous vehicles by ensuring that only relevant ground data are processed. [22].

3.3. Semantic Segmentation Based on PointNet

Semantic segmentation involves classifying each point in a point cloud into a predefined category. Traditional methods rely on hand-crafted features and complex pipelines, which often suffer from limited scalability and accuracy. PointNet, introduced by Qi et al. in 2017, revolutionized this task by directly operating on unordered point sets using deep learning techniques [23].

As illustrated in Figure 5, a point cloud is a collection of points in a 3D space, where each point is defined by its coordinates (x, y, z) and additional attributes such as color or intensity. Unlike structured data such as images, point clouds are unstructured, making them unsuitable for direct processing with conventional convolutional neural networks (CNNs) [24]. Therefore, PointNet processes point clouds by learning spatial features directly from raw point data [25]. As shown in Figure 6, the algorithm is structured with three main components in its architecture.

• Input and embedding layers: PointNet takes N points as the inputs and maps each point to a higher-dimensional space through shared multi-layer perceptrons (MLPs). This step ensures that each point is represented by a feature vector.
• Symmetric function for aggregation: To handle the unordered nature of point clouds, PointNet employs a symmetric function, specifically max pooling, to aggregate features from all points. This function ensures that the network is invariant to the permutation of input points.
• Output layers: The aggregated global feature vector is processed by further MLPs to produce the final output, which can be classification scores for object recognition or per point scores for semantic segmentation.

Input and embedding layers: Each point in the point cloud is represented by a 3D coordinate (x, y, z). The embedding layer uses shared MLPs to map each point to a higher-dimensional feature space. This shared MLP is represented as Equation (4), where $x_i$ is the input point and $h(x_i)$ is the higher-dimensional feature vector:

$$h(x_i)=MLP(x_i)$$

(4)

Symmetric function for aggregation: PointNet employs max pooling as a symmetric function to aggregate features from all points. Here, $g(X)$ is the global feature vector representing the entire point cloud, ensuring invariance to point permutations:

$$g(X)=\max \{h(x_1),h(x_2),\dots ,h(x_N)\}$$

(5)

Output layers: The global feature vector $g(X)$ is passed through additional MLPs to produce the final output. For semantic segmentation, the output is a per point classification score. The output layer can be represented as Equation (6), where ⊕ denotes concatenation of the global and local features:

$$y_i=MLP(g(X)\oplus h(x_i))$$

(6)

Using the previously calculated $y_i$, data augmentation techniques were performed to improve the robustness of the dataset generated for PointNet-based semantic segmentation.

Improved robustness: By exposing the model to various transformations of the data, the model becomes more robust to variations in the input.

Better generalization: Data augmentation increases the diversity of the training set, which helps in improving the model’s generalization to unseen data.

Reduced overfitting: Augmenting the data can reduce overfitting by preventing the model from memorizing the training data, encouraging it to learn more general features.

These data augmentation techniques were carried out through three theoretical processes—random rotation, scaling, and jittering [22].

Training PointNet for semantic segmentation involves using labeled point cloud datasets. Data augmentation techniques such as random rotation, scaling, and jittering are employed to improve the network’s robustness and generalization.

The loss function for semantic segmentation is typically the cross-entropy loss, which measures the difference between predicted and true labels for each point. The network is trained to minimize this loss, ensuring the accurate classification of each point in the point cloud:

$$L=-{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N{\displaystyle \sum _{c=1}^C{y}_{i,c}\log ({\hat{y}}_{i,c})}}$$

(7)

where $N$ is the number of points, $C$ is the number of classes, and ${\hat{y}}_{i,c}$ is the predicted probability for class $c$ for point $i$.

The common metrics for evaluating semantic segmentation performance include the mean intersection over union (mIoU), accuracy, and precision. These metrics assess the network’s ability to correctly classify points across different categories. Here, $P_c$ and $G_c$ are the predicted and ground truth sets of points for class $c$, respectively:

$$mIoU={\displaystyle \frac{1}{C}}{\displaystyle \sum _{c=1}^C\frac{\left|P_c\cap G_c\right|}{\left|P_c\cup G_c\right|}}$$

(8)

In this paper, the number of classes is defined as two, and the types of aircraft engines to be detected are categorized. The aircraft engines are broadly classified into Boeing series engines and Airbus series engines. As shown in Figure 7, scanning data and distance data from various positions for each type of aircraft engine were collected for training. However, it was not possible to collect data for all Boeing and Airbus aircraft. Instead, the engine scanning data for the Airbus A330 and Boeing 787 models were used as representative examples.

4. Experiments and Results

The aircraft engine detection tests were conducted by distinguishing between the target aircraft and the time of day. The target aircraft were broadly categorized into two types, and as with the previously trained dataset, tests were performed on the B787 and A330 models. These tests were conducted both in the evening and during the day, as the amount of light reflected on each aircraft’s exterior varied depending on the time of day. This variation led to relative distortions in the data collected from areas where a significant amount of sunlight was reflected. As shown in Table 1, the variance error in the data caused by these distortions showed an error range of more than 0.5 cm, deviating from the specifications provided by the basic system. To address this, various test scenarios were introduced to differentiate between the technical limitations of the sensor system and the algorithm under different conditions.

Although the B787 and A330 models were from different airlines, the engine areas of both aircraft were white, meaning there was no difference due to the basic engine color. The first test was conducted from 10:00 to 11:00 in the winter, and the second test was performed from 21:00 to 23:00 during the same season. Figure 8 shows a photo of the aircraft fuselage taken from the top of the boarding bridge during the test, with light and shadows appearing as shown in the image. Additionally, as shown in Figure 9, the result data include the measured straight-line distance from the center of the rotating part of the boarding bridge to the nearest point of the aircraft engine.

In the first experiment, as shown in Figure 9, we conducted a test for aircraft engine data segmentation and distance measurements by moving the jetbridge to approximately 3.5 m in front of the aircraft engine. The actual test was performed 10 times, and with the previously collected dataset, it was repeated 50 times. The data sampling rate was set to 10 Hz. As a result, the difference in the outcomes between the B787 and A330 aircraft was not significant. For the B787, the segmentation results showed an accuracy of over 81% when the distance from the jetbridge was between 10 m and 4 m, although the accuracy dropped to 88% when the distance was between 4 m and 3.5 m. Consequently, the mIoU for the B787 in the first experiment was recorded at 0.805. For the A330, there was no significant difference across all distances, and the mIoU was recorded at 0.82. Thus, the semantic segmentation mIoU for the two aircraft types in the first test scenario was 0.8125.

Additionally, we measured the distance error between the segmented data classes and the jetbridge. The measurements were compared at distances of 10 m, 5 m, and 3 m using a high-precision radar distance measuring sensor from SICK. The purpose of measuring the distance error was to compare the performance of the semantic segmentation method under external environmental changes, such as external vibrations affecting the sensor system or interference in the sensor data field of view during scenarios where the target object is stationary and the sensor system is moving.

As shown in Figure 10, changes in cabin heading angles affect the measured distance data between the aircraft engine and the jetbridge. When the driving trajectory is straight, the error between the reference data and the measured data is close to zero. However, in sections where the jetbridge’s heading changes, vibrations affecting the sensor system result in data errors of up to 0.53 m.

In fact, as shown in the upper graph of Figure 10, the high frequency of vibrations caused by angle changes significantly affects the sensors. The correlation between the two graphs indicates that the greater the distance between the aircraft engine and the jetbridge, the larger the measurement data error due to the same system vibrations.

As shown in Figure 11, the results of the nighttime tests indicated that while the segmentation techniques used for object detection had some impact, the actual distance measurement yielded the same results. Ultimately, to accurately measure the distance to the aircraft after applying the segmentation technique, the inherent vibration characteristics of the jetbridge must be reduced.

5. Conclusions

This study investigated a method for detecting the locations of aircraft engines to develop autonomous driving technology for jetbridges. Initially, a 3D LiDAR scan was conducted in an apron environment where actual aircraft were parked. The scanned data underwent a preprocessing phase to filter out unnecessary information by utilizing 3D RANSAC technology and setting specific regions of interest (ROIs) to extract only the necessary areas. Real aircraft data were then extracted and used for training to apply the PointNet algorithm.

Nevertheless, this study achieved a mean intersection over union (mIoU) of 81.25% for aircraft engine detection using semantic classification methods, as shown in

Table 2. Comparative of semantic segmentation models.

Model	Dataset	Mean IoU
PointNet	-	81.25%
PointNet [26]	ShapeNet	83.7%
PointNet++ [27]	ShapeNet	85.1%
DGCNN [28]	ShapeNet	85.2%
PointConv [29]	ShapeNet	85.7%
PointSIFT [30]	ShapeNet	84.4%
Point TF [31]	S3DIS Area 5	70.4%
PointNeXt [32]	S3DIS (6-fold)	68.1%

, even under challenging conditions. This performance is by no means low compared to other studies. It is particularly noteworthy that the study addressed complexity and environmental challenges by utilizing a proprietary dataset. However, since this study focused only on two types of aircraft, it highlights the need for more datasets to classify engine data for all aircraft operating at the airport. This issue will be addressed through further research and testing.

In addition, continuous efforts will be made to reduce physical noise and distortion in the sensor data to improve the system’s reliability. Enhancing the robustness of the detection system against various external disturbances, such as vibrations and lighting variations, will be crucial for the successful deployment of this technology in real-world applications.

Our future research will also explore the integration of this detection method with other autonomous systems within the airport environment. Such integration aims to create a more comprehensive and reliable autonomous docking system for jetbridges, thereby improving the efficiency and safety of airport operations. Furthermore, collaboration with airport authorities and aircraft manufacturers will be pursued to refine and validate the technology in diverse operational conditions.

Given the frequent occurrence of sea fog and the prevalence of snow and rain in certain airport environments, we are considering combining RGB image data with 3D LiDAR information in future studies. However, to optimize the sensor configuration, our future research will focus primarily on 3D LiDAR, utilizing more advanced deep learning networks to continue improving the aircraft engine localization process.

The implications of this research extend beyond jetbridge operations. The methodologies and findings can contribute to the broader fields of autonomous vehicles and robotics, particularly in environments that require high precision and reliability. By addressing the current limitations and continuously enhancing the system, this research aims to make significant progress in the development of autonomous technologies for aviation and other sectors.