Research Status and Prospect of the Key Technologies for Environment Perception of Intelligent Excavators

Abstract

With the urgent need of the industry and the continuous development of artificial intelligence, research into intelligent excavators has achieved certain progress. However, intelligent excavators often face strong vibrations, dense dust, and complex objectives. These have brought severe challenges to environmental perception, and are important research difficulties that must be overcome in realizing the practical engineering applications of intelligent excavators. Many researchers have studied these problems in reducing vibration and dust noise for light detection and ranging (LiDAR) scanners, multi-sensor information fusion, and the segmentation and recognition of 3D scenes. This paper reviews the research status of these key technologies and discusses their development trends.

1. Introduction

Excavators are widely used in engineering construction, emergency rescue, energy extraction, and other important fields, which is a powerful guarantee to promote the development of the national economy. They play an important role in improving construction efficiency, reducing labor intensity, guaranteeing project quality, and reducing operational risks [1,2]. In various construction sites, excavators perform 65% to 70% of earthmoving work. Large excavation is mainly used in mining, large-scale infrastructure projects, and other fields. Medium excavation is mainly used in real estate, urban construction, transportation construction, and other fields. Small excavation is mainly used in small earth and rock works, pavement repair, and other fields. There are requirements for construction accuracy, efficiency, energy consumption, and other aspects of continuous improvement, as well as the serious challenges posed by the special working environment [3,4,5]. Traditional driver-operated excavators can no longer meet the requirements of projects, and there are mainly the following problems, as shown in Figure 1.

Figure 1. Common problems of traditional excavators in extreme working conditions.

• Low construction accuracy, low work efficiency, and high energy consumption. Conventional excavators are affected by human factors, and the accuracy of operation cannot be guaranteed. The low full bucket rate reduces the mining efficiency and energy utilization rate and requires a lot of energy consumption.
• Severe challenges are brought by special working environments. In the process of underwater operation, space development, emergency rescue, etc., the traditional excavator is restricted by the uncertainty and safety of the working environment, so it is difficult to realize normal operation.
• Frequent failures and poor safety. Some operators are accustomed to digging deeply with brute force, resulting in strong impact loads on the bucket. The whole machine vibrates violently, and accidents such as broken shafts and pins and overturning of the lifting arm are likely to happen. When the bucket is loading, collisions happen easily due to human error or limited vision.

To address the challenges faced by traditional excavators, advanced technologies such as artificial intelligence, robotics, and automatic control are applied to upgrade conventional excavators. The goal is to develop an intelligent excavator capable of autonomous operation. This system integrates functions such as environmental perception, excavation trajectory planning, and path planning. Such an intelligent excavator has significant practical implications, offering considerable benefits both economically and socially.

In addition to the basic system of the traditional excavator, the intelligent excavator also has an environmental perception system, pose detection system, walking path planning system, excavation trajectory planning system, and intelligent control system, as shown in Figure 2.

Figure 2. Main systems of intelligent excavators.

The environmental perception system uses visual sensors such as cameras and light detection and ranging scanners to perceive the mining target, loading vehicle, road conditions, and other working environment information, and then synchronizes and fuses the relevant information, and uses relevant algorithms to model, identify and segment the working target; the position and attitude detection system uses Global Positioning System (GPS), inertial navigation module, and other sensors to measure the position and attitude information of excavator car body and bucket in real time; the walking path planning system can plan the walking path of the excavator chassis in real time according to the road condition information and the excavator position information to ensure efficient, stable and safe walking. The excavation and loading trajectory planning system performs path planning for both excavation and loading tasks based on environmental perception data, including excavation and loading target information, as well as bucket pose data. The intelligent control system receives the relevant information of other systems in real time, and adopts the robust adaptive control algorithm to control the intelligent excavator. Among them, the environmental perception system is the premise and basis of the walking path planning system, the excavation and loading trajectory planning system, and the intelligent control system. Its performance directly determines the realization level of the intelligent excavator.

1.1. Literature Review Methodology

This survey reviews the latest research progress of intelligent excavators and key environmental perception technologies for intelligent excavators at home and abroad. After screening and analysis, a total of 126 published papers were selected. As can be seen from the literature in the list, research on intelligent excavator technology has received more and more attention. In this survey, we conducted a systematic literature search through multiple academic databases, mainly using platforms including Google Scholar, IEEE Xplore, and Web of Science. To accurately locate the research literature in the field of intelligent excavators, we used a search method of filtering keywords, such as “intelligent excavators”, “environmental perception”, “Light Detection and Ranging (LiDAR)”, “vibration reduction technology”, “dust reduction technology”, “multi-sensor information fusion”, “segmentation and recognition of 3D scenes”, “visual perception”, “deep learning”, “Robot perception”, etc. At the same time, to understand the development of key environmental perception technologies for intelligent excavators, we screened the literature published between 1999 and 2024 and focused on the research results of the last five years. Subsequently, the screening results were filtered based on relevance as well as references. Finally, based on the research topics and technical content, we categorized all the selected literature into several main categories. This classification allows for a systematic summary of the development trends in the environmental perception technologies for intelligent excavators.

1.2. Research Status of Intelligent Excavators

Since intelligent excavators have the characteristics of high scientific and technological content, clear application objects, and good market prospects, since Europe and the United States began to develop the first intelligent excavator in 1995, research on intelligent excavators has been widely valued by many universities and scientific research institutions at home and abroad. After more than 20 years of development, certain research results have been achieved. The intelligent excavator developed by the University of Tsukuba in Japan uses a variety of sensors, as shown in Figure 3.

Figure 3. Various sensors on intelligent excavators.

As can be seen from the analysis of Table 1, the research of intelligent excavators has made certain progress, and rich research results have been formed in trajectory planning, obstacle avoidance, excavation trajectory planning, path planning, automatic control, and other aspects. Most of these research results are oriented to specific ideal working conditions or laboratory environments. However, intelligent excavator operation is often faced with strong vibrations, high dust levels, and complex and dynamic working object structure, which brings severe challenges to the intelligent excavator operating environmental perception.

Table 1. Research status of intelligent excavators at home and abroad.

Time	Research Institution	Major Contribution	Literature
1995	Lancaster University, UK	Using a potentiometer, axis tilt sensor, optical laser ranging sensor global positioning system, and other equipment, it can efficiently complete the independent excavation of long straight ditches.	[6,7,8]
1999	Carnegie Mellon University	Two laser rangefinders are used to scan the working space, identify the excavation surface, trucks, and obstacles, and combine the on-board computer and positioning system to carry out the excavation motion planning and dumping motion planning, which can realize the autonomous operation of slope excavation.	[9,10]
2000	University of Sydney	By using a laser rangefinder to scan the working environment, the functions of task decomposition, state monitoring, and path planning are realized, and the automatic mining operation in simple working conditions is realized.	[11,12]
2006	Commonwealth Scientific and Industrial Organisation of Australia	SICK LMS LiDAR is used to scan the working environment and construct a 3D topographic map to describe the working environment and excavation surface. The operation time is basically the same as the manual operation time, and the energy consumption is less.	[13]
2009	Northeastern University	The binocular CCD sensor is used to collect environmental information such as road, bucket and target, which lays a foundation for the autonomous walking and operation of the excavator.	[14]
2012	University of Tsukuba and Ministry of Land and Resources, Japan	The use of pressure flow sensors, tilt sensors, gyroscopes, GPS, binocular cameras, LiDAR, and other sensors can sense the working environment independently plan obstacle avoidance, and initially realize the independent operation of the excavator.	[15]
2012	Central South University, Sanhe Intelligent	The kinematics and dynamics model of the excavator are established, and the cerebellar model neural network control method is used to realize the real-time and accurate control of the intelligent excavator.	[16]
2013	Sungkyunkwan University, South Korea	An intelligent mining system for cluster control of construction machinery is developed, which is mainly composed of a task planning system, task execution system and man–machine interface system. The system has a high degree of intelligence and realizes the preliminary independent mining process.	[17]
2018	Dalian University of Technology	The intelligent excavator prototype is developed, and the dynamic excavation resistance prediction model is constructed to provide theoretical support for the intelligent excavator to minimize energy consumption.	[18]
2019	Zhejiang University	The experimental mining robot is developed, which can realize the functions of trajectory planning, trajectory tracking and remote control, and has a certain degree of automation.	[19]
2021	Harbin Institute of Technology	Computer vision was introduced into the identification of the working cycle stage of the excavator, and the depth vision target detector was established by using the YoloV2 algorithm to locate and classify the three feature parts (bucket, arm joint and body).	[20]
2022	Dalian University of Technology	The environmental perception algorithm is designed to process the point cloud data, carry out multi-level ground extraction for the point cloud and identify the mining loading target based on feature computing, and help the excavator to determine the location of the material pile and truck in the working environment, so as to realize the perception of the working environment of the excavator.	[21]
2024	Yanshan University	A time-convolutional recurrent neural network (TCRNN) combining stackable extended convolution with an attention-based Sequence-to-sequence (Seq2Seq) module is proposed to accurately predict excavation forces and a data-driven excavation trajectory planning framework based on TCRNN is established to improve operational performance in autonomous excavation scenarios.	[22]

• The operation of intelligent excavators is often accompanied by strong vibrations, which will cause linear deviation and attitude angle change of the light detection and ranging scanner, thus affecting the detection accuracy of the LiDAR scanner. In addition, strong vibrations will cause fatigue and wear of the precision components inside the light detection and ranging scanner, affect the service life of the equipment, and even cause damage to the equipment. Therefore, it is necessary to study the vibration characteristics of the excavator, design a reasonable vibration-damping platform, and measure the light detection and ranging scanner displacement and attitude data caused by vibration, as well as modify the data through registration, filtering, and other algorithms as far as possible to eliminate the impact of vibration on the measurement accuracy of the light detection and ranging scanner.
• The intelligent excavator will produce a large amount of dust during the excavation and loading process. When the dust concentration in the air reaches a certain degree, it will seriously affect the effective measurement accuracy of the Light Detection and Ranging scanner on the working environment and the working target, and more serious cases will appear blocking phenomenon. To study the acquisition and noise reduction processing of three-dimensional point cloud data of excavator operating environment under dusty conditions, and to reduce the influence of dust on LiDAR measurement data. It is of great significance to improve the sensing ability of the operating environment of intelligent excavators.
• The excavation targets and loading vehicles faced by intelligent excavators in the construction process are often complex in structure, with diverse details such as shape, color, texture, etc., and outdoor operations susceptible to the impact of light, weather, etc. These will seriously affect the environmental perception effect of intelligent excavators. Multi-sensor information fusion technology is of great significance in improving the working environmental perception ability of intelligent excavators under complex working conditions. In environmental perception tasks, a single sensor faces many shortcomings, and LiDAR can obtain accurate position information. But the collected point cloud data lack important information such as color and texture, and the point cloud data are relatively sparse. In complex operating environments, this will affect the ability to identify fine target features. The camera can acquire rich color and texture information, and the resolution of the image data is relatively high. However, since the camera is a passive measurement sensor, it is less robust, seriously affected by the environment, and unable to directly obtain depth information. Compared with the traditional single sensor, the multi-sensor information fusion technology can obtain more comprehensive, accurate, and robust operating environment information. Therefore, it plays a crucial role in improving the operating environmental perception ability of intelligent excavators under complex conditions.
• The segmentation and recognition of the three-dimensional operation scene of the intelligent excavator (as shown in Figure 4) is the basis for realizing the intelligence of the excavator. Since the three-dimensional point cloud can accurately record the position and geometric information of the measured target, it has advantages that cannot be compared with two-dimensional (2D) images. This important information is the basis for intelligent excavators to carry out tasks such as path planning and trajectory planning, so the perception data of the operating scene is mainly based on the three-dimensional point cloud. Compared with two-dimensional images, the data structure of a three-dimensional point cloud is more complex and disordered, and the intelligent excavator operation scene is complex and dynamic, which brings great challenges to segmentation and recognition. Therefore, the research of three-dimensional operation scene segmentation and recognition technology is of great significance to the realization of excavator intelligence.
  

  Figure 4. Segmentation and recognition of the excavator working targets.

In summary, a review and analysis of the current status of key environmental perception technologies, such as LiDAR vibration reduction, LiDAR dust noise reduction, multi-sensor information fusion, and segmentation and recognition of 3D operating scenes, is provided both domestically and internationally. Their research priorities and technological development trends are discussed, which will help improve the environmental perception ability of intelligent excavators. It is of great practical significance to improve the practical application level of intelligent excavators.

2. Research Status of Key Technology for Environmental Perception of Intelligent Excavators

Intelligent excavator operating environmental perception refers to the process of intelligent excavators using the sensor carried by itself to obtain the excavator operating environment information, and to process, interpret, and assign meaning to the information. The sensors carried by the intelligent excavator itself include LiDAR, cameras, a GPS global positioning system, an inertial navigation system and so on. Operating environmental perception is the premise of realizing automatic digging, loading, path planning, control, and other functions, and is the key to realizing an intelligent excavator. However, the complex working conditions faced by the excavator bring severe challenges to the environmental perception, and it is urgently needed to conduct an in-depth discussion on the key technologies of intelligent excavator environmental perception facing complex working conditions. As shown in Figure 5, this paper mainly researches key technologies such as LiDAR vibration reduction, LiDAR dust noise reduction, multi-sensor information fusion, and intelligent segmentation and recognition of excavator operating targets under complex working conditions of intelligent excavators.

Figure 5. Key technologies of environmental perception under complex working conditions.

2.1. Research Status of LiDAR Vibration Reduction Technology

To collect excavator vibration data in real time, Chen et al. [23] designed an excavator vibration monitoring recorder with an STM32F103 microcontroller as the core. Huang et al. [24] used wavelet packet band energy decomposition and Hilbert–Huang transform, respectively, to study the noise reduction methods of vibration signals in the process of excavator shoveling. The research shows that the reconstructed vibration signals based on the Hilbert–Huang transform can better reflect the essential characteristics of real vibration signals. Zhou et al. [25] established the finite element model of the excavator cab mechanism, conducted modal analysis on the noise and vibration characteristics of the excavator cab, and optimized the cab structure based on the analysis results, effectively reducing the vibration and noise. Zhang et al. [26] studied the vibration characteristics of the lifting mechanism of excavators and obtained the reasonable vibration range of the lifting mechanism through experimental analysis. The above studies are all aimed at the vibration analysis of the excavator itself. However, compared with traditional excavators, intelligent excavators need to use LiDAR for environmental perception, and vibration has a greater impact on environmental perception accuracy. Therefore, the research of LiDAR vibration reduction technology is of great importance to intelligent excavators. At present, research on the LiDAR vibration reduction technology of intelligent excavators is still limited. However, there is a solid research foundation in related fields such as unmanned and aerospace vehicles, which can play a certain role in the research of LiDAR vibration reduction for intelligent excavators. These studies are mainly divided into two types.

• In terms of software, vibration theory analysis or simulation studies of the LiDAR detection process under vibration conditions are conducted. A response model is established between the LiDAR measurement data and vibrations, enabling compensation and correction of measurement data affected by vibration.
• In terms of hardware, the main vibration parameters, such as amplitude and frequency, are measured through experiments, and the vibration patterns are analyzed. Then, a vibration-damping platform is designed, and an optimal installation position is selected, effectively reducing the impact of vibration on the LiDAR.

In terms of software, Ma et al. [27] proposed a simulation method to evaluate the impact of vibration error on the scanning accuracy of LiDAR. The research shows that the vibration of LiDAR platforms seriously reduces the position accuracy of the point cloud, and the position accuracy error caused by vibration is relatively complex, and cannot be fitted with a simple polynomial function. Ma et al. [28] proposed a method based on trajectory dual-detector interferometry to estimate and compensate LiDAR phase errors caused by platform vibration. Hu et al. [29] proposed a new LiDAR vibration estimation method to solve the problem of large vibration estimation error under the conditions of wide azimuth beam and low SNR. Cui et al. [30] analyzed the working principle and corresponding vibration of vehicle-mounted three-dimensional scanning LiDAR to solve the problem of ranging accuracy error caused by road impact and vibration, established a mathematical model between shock and vibration excitation and circuit vibration and noise response, and proposed a vibration and noise elimination method based on wavelet analysis. Su et al. [31] studied the imaging ambiguity caused by airborne LiDAR caused by residual vibration after high-frequency vibration is suppressed by a vibration-damping platform, established a stochastic vibration fuzzy theoretical model, analyzed the mechanism of crosstalk effect between pixels, and studied the relationship between exposure time, pixel crosstalk radius and relative ranging error coefficient. The optimal exposure time satisfying the resolution requirement is obtained, and the ranging performance of the system is improved. Hong et al. [32,33] conducted a focused study on how vibration affects the measurement accuracy of LiDAR. They examined the influence of both platform angular vibration and linear vibration, establishing a mathematical model for angular vibration. Their findings revealed that vertical vibrations have a more significant impact on image quality compared to vibrations along the horizontal route. In theoretical analysis, simulation, or modeling, the above methods all set relatively ideal assumptions, which are difficult to meet in practical engineering applications.

In terms of hardware, Song et al. [34] designed a LiDAR vibration-damping platform by comprehensively adopting an aluminum alloy frame, an optical platform, and a wire rope shock absorber to solve the vibration problem of vehicle-mounted LiDAR. Through experimental verification, the platform can effectively reduce the impact of vibration on the accuracy of LiDAR measurement. Veprik et al. [35] established an anti-vibration system model to solve the problem that optically sensitive parts were affected by vibration under harsh conditions. They selected wire rope shock absorbers for reasonable structural layout and design, which effectively suppressed the influence of vibration. In addition, to solve the vibration problem of key electronic equipment in harsh environments, they optimized the selection of elastic and damping shock absorbers and selected the best installation position of the vibration absorber according to the vibration response characteristics, thereby providing effective vibration protection for critical electronic components [36]. Aiming at the impact of vibration on the measurement accuracy of airborne vision sensors, Wang et al. [37] comprehensively compared the vibration damping performance and selected wire rope shock absorber with nonlinear hysteresis characteristics. They reasonably designed the shock absorber layout to avoid vibration coupling, effectively isolated the onboard vibration input, and improved the measurement accuracy and stability of the optoelectronic platform. Chen et al. [38] designed a reasonable layout of a metal wire rope shock absorber to solve the problem of vibration of vehicle-mounted LiDAR, designed a vibration reduction scheme for the optical platform, and effectively improved the measurement stability and accuracy. Li et al. [39], aiming at the imaging quality problem caused by the vibration of the airborne photoelectric detection platform, determined the installation position of the photoelectric detection platform by studying the carrier vibration environment. They established the vibration model of the vibration absorber of the photoelectric detection platform with the support of vibration theory and designed the vibration absorber. The experiment proved that the vibration absorber can effectively reduce the vibration intensity of the platform.

Try et al. [40] proposed a sensing device that combines a single six-axis IMU with a beam structure to achieve the measurement of tiny vibrations. Sun et al. [41] introduced a method for measuring ship ice-induced vibrations based on a miniature inertial sensing device. This method effectively collects acceleration data and enables real-time data storage through the integration of an internal inertial sensor module. Kaswekar et al. [42] proposed a comprehensive motion measurement method for real-time estimation of vibrations in the critical telescope of the stratospheric infrared observatory. Distributed inertial sensors were used to capture motion signals, while strain gauges also assisted in these measurements. The sensor fusion method developed consisted of a continuous–discrete extended Kalman filter. The above methods have a certain effect on vibration reduction, but it is difficult to quantify the actual vibration situation.

Table 2 summarizes the literature [23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,43,44,45,46] and analyzes the research status of LiDAR vibration reduction technology.

Table 2. Research status of LiDAR vibration reduction technology.

Literature	Keywords	Major Contribution	Time
[23]	Vibration monitoring recorder	To collect real-time excavator vibration data, an excavator vibration monitoring recorder is designed.	2019
[24]	Vibration signal noise reduction	Wavelet packet frequency band energy decomposition and Hilbert–Huang transform are used to study the vibration signal denoising method in the process of excavator shoveling.	2013
[25]	Cab vibration characteristics	The modal analysis of the noise and vibration characteristics of the excavator cab is carried out, and the cab structure is optimized to reduce the vibration and noise effectively.	2013
[26]	Vibration characteristics of the lifting mechanism	The vibration characteristics of the lifting mechanism of the excavator are studied, and the reasonable vibration range of the lifting mechanism is obtained through experimental analysis.	2017
[27]	Simulation and evaluation	The simulation method is used to evaluate the influence of vibration error on the scanning accuracy of LiDAR.	2012
[28]	Estimation and compensation	Estimation of phase error of LiDAR caused by compensated vibration based on trajectory two-detector interferometry.	2014
[29]	Vibration estimation	To solve the problem of large vibration estimation error under the condition of a wide azimuth beam and low SNR.	2016
[30]	Mathematical model, noise cancellation	The mathematical model of vibration and noise is established, and the method of vibration and noise elimination based on wavelet analysis is proposed.	2017
[31]	Vibration blur, exposure time	A random vibration fuzzy theoretical model is established to obtain the optimal exposure time satisfying the resolution.	2011
[32,33]	Angular vibration, line vibration	The vibration is divided into angular vibration and linear vibration, and the mathematical model of angular vibration and linear vibration is established.	2012
[34]	Damping table	An aluminum alloy frame, an optical platform, and a steel wire rope damper are used to design the radar damping table.	2013
[35]	Wire rope shock absorber	The model of the anti-vibration system is established, and the structure layout and design of the steel wire rope shock absorber are selected.	2001
[36]	Damping table	The elastic damping damper and installation position of the damping table are selected according to the vibration response characteristics.	2003
[37]	Shock absorber	The shock absorber with nonlinear hysteresis characteristics is selected, and the layout of the shock absorber is designed.	2011
[38]	Wire rope shock absorber	A metal wire rope shock absorber is selected, a reasonable layout is designed, and a vibration reduction scheme for the optical platform is also designed.	2011
[39]	Damping table	The installation position of the platform is determined, the vibration model is established, and the damping table is designed.	2014
[43]	Passive vibration isolator	Through mathematical modeling, dynamic simulation, and response surface analysis, the structure parameters of optimal rod length, assembly angle, spring stiffness, and damping coefficient are selected, and a passive isolator mechanism based on a second-order spring damping vibration system is constructed.	2024
[44]	Rubber shock absorber	According to the vibration-damping design requirements, the rubber shock absorber was designed, and the finite element simulation method of rubber structural parts was summarized. According to the simulation requirements, the material parameters of rubber samples were tested, and the superelastic model parameters of rubber materials and the viscoelastic results of materials were obtained through the test.	2020
[45]	Multimodal vibration suppression	According to the characteristic design and equivalent principle of the crawler chassis of hydraulic excavators, the multi-mode vibration suppression algorithm is adopted to realize the multi-mode vibration suppression of the crawler chassis of hydraulic excavators.	2024
[46]	Finite element; substructure	The substructure power flow method is comprehensively used to study the influence of vibration characteristics of the rotating platform on cab noise and improve the dynamic solving efficiency to accelerate the analysis and optimization process of excavator noise and vibration performance.	2021

2.2. Research Status of LiDAR Dust Noise Reduction Technology

Stentz et al. [9] studied the problem of dust affecting loading target recognition in the automatic loading system of excavators. Based on the final echo technology, Anthony et al. improved the technology and distinguished the target and dust signals by extracting the edge of the final pulse, effectively improving the loading target recognition effect under the condition of dust. Phillips et al. [47] summarized the dust influence rules of common LiDAR under different measuring distances, different dust concentrations, and different reflectance of measured objects, aiming at the problems of terrain scanning reconstruction and loading target identification faced by excavators in the mine operating environment affected by dust. It provides a reference for how to make better use of LiDAR for distance measurement under dust conditions. At present, the research on laser radar dust noise reduction technology for intelligent excavators is still relatively lacking, but the relevant research in other fields has certain reference significance for the development of intelligent excavators.

Laux et al. [48] conducted a detailed experimental analysis to address the issue of adverse environmental conditions, such as smoke and dust, which negatively impact the performance of sensors like radar and cameras. Their study demonstrated that fusing two-dimensional color images with three-dimensional point cloud registration can effectively reduce noise and enhance environmental perception capabilities. Cao et al. [49] calculated the noise level according to the LiDAR equation, and verified the background noise equation by using the observed LiDAR data and the simulated signal, indicating that the equation provides a reasonable method for measuring the background noise level of LiDAR data. Cheng et al. [50] proposed a LiDAR denoising method based on differential optical path, derived and verified the mathematical model, and simulated the signal-to-noise ratio based on this method. Goodin et al. [51] established a probabilistic model of the interaction between LiDAR and dust and verified the model through simulation and experiment. Ryde et al. [52] developed a method to measure the level of suspended dust by determining the transmission coefficient and used it for quantitative evaluation of the environmental perception performance of LiDAR. The reliability and accuracy of target detection were evaluated by conducting measurement experiments with sensors in a controlled environment. The above methods focus on evaluating the impact of smoke, dust, etc. on the sensor’s environment perception through techniques such as simulation or modeling. However, all of them adopt ideal assumptions, which are difficult to meet in practical engineering applications.

Because the dust will make the light emitted by the LiDAR offset, it is easy to cause the outlier. From the perspective of filtering and noise reduction, Du et al. [53] carried out filtering and noise reduction for point clouds based on a bilateral filtering algorithm while maintaining the characteristics of point clouds. The algorithm fully considered the normal variation of the surface of the point cloud model and the relationship between the distance between neighboring points and noise and realized point cloud noise reduction by adjusting the normal position of sampling points. Cao et al. [54] used the improved bilateral filtering algorithm to remove the point cloud noise. First, the point was judged to be a feature point or a non-feature point. Then, the bilateral filtering factors of feature points and non-feature points were calculated according to the point clouds in different ranges. In this way, bilateral filtering point cloud denoising based on feature selection is achieved, which effectively maintains the features of the scanned object while removing the noise. Wu et al. [55] proposed a 3D point cloud denoising algorithm based on feature information classification. The principal component analysis method and quadric surface fitting method were integrated to estimate the geometric information of 3D point clouds, and then point clouds were divided into regions with rich feature information and regions with less feature information according to the average curvature of point clouds. Adaptive bilateral filtering algorithm and neighborhood distance average filtering algorithm were used to reduce noise in different regions. Table 3 summarizes the methods proposed in the literature [9,47,48,49,50,51,52,53,54,55,56,57,58,59]. All the above methods adopt noise reduction filtering to filter the noise generated under the influence of harsh environments. However, the data generated after noise filtering were not effectively repaired.

Table 3. Research status of LiDAR dust reduction technology.

Literature	Keywords	Major Contribution	Time
[9]	Final echo technique	By extracting the edge of the final pulse, the target and dust signal are distinguished, which effectively improves the loading target recognition effect under dust conditions.	1999
[47]	Dust influence rule	The dust influence rule of common LiDAR under different measuring distances, different dust concentrations, and different reflectance is summarized.	2017
[48]	Registration fusion	The influence of dust on environmental perception accuracy is reduced through the fusion of two-dimensional color images and three-dimensional point cloud registration.	2014
[49]	Background noise equation	The noise level equation is derived from the LiDAR equation, and the background noise equation is verified by using the observed LiDAR data and the simulated signal.	2013
[50]	Differential optical path	A LiDAR denoising method based on differential optical path is proposed, the mathematical model is derived and verified, and the signal-to-noise ratio is simulated based on this method.	2017
[51]	Probabilistic model	A probabilistic model of the interaction between LiDAR and dust is established, and the model is verified by simulation and experiment.	2013
[52]	Measuring suspended dust levels	A reliable method for measuring suspended dust levels by determining the transmission coefficient is developed and used for quantitative evaluation of the environmental sensing performance of LiDAR.	2009
[53]	Bilateral filtering	Based on the two-sided filtering algorithm, the point cloud is filtered and denoised, and the point cloud is denoised by adjusting the normal position of sampling points.	2010
[54]	Bilateral filtering	A two-sided filter point cloud denoising method based on feature selection is proposed, which can effectively preserve the features of scanned objects while removing noise.	2013
[55]	Feature information classification	A 3D point cloud denoising algorithm based on feature information classification is proposed.	2016
[56]	Linear grey Wolf optimization algorithm; Singular value decomposition; Empirical mode decomposition	A new joint denoising method EEMD-GGO-SVD, which includes empirical mode decomposition (EEMD), Grey Wolf optimization (GWO), and singular value decomposition (SVD), is proposed to improve the signal-to-noise ratio and extract useful signals.	2023
[57]	Local mean score; Threshold method	The local mean decomposition and the improved threshold method (LMD-ITM) are combined to process the noisy LiDAR signals, thus avoiding the loss of useful information.	2020
[58]	Singular value decomposition; Variational mode decomposition	A denoising method for wind LiDAR based on singular value decomposition (SVD) and variational mode decomposition (VMD) is proposed.	2021
[59]	Particle filter	A new scheme is proposed, which uses a particle filter (PF) instead of EnKF in the denoising algorithm to avoid the near distance deviation (bias) caused by excessive smoothing caused by EnKF.	2015

2.3. Research Status of Multi-Sensor Information Fusion Technology

Yamamoto et al. [15] comprehensively adopted monocular cameras, binocular stereo cameras, and LiDAR as environmental sensing devices to develop an autonomous intelligent excavator based on a comprehensive sensing system. Through multi-sensor information fusion, they realized the detection of obstacles and mining targets in the mining operation scene and the real-time three-dimensional modeling of mining results. Kwon et al. [17], addressing the challenge of complex and dynamic environments that are difficult to perceive during earthmoving operations of intelligent excavators, employed both cameras and LiDAR to sense the surroundings. They used the iterative closest point (ICP) algorithm to fuse image and point cloud data, effectively enhancing the environmental perception capabilities of the excavator during operation. Yoo et al. [60] developed a local terrain reconstruction system for intelligent excavators by comprehensively using LiDAR and binocular cameras, which effectively improved the environmental perception ability of intelligent excavators. At present, the research on multi-sensor information fusion technology for intelligent excavators is still relatively lacking, but the relevant research in other fields also has certain reference significance for the research and development of intelligent excavators.

Graeter et al. [61,62] comprehensively adopted LiDAR and a camera for data acquisition, and projected point cloud data onto images for data fusion, which added depth information to image pixels and effectively improved the ability of environmental perception. However, the resolution of the camera was higher than that of the radar, resulting in the loss of depth information of some image pixels. To solve the problem of resolution matching between LiDAR and the camera, De Silva et al. [63] calculated the geometric transformation relationship between different sensors and adopted the Gaussian interpolation algorithm to compensate for the image pixels lacking depth information, effectively improving the feature representation ability of data fusion. To meet the environmental perception requirements of auto-autonomous driving, Akhtar et al. [64] fused lidar point cloud data with camera image data, allocated color information to the point cloud and formed a 3D model. They assign the depth information to the image pixels to form a depth map. At the same time, to match the resolution of lidar data and image data, Gaussian process regression was used to interpolate the depth map. Lee et al. [65] proposed a new method to reconstruct 3D scenes by integrating data from different types of sensors, which mainly includes two main stages: local distance modeling and 3D depth map reconstruction. In the local range modeling stage, Gaussian process regression interpolation is used to perform the difference of 3D point clouds. In the reconstruction stage, images and interpolation points are fused. A 3D depth map is built, and the data are optimized based on Markov random fields. An et al. [66] developed an all-around three-dimensional color laser ranging system using a two-dimensional laser rangefinder, a color camera, and a rotating platform. Based on the improved rectangular hole checkerboard calibration method, two-dimensional images, and point clouds were fused into three-dimensional color point clouds.

In terms of building environmental perception, Yang et al. [67] proposed a method that uses LiDAR and image data fusion for feature extraction and three-dimensional reconstruction of conventional building surfaces. They extracted initial features from images according to gradient direction and realized the conversion of 2D image features to 3D point cloud space based on registration. Finally, plane-fitting of the point cloud data around the feature lines was carried out to achieve better model reconstruction. Li et al. [68] made full use of the complementary advantages of LiDAR data and optical imaging to extract different building features from the two data sources and fuse them to form the final complete building boundary. This method can automatically and accurately extract building boundaries of complex shapes and has strong robustness in complex environments.

In terms of the outdoor environmental perception of robots, Yu et al. [69] proposed a positioning algorithm based on the fusion of camera and LiDAR. By combining the depth information of LiDAR with the color texture information of the image, a feature point matching relationship is established. This method solves the localization problem of outdoor robots under the influence of complex environmental factors such as light and ground fluctuation. Wang et al. [70] fused visual information, LiDAR information, and odometer information to solve the problem of timely positioning and accurate composition of mobile robots in unknown environments. Experiments proved that multi-sensor information fusion can make the composition more accurate and robust.

In terms of unmanned road environmental perception, Qu et al. [71] fused LiDAR depth information by mapping it to two-dimensional camera images and realized real-time pedestrian detection through cluster analysis and support vector machine classification. Wu et al. [72] proposed a real-time vehicle detection method based on the fusion of radar and vision sensors. Hough transform and Chebyshev’s theorem were used to filter the non-target points in the LiDAR data channel, significant vehicle images were generated in the image data channel, and the vehicle position was located by segmentation method. Table 4 summarizes the methods proposed in the literature [15,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76]. At present, most of the multi-sensor information fusion technologies use LiDAR and cameras for fusion, and the mode is relatively simple. More model data need to be added for the necessary supplement to express the measured target more comprehensively.

Table 4. Research status of multi-sensor fusion technology.

Literature	Keywords	Major Contribution	Time
[15]	Monocular camera, binocular stereo camera, and LiDAR	Using the monocular camera, binocular stereo camera, and LiDAR as environmental sensing equipment, an autonomous intelligent excavator based on a comprehensive sensing system was developed. Through multi-sensor information fusion, obstacles and targets in the mining scene were detected, and real-time three-dimensional modeling of mining results was achieved.	2012
[17]	Cameras and LiDAR	Cameras and LiDAR are used for environmental perception, and the ICP algorithm is used to fuse images and point cloud information.	2013
[60]	LiDAR and binocular camera	The local terrain reconstruction system of intelligent excavators is developed by using LiDAR and binocular cameras, which effectively improves the environmental perception ability of intelligent excavators.	2017
[61,62]	projection	The point cloud data is projected onto the image for data fusion.	2018
[63]	Gaussian interpolation	After projection fusion, the geometric transformation relationship between different sensors is calculated, and the image pixels lacking depth information are compensated by the Gaussian interpolation algorithm.	2018
[64]	Gaussian process regression interpolation	The color information of the image is assigned to the point cloud by projection, and the resolution of the LiDAR data is matched by Gaussian process regression interpolation.	2019
[65]	Local distance modeling, 3D depth map reconstruction	In the local scale modeling stage, the three-dimensional point cloud is interpolated using Gaussian process regression, and in the reconstruction stage, the image and interpolation points are fused to build a three-dimensional depth map, and the merged data is optimized based on Markov random fields.	2016
[66]	Full-dimensional color laser ranging system	An omnidirectional 3D color laser ranging system is developed. Based on the improved rectangular hole checkerboard calibration method, the 2D image and point cloud are fused into a 3D color point cloud.	2019
[67]	Registration, plane fitting	The initial feature is extracted from the image according to the gradient direction, and the conversion from 2D image to 3D point cloud is realized on the basis of registration, and then the point cloud data around the feature line is fitted in the plane.	2016
[68]	Complementary advantages	Taking full advantage of the complementary advantages of LiDAR data and optical imaging, different architectural features are extracted from the two data sources and fused to form the final complete architectural boundary.	2013
[69]	Feature point matching	Combining the depth information of LiDAR and the color texture information of an image, the matching relationship of feature points is constructed to solve the location problem of outdoor robots in complex environments.	2019
[70]	Timely positioning and accurate composition	The visual information, LiDAR information, and odometer information are integrated to solve the problem of timely positioning and accurate composition of mobile robots in an unknown environment.	2019
[71]	Fusion, clustering	The laser radar depth information is mapped to two-dimensional camera images for fusion, and real-time pedestrian detection is realized by cluster analysis and support vector machine classification.	2019
[72]	Vehicle real-time detection	Hough transform and Chebyshev’s theorem is used to remove the non-target points in the radar data channel, and the significant image of the vehicle is generated in the image data channel, and then the vehicle position is located in real-time by segmentation method.	2015
[73]	Loop, high-precision	A multi-sensor data fusion scheme of a tightly coupled LiDAR-camera-IMU system is proposed, and the loop optimization of single-vehicle and multi-vehicle is realized.	2024
[74]	Information fusion; Tracking method; Particle filter algorithm	Multiple sensors are used to collect the trajectory information of the robot at the same time, and the information is fused accordingly. Then, the particle filter algorithm is introduced to realize the trajectory tracking of the robot. Finally, the comparison experiment of the trajectory tracking of the robot is carried out with other methods.	2024
[75]	Monocular Visual SLAM	A monocular VISLAM algorithm based on EKF and graph optimization complementary framework is proposed to improve the positioning accuracy and computational cost ratio of free mobile robots in three-dimensional space.	2021
[76]	Data association	An object association fusion algorithm among distributed sensors based on the covariance of object state estimation error is proposed. Independent sequential object association algorithm and weighted fusion algorithm are used to associate and fuse each sensor object after a coordinate transformation, which effectively improves the reliability and accuracy of the fusion system.	2020

2.4. Research Status of Segmentation and Recognition Technology of 3D Operation Scenes

The three-dimensional point cloud can accurately record the location and geometric information of the measured target; it has advantages that two-dimensional images cannot match. This important information is the basis for intelligent excavators to carry out tasks such as path planning and trajectory planning, so the intelligent excavator operation scene perception data is mainly the three-dimensional point cloud. The three-dimensional operation scene of intelligent excavators is complex and dynamic, and the main targets to be divided and identified include piles, vehicles, personnel, trees, buildings, ground, etc., which is extensive and diverse.

Aiming at the segmentation and recognition of the intelligent excavator’s three-dimensional operation scene, Yang et al. [77] proposed a multi-scale edge detection algorithm by constructing a quadratic B-spline wavelet. Based on this algorithm, the edge features of the loading vehicle of an intelligent excavator were detected and recognized. Zhu et al. [78] proposed a method based on point cloud clustering feature histogram for the identification of intelligent excavator operation targets. This method can simultaneously identify multiple targets such as intelligent excavator loading vehicles and material piles and has good stability. Aiming at intelligent mining excavators under the influence of dust, Phillips et al. [79] proposed a target segmentation and identification method based on maximum evidence, which improved the accuracy and robustness of segmentation and identification. This method achieves the segmentation and recognition of operational targets such as buckets and piles. In addition, aiming at the anti-collision problem of mining excavator buckets and mining cards, they divided and identified the edges of buckets and mining cards [80]. At present, there are few research results on the segmentation and recognition technology for the three-dimensional operation of intelligent excavators, but relevant research in other fields can have certain reference significance for the research and development of intelligent excavators. This research can be divided into traditional machine learning algorithms and deep learning algorithms.

The traditional machine learning algorithm extracts sample features and uses the classifier to complete the segmentation and recognition of 3D job scenes. The use of Markov random fields [81], random forests [82], support vector machines [83,84] and other models has improved the recognition speed and accuracy. Niemeyer et al. [85] comprehensively adopted a random forest classifier and conditional random field and used reflection intensity, curvature and other features to identify buildings in three-dimensional point cloud scenes. Golovinskin et al. [86] used the graph-cutting method to segment the point cloud scene, and then extracted features such as target location and nearest distance to the roadside, and applied a support vector machine to classify and recognize objects such as vehicles and street lights. Yao et al. [87] extracted spectral, geometric, and spatial features of 3D point cloud data, and then used the AdaBoost classifier to extract trees from urban scenes. Zhao et al. [88] segmented a three-dimensional point cloud using the scanning line algorithm and extract the height of the segmented surface, normal vector variance, and other features. They also used a support vector machine to classify and recognize targets such as pedestrians and trees. Wang et al. [89] segmented point cloud data after the removal of ground points through raster processing, and set the grid and its neighborhood as local point clouds. Then, they extracted the reflective features of the local point cloud and used Hof forest to segment and identify the targets such as trees and vehicles in the urban environment. Wang et al. [90] conducted multi-scale sampling of point cloud data and classified the sampling points into different levels of point cloud clusters. Then, vehicles, pedestrians, buildings, and other targets in the urban scene were identified based on the Bayesian probabilistic model and AdaBoost classifier. Traditional machine learning algorithms rely heavily on artificial feature extraction. It is difficult to extract complex features accurately, resulting in lower segmentation recognition accuracy.

Deep learning algorithms can be divided into two categories according to the different processing methods of the point cloud. One is to regularize the disordered point cloud first, such as projecting a 3D point cloud onto a 2D plane [91,92,93,94,95]. Alternatively, the disordered point cloud is rasterized and voxelized [96,97,98,99], and then processed by convolutional neural networks (CNNS). This kind of algorithm is easier to understand, and can effectively combine the successful experience of a two-dimensional image semantic segmentation algorithm to easily transplant the image semantic segmentation algorithm to three-dimensional point cloud semantic segmentation. However, point cloud preprocessing will lose certain point cloud information, which will have adverse effects on point cloud feature extraction, especially point cloud processing in fine-grained complex scenes [100]. The other type directly processes the original point cloud data and does not transform the point cloud data before input to the network to save the point cloud information to the maximum extent. This type of algorithm has strong anti-noise ability and a high intelligence degree and can obtain relatively robust and accurate segmentation and recognition results for point clouds with different quality and density distributions. It is more in line with the need for intelligent autonomous operation of excavators.

PointNet, proposed by Qi et al. [101], was the first network to directly input original point cloud data for segmentation and recognition. This network solved problems such as point cloud disorder and geometric rotation invariance, and the experimental results showed that breakthrough progress had been made in both point cloud classification and semantic segmentation. However, the algorithm does not make use of local feature information effectively, and the perception ability of local spatial features is poor. To solve the problems existing in PointNet, Qi et al. [102] proposed PointNet++ and constructed a hierarchical neural network structure. At each layer, point clouds were first sampled and divided into regions, and then the basic PointNet network was used to extract local features in each region. These local features are then further sampled and zoned as part of the input to extract higher-level features. Pointnet ++ improves the ability to extract local features of the point cloud and effectively improves the segmentation and recognition effect of point cloud targets such as walls and roofs. Jiang et al. [103] proposed PointSIFT, a three-dimensional point cloud segmentation model based on scale-invariant feature transformation. In this model, multiple main direction information points are encoded by directional coding units, and multiple coding units are stacked to obtain multi-scale features. Zhao et al. [104] proposed a PointWeb 3D point cloud segmentation model, which connected each point in the local neighborhood to better extract local point cloud features, and proposed an adaptive feature adjustment module to enhance point-by-point feature expression capability. Wang et al. [105] from MIT applied graph convolutional neural network to 3D point cloud processing based on PointNet, and proposed DGCNN, a dynamic graph volume point cloud processing algorithm, which effectively solved the problem of poor perception of local spatial features of PointNet, and greatly improved its point cloud classification and segmentation ability and robustness. To improve the efficiency of semantic segmentation of three-dimensional point clouds in large scenes, Hu et al. [106] proposed RandLA-Net, which selected the most efficient random sampling by comparing existing sampling methods. They proposed a local feature sampler to reduce the information lost by random sampling. The network realizes efficient segmentation and identification of objects such as cars and trees in the cloud data of outdoor scenic spots.

Ren et al. [107] introduced a multi-modal feature fusion network (MFFNet) for 3D point cloud semantic segmentation. Unlike the previous approach of learning directly from colored point clouds (XYZRGB), MFFNet converts point clouds into 2D RGB images and frequency image representations for efficient multimodal feature fusion. Poliyapram et al. [108] proposed a novel end-to-end point-by-point LiDAR and image multimodal fusion network (PMNet) based on deep learning, which is used to fuse aerial image features for 3D segmentation of aerial point clouds. Due to the low resolution of the point cloud model and the lack of spatial location information of the image model, a single model cannot accurately calculate ore fragmentation. To solve this problem, Peng et al. [109] proposed an ore fragmentation calculation method (ORFCM) based on the multi-modal fusion of point cloud and image. ORFCM takes full advantage of multi-modal data, including fine-grained object segmentation of images and spatial location information of point clouds. An et al. [110] proposed a multi-level framework for three-dimensional point cloud ground segmentation of outdoor scenes based on shape analysis. The method explores the local shape and multilevel structure of the outdoor scene and constructs a probabilistic ground model to improve the accuracy and adaptability of the ground segmentation. Hu et al. [111] proposed a semantic segmentation network for outdoor panoramic images based on distorted convolution. In this network, the distortion convolution module (DCM) and depth feature aggregation network (DFAN) are introduced into the semantic segmentation of panoramic images, which improves the segmentation accuracy. Table 5 summarizes the methods proposed in the literature [77,78,79,80,85,86,87,88,89,90,101,102,103,104,105,106,109,112,113,114,115]. All of the above deep learning methods rely on a large set of labeled data. However, the process of data set annotation is time-consuming and labor-intensive, and the degree of automation is low. Therefore, the development of deep learning algorithms is limited.

Table 5. Research status of segmentation and recognition technology for 3D scenes.

Literature	Keywords	Major Contribution	Time
[77]	Multi-scale edge detection	A multi-scale edge detection algorithm is proposed by constructing quadratic B-spline wavelet, and based on this algorithm, the edge features of intelligent excavator loaded vehicles are detected and recognized.	2004
[78]	Point cloud clustering feature histogram	In this paper, a method based on point cloud clustering feature histogram is proposed, which can identify multiple targets such as intelligent excavator loading vehicle and material pile at the same time and has good stability.	2017
[79]	Maximum evidence	Aiming at the intelligent mining excavator under the influence of dust, a method of target segmentation recognition based on maximum evidence is proposed, which realizes the segmentation and recognition of operation targets such as buckets and piles.	2018
[80]	Buckets and mining cards	Aiming at the problem of anti-collision between the mine excavator bucket and the mine card, the paper divides and identifies the bucket and mine card.	2016
[85]	Random forest classifier, conditional random field	By using a random forest classifier and conditional random field, three-dimensional point cloud segmentation is realized by using reflection intensity and curvature.	2014
[86]	Graph cutting method, support vector machine	The point cloud scene is segmented by graph cutting method, then the features of target location and distance are extracted, and the target is classified and recognized by support vector machine.	2009
[87]	AdaBoost classifier	The spectral and spatial features of point cloud are extracted, and the target is identified by AdaBoost classifier.	2013
[88]	Scan line algorithm	The three-dimensional point cloud is segmented by scanning line algorithm, and the features such as height and normal vector variance are extracted, and then the target is classified and recognized by support vector machine.	2010
[89]	Rasterization, Hofer Forest	The point cloud data after the removal of ground points is segmulated by raster processing, and the grid and its neighborhood are set as local point clouds, then the reflection features are extracted, and the Hough Forest is used to achieve multi-target recognition.	2014
[90]	Multi-scale sampling, Bayes	Multi-scale sampling of point clouds is carried out, and the sampling points are classified into different levels of point cloud clusters, and then the target is identified based on Bayesian probability model and AdaBoost classifier.	2015
[101]	Work directly with the original point cloud	Directly input the original point cloud data for segmentation and identification, to solve the problem of point cloud disorder, geometric rotation invariance and so on.	2017
[102]	Hierarchical neural network structure	A hierarchical neural network structure is constructed to improve the local feature extraction ability of point cloud, and effectively improve the accuracy of point cloud classification and semantic segmentation.	2017
[103]	Scale invariant feature	A 3D point cloud segmentation model based on scale-invariant feature transformation is proposed. Multiple main direction information is encoded by directional coding units, and multiple coding units are stacked to obtain multi-scale features.	2018
[104]	k-means, k-NN	The k-means clustering algorithm and k-NN algorithm are used to select the neighborhood in the global space and the feature space, respectively, and pairwise distance loss and centroid loss are introduced into the loss function.	2019
[105]	Dynamic graph convolution	The graph convolutional neural network is applied to 3D point cloud processing, and a dynamic map volume point cloud processing algorithm is proposed, which effectively improves the local spatial feature perception ability.	2019
[106]	Random sampling	The most efficient random sampling is selected, and a local feature sampler is proposed to reduce the information lost in random sampling, which effectively improves the efficiency of target segmentation recognition.	2020
[112]	Three-dimensional point cloud semantic segmentation	A three-dimensional point cloud semantic segmentation network based on memory enhancement and lightweight attention mechanism is proposed to solve the problems of semantic segmentation accuracy and deployment capability.	2024
[109]	Multi-mode fusion, segmentation map	A new ore-rock fragmentation calculation method (ORFCM) based on multi-modal fusion of point cloud and image is proposed to solve the problem that the resolution of point cloud model is low and the image model lacks spatial position information, so the ore-rock fragmentation can not be accurately calculated.	2023
[113]	Point cloud data segmentation	A simplified method of coal-rock point cloud data is designed on the basis of retaining the characteristic points, and an improved ant colony optimization (IACO) algorithm is proposed to realize the segmentation and recognition of coal-rock point cloud data.	2021
[114]	Adaptive watershed segmentation	By introducing the Phansalkar binarization method, a watershed seed point marking method based on block contour firmness is proposed, and an adaptive watershed segmentation algorithm based on block shape is formed to better solve the segmentation errors caused by adhesion, stacking, and edge blurring in blast rock images.	2022
[115]	DUNet; Residual connection	By combining the residual structure of the convolutional neural network with DUNet model, the accuracy of image segmentation is greatly improved.	2020

3. Research Prospect of Key Technologies of Intelligent Excavator Environmental Perception

3.1. Research Prospect of LiDAR Vibration Reduction Technology

Although the existing achievements can provide some reference for the research on the LiDAR vibration reduction technology of intelligent excavators, its research background is mainly aimed at the airborne and general vehicle environment. Intelligent excavators have unique vibration characteristics, so the subsequent research on vibration reduction of intelligent excavators needs to be carried out according to the vibration characteristics of intelligent excavators. The research prospects are as follows.

3.1.1. Experimental Measurement of Vibration

The vibration test and measurement method of intelligent excavators are studied. The appropriate vibration meter is selected according to the excavator’s main vibration parameters, such as vibration frequency and amplitude, and a detailed vibration measurement scheme is developed. The vibration parameters are measured at the installation position of important environmental sensing sensors, which provides data support for the establishment of a vibration analysis model and the design of a vibration damping table.

3.1.2. The Vibration Analysis Model Is Established

According to the measurement data of the excavator vibration experiment, the vibration analysis model is studied and established. Analyzing the influence of the stiffness and damping of the shock absorber on the performance of the vibration absorber provides theoretical and data support for the design of the vibration absorber.

3.1.3. Vibration Mechanism Analysis

By collecting vibration data in the process of operation, the vibration signal is converted into frequency domain information through spectrum analysis. Different frequency components are identified. Each frequency peak in the spectrogram was analyzed to reveal the characteristics of the vibration.

3.1.4. Hardware Design of Vibration Reduction Platform

Based on the above theoretical results, the design method of a shock absorber is studied, including scheme design, shock absorber selection, shock absorber layout, and installation mode. Considering the installation space, the vibration excitation in the working environment, and the natural frequency and vibration transmission of the shock absorber, the appropriate type of shock absorber is selected. The reasonable design of the shock absorber layout ensures that linear vibration and angular vibration are sufficiently decoupled to avoid large angular vibration. In addition, it is also necessary to fully consider the installation space limitation, maintainability, manufacturability, and other aspects. To verify the damping effect of the damping table on 3D LiDAR, a random vibration test was carried out. The vibration table generates random vibration excitation along the horizontal direction and vertical direction, respectively, and the acceleration response of the vibration absorbing table in different directions is obtained through the measuring point. According to the amplitude change of the excitation signal and response signal in the random vibration test, the damping effect of the damping table can be measured.

3.1.5. Point Cloud Vibration Error Correction

The method of point cloud vibration error correction is explored. The position and posture sensor are used to measure the displacement and attitude angle of LiDAR relative to the installation platform of the excavator under vibration conditions, and the error transfer model is established to correct the three-dimensional point cloud data affected by vibration.

3.2. Research Prospects of LiDAR Dust Noise Reduction Technology

At present, LiDAR dust and noise reduction technology has a certain research foundation. However, there are fewer studies on the operating environment of intelligent excavators, and most of them use experimental analysis methods, and the effect of dust and noise reduction is seriously dependent on hardware. Combined with the characteristics of the operating environment of intelligent excavators and the research achievements of LiDAR dust and noise reduction technology in other fields, the research prospects are as follows:

3.2.1. Final Echo Technique

The measurement principles and hardware characteristics of different types of LiDAR were studied and analyzed. For three-dimensional LiDAR with multiple echo technology, the point cloud data obtained from the last echo were selected as the target point cloud data. This minimizes the effect of dust on the accuracy of 3D LIDAR environmental measurements at the hardware level.

3.2.2. Compare and Select the Best Quality Point Cloud Data

Since the concentration and distribution of dust in the air have dynamic characteristics and will change with the change of time, it is possible to select a reasonable installation position through experimental research and install multiple sets of LiDAR on the intelligent excavator. From the multi-frame point cloud data collected at different locations at different times, the environmental point cloud data with the best quality and the least influence of dust is selected as the main point cloud data. At the same time, aiming at problems such as occlusion, the registration fusion method is studied, and other point cloud data collected at the same time are used to compensate and repair them.

3.2.3. Filtering and Noise Reduction Based on Multi-Modal Fusion

The point cloud is fused with the image information to extract the color, texture, and other details of the dust noise. By comprehensively comparing the typical three-dimensional point cloud filtering algorithms such as bilateral filtering and statistical filtering, the best algorithm is selected to filter the point cloud and remove the dust noise based on protecting the important features of the point cloud.

3.2.4. Sparse Repair

Aiming at the point cloud sparsity caused by point cloud filtering and noise reduction, the interpolation method is studied to repair the sparsity of the point cloud, to ensure the expression ability of the three-dimensional point cloud for a real mining environment to the greatest extent. For example, moving least squares algorithms fill holes or sparse areas by locally fitting smooth surfaces. By least square fitting in the neighborhood, the surface of the point cloud is smooth and close to the original point cloud.

3.3. Research Prospect of Multi-Sensor Information Fusion Technology

Intelligent excavators have the characteristics of complex working environments and a wide range of working scenes, which determine that more types and more sensors are needed for intelligent excavators’ environmental perception, which brings great challenges to multi-sensor information fusion. Combined with the characteristics of the operating environment of intelligent excavators and the existing research results of multi-sensor information fusion technology, the research prospects are as follows.

3.3.1. Sensor Calibration

In order to overcome the lack of information acquisition ability of a single sensor and make the intelligent excavator have better environmental perception ability, a calibration tool suitable for the working environment of the intelligent excavator is designed. The internal and external calibration methods for both the camera and LiDAR are investigated to accurately determine the internal parameters of the camera and the external parameters of both the camera and LiDAR. This enables the construction of a mapping relationship between the laser points and pixel coordinates. Sensor calibration is the basis and key to realizing point cloud and image fusion and obtaining 3D color point cloud data, so it is very important to study this technology.

3.3.2. Data Synchronization

To realize the intelligent excavator’s all-around perception of a wide range of operation scenarios, the data synchronization module was developed using the precision time protocol (PTP), presentation time stamp (PTS), and hardware trigger methods to achieve high-precision synchronization of sensors such as LiDAR, camera, GPS, inertial measurement unit (IMU), etc. Data synchronization is the key to realizing multi-view three-dimensional color point cloud data fusion, and the accuracy of data synchronization directly determines the all-around perception accuracy of intelligent excavators for a wide range of work scenes. Therefore, it is necessary to conduct in-depth research on this technology.

3.3.3. Multiple Sight Registration

To obtain the complete three-dimensional point cloud data of the large-scale operation scene of the intelligent excavator, it is necessary to conduct multi-angle scanning of the operation scene. Therefore, we need to study the multiple-view registration method, accurately register the multiple-view laser point cloud under the unified coordinate system, and realize the integration of large-scale operation scenery. However, most of the existing registration methods are subject to too much interference from human factors, and there is still a gap between the registration accuracy and efficiency and the practical application requirements. So, it is of great practical significance to study the high-precision and high-efficiency 3D point cloud multi-view registration method with a higher degree of automation.

3.4. Research Prospect of 3D Operation Scene Segmentation and Recognition Technology

The working environment of intelligent excavators is complex, and the working target is dynamic and changeable, which brings great challenges to the segmentation and recognition of 3D working scenes. Combined with the characteristics of the operating environment of intelligent excavators and the existing research results of multi-three-dimensional scene segmentation and recognition technology, the research prospects are as follows.

3.4.1. Point Cloud Data Feature Enhancement

Most of the original LiDAR point cloud data only have spatial coordinate information, and some can provide additional information about the reflection intensity of the target surface, but few features can be used. For complex terrain and targets, it is difficult to extract sufficient features for effective segmentation and recognition of point cloud data through nonlinear computing. The feature enhancement method of point cloud data can be studied. By analyzing the surface topography, physical features, dimensions, and other physical factors of intelligent excavator operation targets such as material piles, buckets, loading trucks, etc., representative features with physical significance and interpretability are designed. This can effectively enhance the features of the original data, which is conducive to improving the accuracy and robustness of segmentation recognition.

3.4.2. Network Structure Design

The operation scene of intelligent excavators is complex, large-scale, and dynamic, which poses great challenges to the accuracy and efficiency of 3D scene segmentation and recognition. Building on the successful application of the attention mechanism [116,117,118,119,120,121] in 2D image processing, this paper explores a method to incorporate the attention mechanism into 3D deep learning semantic segmentation. By assigning different weights to various features, the approach enhances the contribution of effective features while suppressing the influence of ineffective ones. The accuracy of 3D scene segmentation recognition is higher through feature re-calibration.

In addition, we can learn from the successful experience of the lightweight network [122,123,124,125,126] in two-dimensional image processing to explore the lightweight mechanism suitable for a three-dimensional deep-learning semantic segmentation algorithm. The computational complexity of the algorithm is reduced by changing the convolution mode, using convolution instead of full join, and designing an efficient sampling mode. These methods can also reduce the number of parameters and improve the efficiency of segmentation recognition.

3.4.3. Online Learning Style Changes

The intelligent excavator 3D operation scenario has a large range, many targets, a large amount of point cloud data, and a high cost of 3D data annotation, which has a great impact on the application of conventional deep learning algorithms. However, by changing the learning mode and exploring and studying weakly supervised, small sample learning algorithms or unsupervised self-learning algorithms, the workload of data annotation can be effectively reduced. Currently, there is little research on this type of deep learning algorithm, which has a very broad research value and application prospect.

4. Conclusions

Environmental perception is a hot and difficult research topic in the field of intelligent excavators. This paper summarizes and analyzes this problem in detail. Firstly, the main system composition of the intelligent excavator is introduced, and by summarizing the research status of the intelligent excavator, it is found that there is still a large gap in the current research on the environmental perception system. Thus, there is an urgent need to carry out in-depth research and discussion on the key technology of environmental perception of intelligent excavators. Secondly, by analyzing the characteristics of the operating environment of intelligent excavators, the key technologies that need to be solved urgently for the environmental perception of intelligent excavators are proposed. These technologies include LiDAR vibration reduction, LiDAR dust and noise reduction, multi-sensor information fusion, 3D operation scene segmentation and recognition, etc. In addition, the research status at home and abroad is summarized in detail because of these key issues. Finally, follow-up research on the key technology of environmental perception of intelligent excavators is recommended.