Highlights
What are the main findings?
- A two-stage camera–LiDAR extrinsic calibration algorithm is proposed to address the issues of initial value dependence and poor scene adaptability in existing targetless extrinsic calibration methods.
- To enhance the scene adaptability of the calibration algorithm, a cascaded optimization scheme of extrinsic parameter is designed in light of the varying constraint capabilities of edge features with different directions on individual extrinsic parameters.
What are the implications of the main findings?
- Cascaded optimization significantly reduces computational complexity compared to direct optimization, enabling fast and accurate extrinsic estimation.
- Compared to existing camera–LiDAR extrinsic calibration algorithms, the proposed algorithm achieves an accuracy advantage.
Abstract
Accurate and stable extrinsic calibration is the foundation of high-quality fusion sensing and positioning of camera and Light Detection and Ranging (LiDAR). However, traditional targetless calibration methods suffer from limitations such as poor scene adaptability and unstable convergence, which significantly restrict calibration accuracy and robustness in complex environments. Aiming at solving those problems, we propose an online cascade-optimization-based extrinsic calibration method of combining motion trajectory alignment and edge feature alignment. In the initial calibration stage, a hand–eye calibration algorithm is designed by minimizing the residual discrepancies between camera odometry and LiDAR odometry sequences. It establishes a robust initialization for subsequent optimization. Then, in order to extract robust edge line features from sparse point clouds, we employ depth difference and planar edges of point clouds in the optimization process. Subsequently, principal component analysis (PCA) is applied to compute the principal direction of the extracted line features, enabling a decoupled optimization scheme that accounts for directional observability. This approach effectively mitigates the adverse effects of uneven environmental feature distributions. Experimental validation on typical urban datasets demonstrates the method’s generalizability and competitive accuracy: rotational parameter errors are constrained within 0.25°, and translational errors are maintained below 0.05 m. This affirms the method’s suitability for high-accuracy engineering applications.
1. Introduction
With the escalating complexity of application scenes, intelligent vehicles and other autonomous systems are increasingly compelled to employ multi-modal sensor configurations for achieving accurate and comprehensive environmental perception. They serve as a foundational prerequisite for implementing safe motion decision and control. The multi-modal sensor fusion technology effectively mitigates the inherent limitations of single-sensor systems, enabling complementary integration across heterogeneous data. Within this framework, the LiDAR–camera sensor ensemble stands as the preeminent configuration in intelligent transportation and robotic systems [1,2,3]. LiDAR provides precise 3D spatial target information with high ranging accuracy and robust immunity to illumination variations, yet it lacks color and texture detail. Conversely, cameras excel at capturing rich pixel-level features and semantic information, including target color and contour. But cameras suffer from inherent limitations in direct depth measurement and are susceptible to environmental variations such as illumination and weather conditions, leading to complex imaging dynamics and reduced stability. Therefore, LiDAR and cameras have strong complementarity, and the fusion of the two can simultaneously obtain spatial distance, geometric measurement, and semantic information so as to perceive the application scene with higher quality.
Critical to LiDAR–camera data fusion is the accurate and stable rigid body transformation between sensor coordinate systems, which can be represented by the rotation matrix and translation vector. It establishes the precise spatial correspondence between point cloud data and image pixels. According to whether a target is required, the LiDAR–camera’s extrinsic calibration can be divided into two categories: manual calibration and automatic calibration.
Manual calibration methods (also referred to as target-based calibration) utilize artificial markers with distinct geometric features and high-accuracy fabrication, such as checkerboards, triangular fixtures, and spherical targets. Zhang et al. [4] pioneered a calibration approach using planar checkerboard patterns viewed from unknown orientations, leveraging geometric constraints of coplanar points to solve the pose transformation. Subsequent studies addressing Zhang’s limitations focused on improving marker pose placement, constraint models, and geometric designs. ITAMI et al. [5] addressed checkerboard pose sensitivity by introducing point-plane constraints via LiDAR horizontal–vertical sampling. XIE et al. [6] enhanced Dhall’s method [7] of introducing point–line–plane constraints to improve calibration convergence and accuracy. Shi et al. [8] designed a hybrid ball-hollow plate marker using hollow plate edge features for initial estimation and projection iteration of spherical center for optimization, achieving 0.01 m translational and 0.07° rotational accuracy. Due to the high manufacturing accuracy of markers and the uniform distribution and easy extraction of artificial point, line, and plane features, manual calibration methods can achieve relatively high accuracy. Nevertheless, such methods necessitate the setup of dedicated calibration environments and exhibit significant reliance on manual intervention, rendering them unsuitable for online calibration scenes. This significantly diminishes the autonomy of unmanned vehicles and intelligent robots in complex operational environments.
Targetless methods do not require artificial targets and can extract point–line–surface scenes features or trajectory features in the environment to construct constraints for extrinsic calibration. Such methods can generally be classified into motion-based methods and feature-based methods. Motion-based methods leverage hand–eye calibration to match and align sensor trajectories, operating without requiring a common field of view between the camera and LiDAR. Nevertheless, their calibration accuracy remains limited by scale ambiguity in monocular vision systems and inherent errors in trajectory estimation, leading to suboptimal parameter estimation. Zhao Hang et al. [9] employed raw motion measurements combined with ground feature points and ground elevation constraints to estimate the 6-DoF extrinsic parameters of the LiDAR–camera system. The feature-based approach estimates extrinsic parameters by extracting and matching common-view features from the overlapping field of view between images and point clouds. Exploiting the ubiquity of edge information, Levinson [10] introduced edge mutual information between point clouds and images for calibration. Subsequent works by Yuan et al. [11] and Liu et al. [12] employed fixed-size and adaptive voxelization, respectively, to extract depth-discontinuous boundaries from building structured point clouds, achieving high-accuracy calibration. However, such methods require structured scenes to provide regular and sufficient structural features. Therefore, designing fast, accurate, and highly adaptable feature extraction algorithms is crucial for online feature-based calibration methods. In addition, these methods require relatively accurate initial extrinsic parameters to achieve rapid alignment between point cloud features and visual features, as well as fast convergence of extrinsic parameter optimization [13]. Additionally, deep learning-based approaches have increasingly been adopted for LiDAR–camera extrinsic calibration. Schneider et al. [14] pioneered the use of the RegNet network to achieve end-to-end estimation of 6oD pose transformation parameters. Xudong Lv et al. [15] proposed the self-supervised LCCNet model and achieved stable calibration results by matching the geometric structure and photometric consistency of point-cloud–image pairs. Deep learning-based methods, in general, are subject to the requirement of a large volume of training data while also exhibiting poor generalization ability and restrictive usage conditions.
To address the aforementioned challenges, this paper proposes an extrinsic calibration approach for LiDAR–camera systems based on the cascade optimization of motion trajectories and edge features. This approach enables the online estimation of extrinsic parameters in complex application scenes without the need for artificial markers. The main contributions of the proposed method are as follows:
- (1)
- Based on the coarse-to-fine cascade calibration framework, a motion-trajectory-based calibration algorithm is proposed to enhance the constraint capability of trajectory consistency in scenes where camera scale is lost. And it provides a robust initial value of the extrinsic parameters for the calibration optimization stage.
- (2)
- For weakly structured application scenes such as streets, a depth difference–planar edge joint feature extraction method is designed to process sparse point cloud data. It fully obtains multiple types of vertical and horizontal edge features such as tree trunks, streetlamp poles, walls, and shallow grooves on the road.
- (3)
- The principal direction of the extracted features is calculated via PCA decomposition [16]. Considering the varying constraint capabilities of features with different principal directions on extrinsic parameters, cascaded optimization is adopted to achieve fast and accurate estimation.
2. Joint Calibration Based on Cascade Optimization
2.1. Basic Principle of Extrinsic Calibration
The extrinsic calibration of the LiDAR and camera is essentially to solve the rigid body transformation matrix from the LiDAR coordinate system to the camera coordi-nate system, as shown in the Figure 1. The transformation matrix can be represented by a rotation matrix and a translation vector .
Figure 1.
LiDAR and camera coordinate system conversion relationship.
Let be a point in three-dimensional space that is jointly observed by the camera and LiDAR. is the coordinate value of point in the LiDAR coordinate system and is the coordinate value of point in the camera coordinate system. Then there is
where , , and represent the transformation from the LiDAR coordinate system to the camera coordinate system.
The camera data is usually represented in the form of an image, and the coordinate position of the spatial point in the image is in pixels. The mapping from the three-dimensional point in the camera coordinate system to the two-dimensional planar point in the pixel coordinate system can be realized using the camera’s intrinsic matrix . In the three-dimensional to two-dimensional transformation process, the distance scale information is lost, that is,
where is the two-dimensional pixel point in the pixel coordinate system and is the scale information.
2.2. Online Calibration in the Real Environment
In the field of intelligent driving and autonomous robotics, the core connotation of online calibration is that calibration can be automatically performed during vehicle operation in natural environments such as open roads without human intervention, dedicated sites, or calibration targets. In the dynamic and complex real-world environments such as a typical urban street, it is impractical to pre-deploy artificial calibration features in the surroundings. This renders manual calibration or target-based calibration methods infeasible for online implementation, whereas the targetless extrinsic calibration scheme for LiDAR-camera systems has been validated to exhibit sufficient reliability [9,17]. Accurate extrinsic parameters can be obtained using co-observed specialized calibration devices, which are commonly employed in factory settings for unmanned vehicles and intelligent robots. In addition to this, during operational use, external disturbances such as road vibrations or impacts may alter the transformation matrix between LiDAR and the camera. In such cases, the pre-calibrated extrinsic parameter matrix no longer aligns with the current sensor transformation relationship, leading to misalignment between the camera and LiDAR coordinate systems. This misalignment further results in deviations when projecting point cloud data onto images, as illustrated in Figure 2.
Figure 2.
Project point cloud onto the image. (a) Accurate parameters by factory calibration. (b) Sensor pose changes cause parameter changes.
Under such circumstances, re-calibration of the LiDAR and camera is necessary to reinstate consistent environmental feature observations across sensors. However, target-based calibration methods depend on specialized calibration setups and entail time-consuming operational procedures. As a result, they are poorly suited for online calibration scenarios that require real-time adaptability. In contrast, targetless calibration schemes offer distinct application advantages for online implementation, with their operational workflow illustrated in Figure 3.
Figure 3.
Schematic diagram of online feature-based extrinsic calibration of camera and LiDAR.
The feature-based targetless calibration method needs to identify the co-observed features of the image and the point cloud to solve the extrinsic parameters. Co-observed features include motion characteristics and environmental elements such as points, lines, and planes. The motion-based calibration method inversely solves the calibration parameters through a series of movement sequences of the sensor, but the calibration accuracy is usually not high due to the limitation of trajectory error. Motion-based calibration methods acquire motion sequences using the odometry of each sensor separately. They inversely solve the calibration parameters through the alignment of these motion sequences via the Iterative Closest Point (ICP) [18] or Similarity Transformation of Three Pairs (Sim(3)) [19] matching algorithm. However, due to inherent drawbacks of odometry such as trajectory drift, the calibration accuracy of such methods is generally unsatisfactory. The calibration algorithm based on environmental features constructs geometric constraints through the common view features to solve the transformation relationship between the two sensors. This algorithm exhibits strict dependencies on the initial extrinsic parameters, calibration scene characteristics, and spatial distribution of environmental features, thereby leading to inherent robustness limitations in diverse operational scenes.
In summary, the two aforementioned methods each exhibit complementary advantages in dynamic and complex real-world environments. On this basis, a cascade calibration algorithm can be designed. Specifically, the algorithm leverages a motion-based method to provide initial extrinsic parameter estimation in real time. Subsequently, it adopts the feature-based method to conduct precise estimation of the extrinsic parameters. The cascade framework ultimately achieves online targetless extrinsic calibration with high accuracy and robustness.
3. Methodology Design
3.1. Overall Process of the Methodology
The calibration of cascade optimization integrates motion trajectory alignment and edge feature matching.
The proposed algorithm addresses two key limitations of existing algorithms, namely, suboptimal scene adaptability and convergence instability. Specifically, it employs a coarse-to-fine calibration strategy for varying adaptability scenes. Meanwhile, it implements cascade optimization on the decoupled rotation and translation parameters using different types of feature pairs to mitigate convergence instability. The detailed algorithmic workflow is depicted in Figure 4.
Figure 4.
Processing flowchart of LiDAR and camera cascaded optimization online calibration algorithm. Blue boxes and lines represent the data processing flow, orange ones denote matching and optimization, and purple ones indicate the results of each module and extrinsic parameters to be optimized.
The calibration workflow primarily consists of three key modules: initial calibration based on motion trajectories, edge extraction and feature classification, and cascade optimization on the decoupled parameters relying on different types of features. In the initial calibration, the LiDAR’s motion trajectory is calculated by the lightweight LeGO-LOAM [20], and ORB-SLAM2 [21] is used to calculate the visual odometry to obtain the motion trajectory without scale. After assessing the trajectory morphology, the initial extrinsic parameters are determined by minimizing the trajectory matching residuals. In image edge extraction, an enhanced Canny operator is employed to generate a binary image that solely retains prominent edge information. For edge extraction from low-resolution point clouds, a joint extraction method combining depth differences and planar edges is adopted. It effectively enhances the recognition efficiency of building edges with subtle curvature, such as windowsills and corners. Finally, a cascaded optimization method for decoupling extrinsic parameters is proposed to further mitigate the impact of the uneven distribution of edge features on the convergence stability of the extrinsic parameters. It achieves fast and precise extrinsic calibration by utilizing the constraint capability of directional features in different orientations on extrinsic parameters.
The proposed algorithm is designed to meet the demand for real-time online application of camera–LiDAR calibration in vehicle-mounted intelligent perception scenarios. Its adaptability and automation run through the entire calibration pipeline, which is highly consistent with the engineering requirements of online calibration.
- Data processing: The algorithm does not require offline or pre-processing of perception data. Relying on the real-time frame streams captured by the camera and LiDAR, it can perform end-to-end incremental processing including trajectory extraction, edge feature detection and matching, and extrinsic parameter optimization, enabling calibration to proceed synchronously with data acquisition.
- Execution logic: The algorithm adopts a targetless automatic calibration framework that requires neither human intervention nor a dedicated calibration environment. It can be triggered automatically when the trajectory variation evaluation passes, without stopping the system for calibration, thus adapting to the dynamic operating characteristics of vehicle-mounted systems.
- Algorithm design: The cascaded optimization framework decomposes the original high-dimensional coupled extrinsic optimization problem into low-dimensional subproblems, which significantly reduces the computational complexity of core operations such as Jacobian inversion and Hessian matrix decomposition. Meanwhile, operations including point cloud field-of-view cropping and image HSV filtering achieve lightweight computation for the feature extraction module, allowing for the algorithm to converge rapidly on conventional hardware.
The above characteristics in terms of data stream processing, execution logic, and algorithm design together endow the proposed algorithm with the core attribute of real-time online calibration, which can effectively satisfy the dynamic online recalibration demand of vehicle-mounted sensors for intelligent driving.
3.2. Trajectory-Consistency-Based Calibration
The LiDAR and the camera are installed on the same movable platform. The initial extrinsic parameters can be calculated by aligning the trajectory information of the two at the same time stamp.
The lightweight LeGO-LOAM for ground vehicle applications is used to obtain the LiDAR trajectory. LeGO-LOAM calculates the frame-to-frame pose transformation by segmenting the ground point cloud and matching the plane features of the ground point cloud with the edge features of the non-ground point cloud in steps. If the angle between the vector formed by two points in the point cloud and the horizontal plane is very small (at approximately 10°), it can be approximately considered a ground point. Let the two points be and ; then the angle calculation is
where is the angle between the horizontal plane and the vector composed of and .
After the ground point segmentation, the non-ground points are further clustered, and the simple and effective filtering process is realized by removing the smaller point cloud clusters. The plane features of the ground point cloud and the edge features of the non-ground point cloud are selected according to the curvature. Taking the point to be selected in the point cloud frame as the center, a certain number of points around are taken to form a set to calculate the curvature :
where counts the number of points in the set .
The points with large curvature values are classified as edge features, and the points with small curvature values are classified as plane features. The feature matching process is essentially a projection process. Specifically, the relative pose parameters of the -th frame with respect to the previous frame are estimated by minimizing the point-to-line (edge) residuals and the point-to-plane residuals, as follows:
where and represent the residual distances from the point-to-line and the point-to-plane, respectively. denotes the index of the residual distance. Based on , the -th frame pose can be calculated by combining the historical trajectory points:
where is the point cloud odometry trajectory point of the -th frame.
The ORB-SLAM2 [21] is used to obtain the camera trajectory. The camera odometry is calculated around the ORB features, combining the accuracy and speed advantages of the FAST key points and BRIEF descriptors. After the feature point matching, the Versatile Perspective-n-Point (VPnP) [22] problem is formulated to estimate the motion of the camera. Assuming that the -th matched feature point pair is and and is the essential matrix containing the rotation and translation information,
where and denote the rotation and translation transformation parameters of the current frame relative to the previous frame and the symbol represents the skew-symmetric matrix.
The essential matrix is solved according to the pixel positions of the matched point pairs, and the initial estimates of the camera motion and are obtained by singular value decomposition (SVD). The initial estimates are optimized by constructing the minimization of the reprojection error problem of the feature points. is the Lie algebra representation of the -th frame camera pose transformation. Assuming that the spatial point coordinate set of -th frame is and its projected pixel coordinate set is , is the feature point number and is the scale information; thus, the relationship between and can be expressed as follows:
where denotes the intrinsic matrix of the camera. Initial estimates of the camera pose are inherently erroneous, and observation noise exists in feature point measurements. Due to the existence of above error, the least squares method is adopted to solve for the optimal pose that minimizes the reprojection error, as follows:
where is the optimal estimate of the -th frame camera pose transformation and the -th frame camera pose can be obtained by combining the historical trajectory points. It should be noted that the normalization process in the calculation makes the translation parameter lack scale information.
After obtaining the trajectory sequences of the LiDAR odometry and the visual odometry, the extrinsic parameter matrix is obtained by performing Sim(3) matching on the trajectory points of the camera coordinate system and LiDAR coordinate system at the same time stamp. Unlike ICP matching, Sim(3) matching incorporates a scale factor within the similarity transformation framework, explicitly accounting for monocular vision’s scale ambiguity. The optimization objective function is
where and represent the trajectory point sequences of the LiDAR and the camera, respectively.
The least squares method is used to solve the optimal value of the extrinsic parameters based on the trajectory consistency. The visual and LiDAR odometry have cumulative errors, and the motion trajectory with a single form (such as a straight line) will lead to the poor convergence accuracy of the rotation extrinsic parameters and unobservability of the translation parameters . Therefore, it is imperative to evaluate the attitude variation in the trajectory to be calibrated, and the average attitude angle change in the window is used to evaluate the trajectory variation:
where is the attitude angle increment of the -th frame. For the ground mobile platform, the pitch angle change can be selected, and is the cumulative attitude angle change threshold. The carrier executes a specific steering maneuver, with the set to 90 degrees. By comparing the average attitude angle change in the window with the threshold, it is judged whether the current trajectory contains a significant turning form so as to determine whether it can be used for extrinsic parameter calculation. Figure 5 illustrates the extrinsic parameter estimation process grounded in trajectory consistency, where the red and green lines denote the camera and LiDAR trajectory sequences, respectively.
Figure 5.
Trajectory-consistent initial extrinsic parameter calibration processing result diagram. (a) Initial trajectory sequence. (b) Camera sequence scale recovery. (c) Camera and LiDAR trajectory pose consistency.
As depicted, the original trajectories exhibit pronounced scale and attitude disparities due to the scale uncertainty of monocular vision and sensor-specific motion characteristics. Through Sim(3) similarity transformation, the two trajectory sequences are effectively overlapped without obvious trajectory distortion and deviation. By incorporating trajectory variation evaluation, the proposed method significantly strengthens the constraint on rotational parameters. Motion-based calibration is less susceptible to scene feature adaptability [9] and can provide reliable and robust initial estimates of extrinsic parameters for the subsequent fine-grained calibration stage in real time.
3.3. Edge Extraction in the Transform Domain
Edge extraction for images and point clouds constitutes a fundamental data processing step for edge matching in the calibration. The core objectives are to extract robust edge features from the environment and establish the spatial relative pose constraint between point cloud features and image features. Ultimately, the optimal estimation of the camera–LiDAR extrinsic parameters is obtained via spatial consistency-based accurate registration.
The well-established Canny algorithm [23] is commonly employed for image edge extraction. Due to the inherent characteristics of sensor data perception, the edges retrieved from images typically exhibit greater richness compared to the structural edges present in point clouds [24]. Moreover, these image edges are highly sensitive to elements such as stray textures and shadow edges in the scene. Due to factors such as illumination and shadows, images contain major irregular edges and stray textures, as shown in Figure 6. These edges interfere with the correlation of spatial structural information between images and point clouds, thereby degrading the performance of edge consistency alignment [24].
Figure 6.
Interference of unstructured edges such as stray textures and shadows in the image. (a) Original image information of the calibration scene. (b) Edge feature extraction result of the Canny operator.
The HSV (Hue, Saturation, Value) threshold filtering method is used to reduce the influence of the above stray textures, shadows, and other irregular edges. The HSV color space decouples RGB information into three independent channels of hue (H), saturation (S), and value (V), which ensures the stability of color features in scenes with fluctuating illumination [25]. In addition, filtering on the channel can eliminate interference regions with low saturation, such as the sky and white walls. The shadow and stray texture regions are defined as follows:
where are empirical coefficients according to reference [25].
Threshold-based segmentation is performed on the HSV filtering result to generate a mask. Subsequently, the Canny edges located within the mask region and four pixels of the outer contour are removed. This enables the more reliable extraction of image edge features and reduces the probability of mismatches in nearest neighbor association between 3D structural features and 2D image edges.
The edge extraction results are illustrated in Figure 7. Through image masking processing in the HSV color space, the interference of unstructured factors such as leaf and shadow boundaries on edge extraction is significantly reduced.
Figure 7.
Image edge extraction result after HSV threshold filtering. (a) Threshold segmentation result. (b) Results of outer contour removal based on the segmentation results.
3.4. Point Cloud Edge Extraction and Classification
To improve the efficiency of point cloud edge feature extraction, the field of view (FOV) of the LiDAR is roughly aligned with that of the camera with the extrinsic parameter . The alignment enables the cropping of point cloud data within the camera’s FOV, thereby effectively reducing the computational cost of point cloud edge feature extraction.
The point cloud is projected onto the two-dimensional image using (2). The homogeneous coordinates of the projected points on the image plane form a two-dimensional point set, denoted as , and the point cloud outside the image field of view after projection is removed. The point set covered by the image is retained:
where is the pixel width and height of the image and is the index of projected points. According to the point set within the image field of view, the corresponding three-dimensional point cloud can be indexed. In that case, the subsequent point cloud edge extraction is only for , reducing the computational cost and the interference of other point cloud edges outside the field of view.
The point cloud edges mainly include the object surface contour and the edge where the surfaces intersect. For the cropped point cloud, the vertical edges in the scene point cloud can be calculated using the depth discontinuity. The calculation of the point cloud depth difference is as follows:
And normalized depth difference is
where represents the distance from the point to the LiDAR. The normalized depth difference is used as the edge attribute value of the point . is the normalized depth difference threshold, which is usually set to one-fourth or one-third of . The points with non-zero edge attribute values are extracted as the edge point cloud set .
The depth discontinuity of the point cloud has an obvious effect on the edge feature extraction of slender objects in the scene, such as the regions with significant depth differences around lamp poles and tree trunks. In practical applications, due to the point cloud sparsity, it is difficult for the depth discontinuity method to robustly extract the edges of dense point clouds around corners and window frames. Considering above issue, the proposed method supplements the deficiency of edge extraction by extracting the plane edges of the point cloud. First, three non-collinear points are randomly selected in the original point cloud to calculate the normalized equation of the corresponding plane:
denote the normalized equation parameters of the plane. The distance from the selected point cloud to the plane is calculated as follows:
Whether or not a point belongs to the plane is determined by its distance to the plane. The points within the allowable error range (the settings are as in [26]) are regarded as the inliers of the plane, and the number of inliers is counted. After multiple iterations, the plane parameters corresponding to the maximum inlier number are selected as the plane model. Finally, the inliers in the plane are removed from , and the remaining points are continuously fitted to extract other planes.
After obtaining all the plane point clouds, the Angle-Criterion [27] (AC) method is used for boundary extraction for each cloud. The AC method leverages the neighborhood property of edges, which implies that there necessarily exist other points belonging to the same edge in the neighborhood of an edge point and that the edge points within the same neighborhood are distributed exclusively on one side of the current edge point. To mitigate the bias of neighborhood centers toward high-density directions (caused by uneven point cloud distribution), a symmetry-based strategy is adopted to construct the -neighborhood.
First, the -nearest points are searched around the center point, forming the main circular neighborhood point set . Then, a neighborhood search using a circle with the radius ε is performed, and the resulting -neighborhood point set is denoted as . Then, based on the relationship between adjacent neighborhood point sets, the neighborhood of the points in the region with a sharp change in point cloud density is also added to the neighborhood to form a symmetric -neighborhood:
This means that when the point (in the neighborhood ) belongs to the -neighborhood point of the point (in the neighborhood ), is regarded as the neighborhood point of . The construction process is shown in Figure 8.
Figure 8.
Neighborhood construction flowchart for AC edge extraction.
After constructing the KDTree neighborhood point set , the edge points are determined by the included angle criterion. For the neighborhood point set corresponding to non-edge regions, the included angle distribution within this set is more uniform. In contrast, larger included angles are present in the neighborhood point set that belongs to edge regions. Taking the nearest neighbors of the point in the point set , connecting them pairwise with forms a series of included angles . The maximum value is found, and the probability that the point is an edge point is calculated:
where , and represents the point number of neighborhood point sets.
The edge points obtained according to the probability can be divided into horizontal edge feature , vertical edge feature , and other edge feature , as shown in Figure 9. According to the different constraints of the features on the pose elements, in the feature-matching optimization process, constraints are established for different extrinsic parameters using features with distinct directional distributions.
Figure 9.
Edge extraction in the real scene.
In the feature classification process, the edge lines are segmented via edge point clustering to construct KDTree for searching and processing on clustering region. For each cluster, the principal axis vector is obtained by principal component analysis (PCA), and the principal axis direction is the direction of the edge line. First, the point cloud is decentralized:
The covariance matrix of the point cloud is calculated and the SVD is performed to determine the principal direction of the point cloud. The direction of the eigenvector corresponding to the maximum eigenvalue is selected as the principal direction of the point cloud:
Therefore, the principal direction vector of the point cloud is equal to the unit eigenvector corresponding to . Thus, the angle between the principal direction vector and the normal vector of the X–Y plane (the Z axis) can be calculated, where the normal vector denotes . When the angle between the principal axis and the normal vector is close to a right angle, the edge line is parallel to the X–Y plane and is recorded as a horizontal edge feature . When the angle between the principal axis and the normal vector is close to 0, the edge line is parallel to the normal vector and is recorded as a vertical edge feature . Then the remaining edge lines are recorded as other edge features . The specific process is shown in Algorithm A1 in the Appendix A.
3.5. Cascaded Extrinsic Parameter Optimization
In the refinement calibration, a cascaded reprojection optimization method is proposed by leveraging the edge feature classification results. This method exploits the differential constraint capabilities of features with different principal directions on each dimension of rotation–translation parameters [11].
Firstly, coarse-matching performed on the other edge feature is used to roughly align the observation fields of view of the camera and the LiDAR, obtaining the optimized translation parameters. Other edge features incorporate non-directional and more uniformly distributed scattered point features into the construction of constraints. Positional offsets induced by position changes (i.e., translations) will significantly alter the relative pose of all objects in the scene within the observation field of view [28]. The positional offsets of other edge features can be clearly captured, thus yielding higher observability of matching residuals.
In the optimization of rotation parameters, horizontal edge features and vertical edge features are used to optimize the vertical-direction extrinsic parameters (yaw angle and roll angle) and horizontal-direction extrinsic parameters (pitch angle), respectively.
Taking the horizontal edge features (where the length in the horizontal direction is much greater than the width in the vertical direction) as an example, when an equivalent observation angle deviation occurs in both the vertical and horizontal directions (i.e., vertical and horizontal residuals arise), the relative change in the residual in the vertical direction () is much larger than that in the horizontal direction (). This indicates that the parameters in the vertical direction (roll and yaw angle) are more sensitive to, and constrained by, horizontal features.
The fine-calibration loop sequentially executes the cascaded optimization described in Equations (22) and (23), and finally obtains a stably converged estimation result of the extrinsic parameters. The feature cascaded optimization process is illustrated in Figure 10.
Figure 10.
Cascaded optimization uses different principal orientation features to constrain extrinsic parameters in different directions.
In engineering practice, the solution process of the optimization function composed of reprojection errors such as Equations (22) and (23) is divided into three stages: KD-Tree nearest neighbor search for feature pairs, iterative transformation solving, and transformation application. For an optimization problem with a total number of features (here, it is assumed that the number of image features and point cloud features are both for ease of calculation), the complexity of constructing a KD-Tree for the target point cloud in 3D space is [29]. In a single iteration of transformation solving, the complexity of performing a nearest neighbor search for points via the KD-Tree is . Combined with the linear complexity of centroid calculation, SVD-based rigid transformation solving, and point cloud transformation, the total complexity of a single iteration is . Let the number of iterations required for convergence be a constant ; then the total complexity of the entire process is .
After feature classification, assuming the numbers of features in categories , , and are , , and respectively, satisfying , the computational complexity of the KD-Tree nearest neighbor search and a single iteration is equal to , which satisfies
Since the objective function of low-dimensional subproblems is smoother, the coupling interference between parameters is eliminated, which effectively avoids iterative oscillation and reduces the number of iterations required for the convergence of cascaded optimization being much smaller than (i.e., ). Therefore, the computational complexity of cascaded parameter optimization is much smaller than that of direct optimization . Furthermore, the feature classification optimization strategy has inherent parallel processing characteristics, exhibiting promising potential for parallel acceleration when deployed on vehicle-mounted embedded GPUs/AI chips. The three factors synergistically achieve a significant improvement in computational efficiency.
4. Experimental Verification
The online calibration algorithm proposed in this paper is verified with the data from the KITTI-2011-10-03-0027 dataset, S3E dataset, and TUMTraf A9 Highway Dataset. All experiments were conducted on the robot operating system (ROS) in Ubuntu 20.04, and computing platform was equipped with a 3rd Gen Intel(R) Core(TM) i9-13900H CPU and 32 GB RAM. The KITTI-2011-10-03-0027 dataset was employed for internal validation to verify the accuracy and stability of the proposed algorithm, as well as the effectiveness of the cascaded optimization. The S3E dataset [30] and TUMTraf A9 Highway Dataset [31] were used for horizontal comparison to evaluate the generalization ability and calibration accuracy against existing network-based and semantic segmentation-based methods.
The total length of the KITTI-2011-10-03-0027 trajectory was 3744.0 m, and the continuous measurement time was 470.58 s. As shown in Figure 11, The overall environment had the following characteristics:
Figure 11.
Various scenes of the KITTI dataset.
- This segment of data belonged to the “residential area” part. The vehicle speed was relatively low, and there are many sudden stops.
- There were many houses and trees of various shapes on both sides of the road, which was quite friendly for the selection and matching of image and point cloud feature points.
- During the vehicle’s driving process, due to the narrow road and the large number of pedestrians and vehicles, the LiDAR occluded quite frequently.
Due to the occlusion, there are obvious changes in light and darkness, and overexposure also exists in some sections.
Both the S3E and TUMTraf A9 Highway Dataset are widely used in the fields of vision/LiDAR scene perception, object recognition, and localization and navigation. They collect visual images and LiDAR point clouds via mobile UAVs/vehicles in typical urban driving scenarios such as traffic roads and urban blocks. In addition, the scenes contain numerous dynamic interfering objects, such as pedestrians and motor vehicles, which pose challenges to the robustness and generalization ability of the algorithm. Meanwhile, due to environmental factors, including tree shade and building occlusion, the illumination conditions in the scenes of S3E dataset undergo drastic dynamic variations. Since the TUMTraf A9 Highway Dataset introduces a novel observation paradigm where vehicles on the road are observed from a tilted downward perspective via traffic sign poles, the proposed algorithm initializes the rotation matrix as the identity matrix and the translation vector as the zero vector, and performs fine calibration directly.
In the accuracy evaluation of the experiments, mean squared error (MSE) was adopted to calculate the rotation error and translation error . The formula is given as follows:
where denote the calibration results and denote the ground-truth values.
4.1. Trajectory-Consistency-Based Calibration Results
The selected data are subjected to hierarchical coarse-matching and fine-matching. First, the LiDAR odometry trajectory and visual odometry trajectory of five selected data were obtained using the algorithms LeGO-LOAM and ORB-SLAM2. The trajectory-consistency-based calibration was performed on the odometry trajectory data of the selected road section. The trajectory-matching results are shown in Figure 12.
Figure 12.
Trajectory-matching results of visual odometry and LiDAR odometry. By comparing the five groups of trajectory-matching results, it can be seen that the scale of the visual odometry is accurately restored and two groups of odometry trajectories can be accurately overlapped without significant trajectory divergence.
Meanwhile, the initial calibration quantitative analysis was conducted, as shown in Table 1. The pose initialization by the initial trajectory alignment method has an error between 1° and 1.5° compared to the ground-truth, which can provide a reliable initial value for fine calibration.
Table 1.
Initial calibration error.
4.2. Accuracy of Distributed Reprojection
After obtaining the initial extrinsic parameters, the edge image and edge point cloud are registered by the cascaded optimization. To verify the effectiveness of the proposed optimization method, the direct optimization method (the rotation and translation parameters are optimized simultaneously) is used for comparison on the same data. The accuracy comparison results are shown in Table 2 (averaged for three scenes in each sequence), which describes the deviation degree of calibration results from the true values.
Table 2.
Calibration errors of different optimization methods.
The results in Table 2 show that, compared to the direct reprojection method, the calibration accuracy of the cascaded optimization method has a higher accuracy in each dimension of the rotation error. The overall accuracy can reach that each rotation parameter error is better than 0.25°, and the translation parameter error is better than 0.04 m.
4.3. The Repeatability and Consistency
To assess the consistency of the algorithm, the repeatability of the calibration re-sults within the same scene, as well as the consistency of the calibration results across different scenes within the same sequence and among different sequences, will be se-quentially evaluated, as shown in Table 3.
Table 3.
Calibration error in the same scene.
Taking the frame 000102 in scene 01 as an illustrative case for repeatability verification, the results of multiple experiments consistently exhibit fluctuations around the mean value, with the errors being less than (0.03°, 0.01 m). No outlier singular values are observed, indicating a high level of overall consistency. Subsequently, the deviation values of the calibration result across different scenes within the same sequence are compared. Taking scene 00 as an example, the corresponding results are presented in Table 4.
Table 4.
Calibration result errors of multiple scenes in sequence 00.
Repeated calibration experiments were conducted to demonstrate the stability of the algorithm with the cascaded optimization strategy. The mean errors and population variances of the repeated experimental results are presented in Table 5. Ten independent calibration trials were performed on each scene, and the results are statistically summarized.
Table 5.
Repeated calibration experiments.
Regarding the calibration results of the four sequences, they also fluctuate around the mean value, with the discrepancies remaining within the range of less than (0.03°, 0.01 m). No outlier singular values are detected. The visualization of these results is presented as follows in Figure 13.
Figure 13.
Schematic diagram of calibration results of multiple scenes in sequence 00.
Finally, calibrate different sequences and compare the consistency. The statistical data are as follows in Figure 14.
Figure 14.
Value distribution of the convergence results of direct optimization and cascaded optimization. (a) Convergence of direct optimization. (b) Convergence of cascaded optimization.
The nominal parts are removed from the values shown in the above figure (the 00/02 sequences are from the KITTI-2011-10-03 sequence and the 07/08/10 sequences are from the KITTI-2011-9-30 sequence, and their extrinsic parameters are slightly different). Compared to the direct optimization method, the cascaded optimization strategy achieves significantly narrowed convergence ranges for both yaw and pitch angles. This result demonstrates a pronounced enhancement in the stability of rotational parameters. In particular, the roll angle convergence for sequences 00 and 02 demonstrates moderate enhancement.
4.4. Convergence Efficiency
To quantitatively assess the computational efficiency and real-time capabilities of the proposed feature classification method, a comparative analysis was conducted between the cascaded optimization calibration and the direct optimization calibration. Four representative scenes from the KITTI-00 sequence were carefully selected. After the extraction of edge features, a comprehensive test of the convergence time was carried out.
The convergence condition is usually judged by two metrics: the variation in the optimized extrinsic parameters (rotation and translation) between adjacent iterations, and the decrement of the objective function (e.g., reprojection error or point cloud registration error). The iteration process stops when both the norm of the parameter update and the reduction magnitude of the error are sufficiently small.
As illustrated in Table 6, the Cascaded Optimization method incorporates an additional feature classification stage that includes quadratic fitting clustering of line-feature edges and principal direction decomposition. Despite the introduction of this extra stage, the method achieves a significant reduction in overall time consumption and a substantial acceleration in convergence speed. This is attributed to the fact that distinct features are adopted for the classified optimization of extrinsic parameters. It decomposes the original high-dimensional coupled optimization problem of rotation and translation parameters into low-dimensional subproblems. As a result, the computational complexity of core operations, such as nearest neighbor search, KD-Tree search, and SVD-based rigid transformation solving, is significantly reduced.
Table 6.
Time consumption of direct optimization and cascaded optimization.
Meanwhile, based on the feature sensitivity selection principle, only edge features sensitive to target parameters were selected to participate in the optimization. The decoupled optimization strategy weakens the mutual interference between parameters. It renders the objective function smoother and the parameter update direction more explicit, effectively mitigating iterative oscillations.
4.5. Algorithm Accuracy Comparison
To evaluate the accuracy, robustness, and generalization ability of the proposed algorithm in dynamic scenarios and environments with significant illumination variations, we conducted comparative experiments against existing network-based and segmentation-based algorithms on both the S3E dataset and the TUMTraf A9 Highway Dataset. The comparative algorithms are as follows: The Calib-Anything algorithm [32] leverages the SAM network [33] and achieves training-free scene feature segmentation. Algorithm [34] matches the point cloud image semantic segmentation results through cross-domain registration and the Nelder–Mead simplex optimization algorithm. Each result shown in Table 7 is the average of ten calibration runs across multiple scenes on the dataset, and the visualization calibration results of S3E and TUMTraf A9 dataset are shown in the Figure 15.
Table 7.
Accuracy comparison of various methods.
Figure 15.
Visualization calibration results of the proposed method. (a) Visualization results of calibration on the S3E dataset (edge point cloud projection and all point cloud projection). (b) Visualization results of calibration on the TUMTraf A9 Highway Dataset (edge point cloud projection and all point cloud projection).
While the network-based calibration algorithms [14,15] demonstrate superior performance on the widely adopted KITTI and S3E datasets, their heavy reliance on scenario- and viewpoint-specific pre-training severely limits their generalization ability. Due to the absence of pre-training on such viewpoints, algorithms [14,15] fail to adapt to the new observation distribution of TUMTraf A9 Highway Dataset, resulting in significant performance degradation or even calibration failure. For semantic segmentation-based extrinsic calibration algorithms, the calibration performance is closely correlated with the number of semantic features extracted from the scene. On the S3E dataset, which exhibits significant inter-scene feature heterogeneity, algorithm [32] yields lower calibration accuracy and even fails in some scenes. Meanwhile, algorithm [34] only utilizes edge information of partial semantic objects, leading to insufficient constraints and thus unstable calibration results. Despite not leveraging semantic information, the proposed algorithm extracts rich edge features from the scene and performs classification optimization. It can still stably achieve uniformly excellent calibration results on the S3E dataset with relatively sparse point clouds and the TUMTraf A9 Highway Dataset with significant dynamic interference.
Finally, the time consumption of each calibration algorithm is compared on different datasets to demonstrate the real-time performance of the algorithm, as shown in Table 8. This proves that the proposed method can achieve high-accuracy performance with less time consumption.
Table 8.
Time consumption of different methods.
5. Discussion
This paper employs a coarse-to-fine two-stage calibration strategy to obtain the extrinsic parameters of the camera–LiDAR system. Comparative experiments with multiple mainstream algorithms and the direct optimization calibration method were conducted to verify the calibration efficiency and accuracy of the proposed cascade optimization calibration method. Meanwhile, the drawbacks of current network-based and semantic segmentation-based calibration methods in specific scenarios are briefly analyzed. In the subsequent work, we will combine theoretical tools such as the Fisher Information Matrix (FIM) and Cramér–Rao Lower Bound (CRLB) for pose estimation to quantitatively evaluate the constraint ability of edge features with different orientations on each extrinsic parameter and select the key scene features that contribute most to extrinsic calibration. On this basis, we will optimize the feature utilization mechanism, enabling efficient and accurate extrinsic calibration using only a small number of highly constraining features, thereby further breaking through the performance bottleneck of calibration in sparse-feature scenes. In addition, although Network-based calibration algorithms offer high accuracy and strong robustness, they suffer from significant limitations in cross-scene generalization and struggle to adapt to variations in sensor types (e.g., LiDARs with different channel numbers) and observation perspectives (e.g., vehicle-mounted, pedestrian, and fixed-platform setups). We plan to integrate such high-precision network-based methods with an edge-feature-based cascaded optimization approach, which exhibits superior generalization ability, to achieve stable and accurate extrinsic estimation under diverse scenes and working conditions.
6. Conclusions
Traditional automatic online targetless calibration methods often suffer from poor scene adaptability and the unstable convergence of extrinsic parameters. To tackle these challenges, this paper presents an innovative online calibration approach for camera–LiDAR extrinsic parameters. The proposed method comprehensively analyzes the motion and geometric observation characteristics of both the camera and LiDAR. The method leverages cascade optimization of motion trajectories and edge features, enabling the real-time calculation of extrinsic parameters in complex scenes without relying on artificial markers. Experimental verification using typical urban datasets demonstrate that the proposed method achieves high accuracy and real-time performance, validating its effectiveness in practical applications.
Author Contributions
Conceptualization, C.H. and X.J.; methodology, Z.G. and Z.H.; software, Z.G.; validation, T.Z. and Z.G.; formal analysis, C.H.; investigation, X.J.; resources, Z.H.; data curation, X.J.; writing—original draft preparation, Z.G.; writing—review and editing, Z.H.; visualization, T.Z.; supervision, C.H. All authors have read and agreed to the published version of the manuscript.
Funding
This research was supported by Open Fund of Intelligent Control Laboratory of PLA Rocket Force University of Engineering (No. 2024-ZKSYS-KF02-02).
Data Availability Statement
The original contributions presented in the study are included in the article, further inquiries can be directed to the corresponding author.
Conflicts of Interest
The authors declare no conflicts of interest.
Appendix A
| Algorithm A1. Processing Point Clouds via Euclidean Clustering and PCA Decomposition. |
| Input: Point cloud data matrix , clustering tolerance , minimum cluster size , maximum cluster size Output: Point cloud line features , , , classification results 1: Initialize point cloud 2: Initialize KD—tree 3: Set input point cloud as 4: Initialize Euclidean clustering object 5: Set clustering tolerance 6: Set minimum cluster size 7: Set maximum cluster size 8: Set search method 9: Set input point cloud 10: Execute clustering , get clustering results 11: Initialize dictionary 12: For each cluster do: 13: Initialize the point-set of the current cluster 14: For do: 15: 16: End loop 17: Initialize object 18: Perform on , get the principal axis direction 19: Calculate the angle between the principal axis direction and the plane 20: IF THEN 21: do ; 22: ELSE THEN 23: do ; 24: ELSE 25: do 26: END IF 27: END LOOP 28: Return ,, |
References
- Liu, Y.; Guo, X.; Su, S.; Sun, B. Review of a 3D LiDAR combined with single vision calibration. In Proceedings of the 2021 IEEE International Conference on Data Science and Computer Application (ICDSCA), Dalian, China, 29–31 October 2021; pp. 397–404. [Google Scholar] [CrossRef] [Scilit]
- Huang, Y.; Zhou, J.; Huang, Q.; Li, B.; Wang, L.; Zhu, J. Camera-LiDAR Fusion for Object Detection, Tracking and Prediction. Geomat. Inf. Sci. Wuhan Univ. 2024, 49, 945–951. [Google Scholar] [CrossRef]
- Li, Y.; Hu, H.; Liu, Z.; Xu, X.; Huang, X.; Zhao, D. Influence of Camera-LiDAR Configuration on 3D Object Detection for Autonomous Driving. In Proceedings of the 2024 IEEE International Conference on Robotics and Automation (ICRA), Yokohama, Japan, 13–17 May 2024; pp. 9018–9025. [Google Scholar] [CrossRef] [Scilit]
- Zhang, Z. Flexible camera calibration by viewing a plane from unknown orientations. In Proceedings of the Seventh IEEE International Conference on Computer Vision, Kerkyra, Greece, 20–25 September 1999; Volume 1, pp. 666–673. [Google Scholar] [CrossRef] [Scilit]
- Itami, F.; Yamazaki, T. An Improved Method for the Calibration of a 2-D LiDAR With Respect to a Camera by Using a Checkerboard Target. IEEE Sens. J. 2020, 20, 7906–7917. [Google Scholar] [CrossRef] [Scilit]
- Xie, J.; Lin, X.; Wang, F.; Shi, X.; Yan, L. Extrinsic Calibration Method for LiDAR and Camera with Joint Point-Line-Plane Constraints. Geomat. Inf. Sci. Wuhan Univ. 2021, 46, 1916–1923. [Google Scholar] [CrossRef]
- Dhall, A.; Chelani, K.; Radhakrishnan, V.; Krishna, K.M. LiDAR-Camera Calibration using 3D-3D Point correspondences. arXiv 2017, arXiv:1705.09785. [Google Scholar] [CrossRef] [Scilit]
- Shi, P.; Wei, K.; Wu, H.; Jie, L. External Parameter Calibration Method of LiDAR and Camera Based on Semantic Segmentation in Automatic Driving Environment. Laser Optoelectron. Prog. 2024, 61, 2428011. [Google Scholar]
- Zhao, H.; Ji, X.; Wei, D. GLIC-Calib: Targetless and One-Shot Spatial-Temporal Calibration of LiDAR-IMU-Camera for Ground Vehicles. In Proceedings of the 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Hangzhou, China, 19–25 October 2025; pp. 17964–17971. [Google Scholar] [CrossRef] [Scilit]
- Levinson, J.; Thrun, S. Automatic Online Calibration of Cameras and Lasers. In Proceedings of the Robotics: Science and Systems, Berlin, Germany, 12 June 2013. [Google Scholar] [CrossRef] [Scilit]
- Yuan, C.; Liu, X.; Hong, X.; Zhang, F. Pixel-level Extrinsic Self Calibration of High Resolution LiDAR and Camera in Target-less Environments. IEEE Robot. Autom. Lett. 2021, 6, 7517–7524. [Google Scholar] [CrossRef] [Scilit]
- Liu, L.; Stamos, I. Automatic 3D to 2D registration for the photorealistic rendering of urban scenes. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05); IEEE: New York, NY, USA, 2005. [Google Scholar] [CrossRef] [Scilit]
- Koide, K.; Oishi, S.; Yokozuka, M.; Banno, A. General, Single-shot, Target-less, and Automatic LiDAR-Camera Extrinsic Calibration Toolbox. In Proceedings of the 2023 IEEE International Conference on Robotics and Automation (ICRA), London, UK, 29 May–2 June 2023; pp. 11301–11307. [Google Scholar] [CrossRef] [Scilit]
- Schneider, N.; Piewak, F.; Stiller, C.; Franke, U. RegNet: Multimodal sensor registration using deep neural networks. In 2017 IEEE Intelligent Vehicles Symposium (IV); IEEE: New York, NY, USA, 2017. [Google Scholar] [CrossRef] [Scilit]
- Lv, X.; Wang, B.; Dou, Z.; Ye, D.; Wang, S. LCCNet: LiDAR and Camera Self-Calibration using Cost Volume Network. In Computer Vision and Pattern Recognition; IEEE: New York, NY, USA, 2021. [Google Scholar] [CrossRef] [Scilit]
- Wold, S.; Esbensen, K.; Geladi, P. Principal component analysis. Chemom. Intell. Lab. Syst. 1987, 2, 37–52. [Google Scholar] [CrossRef] [Scilit]
- Li, J.; Lee, G.H. DeepI2P: Image-to-Point Cloud Registration via Deep Classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Denver, CO, USA, 2–6 June, 2026. [Google Scholar]
- Segal, A.; Haehnel, D.; Thrun, S. Generalized-ICP. In Proceedings of the Robotics: Science and Systems, Seattle, WA, USA, 28 June–1 July 2009. [Google Scholar]
- Horn, B.K. Closed-form solution of absolute orientation using unit quaternions. J. Opt. Soc. Am. A 1987, 4, 629–642. [Google Scholar] [CrossRef] [Scilit]
- Shan, T.; Englot, B. LeGO-LOAM: Lightweight and Ground-Optimized LiDAR Odometry and Mapping on Variable Terrain. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS); IEEE: New York, NY, USA, 2019. [Google Scholar] [CrossRef] [Scilit]
- Mur-Artal, R.; Tardos, J.D. ORB-SLAM2: An Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras. IEEE Trans. Robot. 2016, 33, 1255–1262. [Google Scholar] [CrossRef] [Scilit]
- Nakano, G. A Versatile Approach for Solving PnP, PnPf, and PnPfr Problems. In Computer Vision—ECCV 2016; Lecture Notes in Computer Science; ECCV 2016; Leibe, B., Matas, J., Sebe, N., Welling, M., Eds.; Springer: Cham, Switzerland, 2016; Volume 9907. [Google Scholar] [CrossRef] [Scilit]
- Canny, J. A Computational Approach to Edge Detection. IEEE Trans. Pattern Anal. Mach. Intell. 1986, 8, 679–698. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Liu, H.; Xu, Q.; Huang, Y.; Ding, Y.; Xiao, J. A Method for Synchronous Automated Extrinsic Calibration of LiDAR and Cameras Based on a Circular Calibration Board. IEEE Sens. J. 2023, 23, 25026–25035. [Google Scholar] [CrossRef] [Scilit]
- Smith Ray, A. Color gamut transform pairs. Acm Siggraph Comput. Graph. 1978, 12, 12–19. [Google Scholar] [CrossRef] [Scilit]
- Fischler, M.A.; Bolles, R.C. (Eds.) Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography; Morgan Kaufmann: San Francisco, CA, USA, 1981; pp. 381–395. [Google Scholar]
- Bendels, G.H.; Schnabel, R.; Klein, R. Detecting Holes in Point Set Surfaces. J. WSCG 2006, 14, 89–96. [Google Scholar]
- Zhou, L.; Li, Z.; Kaess, M. Automatic Extrinsic Calibration of a Camera and a 3D LiDAR Using Line and Plane Correspondences; IEEE: New York, NY, USA, 2018. [Google Scholar] [CrossRef] [Scilit]
- Bentley, J.L. Multidimensional binary search trees used for associative searching. Commun. ACM 1975, 18, 509–517. [Google Scholar] [CrossRef] [Scilit]
- Feng, D.; Qi, Y.; Zhong, S.; Chen, Z.; Chen, Q.; Chen, H.; Wu, J.; Ma, J. S3E: A Multi-Robot Multimodal Dataset for Collaborative SLAM. IEEE Robot. Autom. Lett. 2024, 9, 11401–11408. [Google Scholar] [CrossRef] [Scilit]
- Zhou, X. Tumtraffic-VideoQA: A Benchmark for Unified Spatio-Temporal Video Understanding in Traffic Scenes. arXiv 2025, arXiv:2502.02449. [Google Scholar] [CrossRef] [Scilit]
- Luo, Z.; Yan, G.; Li, Y. Calib-Anything: Zero-training LiDAR-Camera Extrinsic Calibration Method Using Segment Anything. arXiv 2023, arXiv:2306.02656. [Google Scholar]
- Kirillov, A.; Mintun, E.; Ravi, N.; Mao, H.; Rolland, C.; Gustafson, L.; Xiao, T.; Whitehead, S.; Berg, A.C.; Lo, W.-Y.; et al. Segment Anything; IEEE: New York, NY, USA, 2023. [Google Scholar] [CrossRef] [Scilit]
- Tsaregorodtsev, A.; Müller, J.; Strohbeck, J.; Herrmann, M.; Buchholz, M.; Belagiannis, V. Extrinsic Camera Calibration with Semantic Segmentation. arXiv 2022, arXiv:2208.03949. [Google Scholar] [CrossRef] [Scilit]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2026 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license.














