A Pseudo-Point-Based Adaptive Fusion Network for Multi-Modal 3D Detection

Abstract

A 3D multi-modal detection method using a monocular camera and LiDAR has drawn much attention due to its low cost and strong applicability, making it highly valuable for autonomous driving and unmanned aerial vehicles (UAVs). However, conventional fusion approaches relying on static arithmetic operations often fail to adapt to dynamic, complex scenarios. Furthermore, existing ROI alignment techniques, such as local projection and cross-attention, are inadequate for mitigating the feature misalignment triggered by depth estimation noise in pseudo-point clouds. To address these issues, this paper proposes a pseudo-point-based 3D object detection method that achieves biased fusion of multi-modal data. First, a meta-weight fusion module dynamically generates fusion weights based on global context, adaptively balancing the contributions of point clouds and images. Second, a module combining bidirectional cross-attention and a gating filter mechanism is introduced to eliminate the ROI feature misalignment caused by depth completion noise. Finally, a class-agnostic box fusion strategy is introduced to aggregate highly overlapping detection boxes at the decision level, improving localization accuracy. Experiments on the KITTI dataset show that the proposed method achieves APs of 92.22%, 85.03%, and 82.25% on Easy, Moderate, and Hard difficulty levels, respectively, demonstrating leading performance. Ablation studies further validate the effectiveness and computational efficiency of each module.

1. Introduction

3D object detection [1] is a fundamental capability for autonomous systems, particularly for unmanned ground vehicles (UGVs) and unmanned aerial vehicles (UAVs, or drones). It enables these platforms to accurately perceive and understand the three-dimensional structure of their operating environments. Robust 3D perception is essential for UGVs and UAVs [2,3] to perform core tasks such as obstacle avoidance and trajectory planning, and to operate safely in dynamic urban settings. Within this framework, multi-modal fusion particularly integrating monocular visual data with LiDAR geometric information represents a robust and efficient strategy for reliable perception under stringent real-world constraints. While multi-camera panoramic systems can offer a 360-degree field of view, they tend to be expensive and computationally demanding, and require strict synchronization. In contrast, the fusion of a monocular camera with LiDAR provides a balanced and resource-efficient solution. This multi-modal framework is particularly suitable for UGVs and UAVs operating under payload, power, and cost limitations. It supports a variety of applications such as urban air mobility, infrastructure inspection, and precision agriculture. Moreover, it meets the perception requirements of UGVs navigating complex ground scenarios, as well as drones operating in cluttered aerial environments. As a result, research into monocular camera and LiDAR fusion has garnered increasing interest from both the UGV and drone communities. This fusion strategy offers a scalable and practical pathway toward high-performance 3D perception, enhancing the operational capabilities of autonomous systems across both ground and aerial domains.

In contrast, fusion methods using a front-view monocular camera and LiDAR [4,5,6,7,8] offer a more general and low-cost configuration. This makes them highly valuable for practical applications. Nevertheless, this configuration faces unique challenges. The monocular camera lacks depth information and its images suffer from perspective distortion. LiDAR provides accurate but sparse 3D point clouds. There is a large gap between these two heterogeneous data representations. Therefore, achieving deep complementarity and alignment between them in the front-view direction is a core problem for improving perception system performance.

Current fusion methods are mainly divided into projection-based fusion and model-based fusion. Projection-based fusion relies on camera calibration parameters to project features from one modality into the space of another. Since projection is a type of hard association, when sparse LiDAR points are projected onto dense image pixels, each point can only associate with a limited number of pixels. This makes it difficult to fully utilize the rich contextual semantic information in the image. Taking Frustum-PointNet [9] as an example, it first reduces the 3D search space using 2D detection boxes. However, this restricts fusion to the interior of the 2D box, potentially losing useful contextual information outside the box. Moreover, its performance is highly dependent on the accuracy of the 2D detector. Model-based fusion methods use deep learning models to adaptively learn associations between features of different modalities. They employ specific backbone networks for images and point clouds separately—for example, ResNet for image feature extraction, and PointNet [10] or VoxelNet [11] for point cloud feature extraction. The extracted heterogeneous features are then fused to preserve as much information as possible. This approach is computationally intensive and challenging, as it requires aligning features with different structures. We adopt a pseudo-LiDAR-based fusion framework. It converts images into pseudo point clouds and operates in a unified 3D space. This alleviates the heterogeneity issue and avoids the geometric information loss caused by projection to the 2D view. The fused data can be fed into any high-performance pure point cloud 3D detector, offering good compatibility.

Moreover, existing fusion methods [9,12,13,14,15,16,17,18] often underperform in front-view monocular settings, primarily because they fail to coherently address three key challenges. First, commonly adopted static fusion strategies such as simple addition or concatenation overlook the varying reliability of each modality across different scenarios. For example, EPNet++ [8] employs a complex yet fixed multi-stage fusion process, while BEVFusion [19] lifts image and point cloud features into a unified BEV space and fuses them in a unidirectional, predefined manner. Second, many approaches neglect fine-grained feature fusion at the ROI level. Methods like Voxel-RCNN [20] conduct one-time feature alignment and fusion during early feature extraction, leaving subsequent region-level feature misalignment unresolved. Alternatively, current ROI-aligned fusion techniques such as local mapping and cross-attention struggle to handle the noisy pseudo-LiDAR point clouds in our architecture. Third, at the decision level, prevalent post-processing techniques are inadequate. Non-Maximum Suppression (NMS) suffers from its greedy suppression mechanism, which discards valuable overlapping predictions. While Weighted Box Fusion (WBF) improves upon this by aggregating boxes, its framework is designed for fusing independent models. WBF lacks a guidance mechanism, resulting in a conservative localization estimate through naive averaging.

To address these challenges, this paper proposes an adaptive network specifically designed for front-view monocular camera and LiDAR fusion. The network contains three modules: (1) a Meta-Weight Fusion module (MWF) that adaptively balances modal contributions at the global level; (2) a Cross-modal Attention alignment module (CMA) that tackles feature misalignment at the regional level; (3) a Class-agnostic 3D Box Fusion (C3BF) strategy that refines the final output at the decision level. Specifically, the MWF module uses a lightweight meta-network to analyze the global features of the input in real time. It dynamically generates optimal fusion weights. The CMA module addresses feature misalignment between point cloud and pseudo-point cloud sources at the ROI level, which is caused by model noise. It employs a bidirectional cross-attention mechanism. This allows features from each modality to query relevant information from the other modality. Thus, deep feature interaction and alignment are achieved. A subsequent adaptive gating mechanism further selects and enhances the most discriminative features after alignment. This ensures geometric structures effectively complement detailed information. The C3BF strategy adopts the idea of Weighted Box Fusion. It treats highly overlapping detection boxes as different observations of the same real object. It aggregates the advantages of multi-modal predictions. This makes the final output 3D bounding boxes more stable and precise.

The main contributions of this paper are as follows:

1.

We propose a lightweight Meta-Weight Fusion module to balance the contributions of LiDAR and pseudo-point features, overcoming the rigidity of conventional fixed-weight fusion methods.

2.

A Cross-modal Attentive Alignment module is designed. It employs a bidirectional cross-attention mechanism and an adaptive gating strategy to effectively mitigate feature misalignment caused by depth completion noise.

3.

A Class-agnostic 3D Box Fusion strategy is designed as a robust post-processing step. By aggregating highly overlapping predictions through confidence-based weighting, it yields more accurate and stable 3D bounding boxes.

4.

Extensive experiments on the KITTI dataset demonstrate that our approach achieves state-of-the-art performance across all difficulty levels. Ablation studies confirm that the significant performance gains are achieved with only a moderate increase in parameters.

2. Related Work

To clearly summarize the limitations of the existing LiDAR-based and Multimodal 3D object detection techniques discussed in this section and motivate our proposed work, a detailed comparison is provided in Table 1.

Table 1. Limitations in Existing 3D Detection Techniques.

Technology Type	Specific Method	Limitation	Reference
	Point-based	Uses geometric structure only, lacks rich semantic and texture information from images.	[10,21,22,23]
LiDAR-based 3D Detection	Voxel-based	Spatial resolution is limited by voxel size, potentially losing fine-grained point cloud geometric details.	[11,24,25]
	Projection-view-based	Loses 3D geometric information; distant objects are less distinct in Range View.	[26,27,28]
	Hard Fusion	Wide search space of point clouds; corrupts semantic information; time-consuming.	[2,29]
Multimodal 3D Detection	Image-Point Fusion	Heterogeneous data making feature alignment difficult; high computational cost.	[19,30,31]
	Pseudo-Point-based	Noise from depth estimation and feature extraction leads to feature misalignment.	[32,33]

2.1. LiDAR-Based 3D Object Detection

LiDAR-based 3D object detection methods can be divided into three main categories based on how point cloud data is represented. These are point-based, voxel-based, and projection-view-based methods. Point-based detection methods process raw point clouds directly. This preserves the geometric structure. PointNet [10] and PointNet++ [21] use shared multi-layer perceptrons and symmetric functions to handle unordered point sets. PointRCNN [22] generates 3D proposals from point clouds in a bottom-up manner and then refines them. 3DSSD [23] introduces a fusion sampling strategy and proposes a single-stage, anchor-free detector to improve speed. Voxel-based methods divide the point cloud into a regular 3D grid. VoxelNet [11] pioneered this approach by introducing Voxel Feature Encoding (VFE) layers for end-to-end detection. SECOND [24] significantly improved the computational efficiency of VoxelNet using sparse convolutions. PointPillars [25] organizes the point cloud into a pseudo-image using the concept of “pillars,” balancing efficiency and information retention. Projection-view methods project the point cloud onto a 2D plane. PIXOR [26] projects the point cloud into a Bird’s Eye View (BEV) for detection. MVF [27] fuses features from the BEV projection and the Range View (RV) of the LiDAR point cloud. Range view methods, like LaserNet [28], perform detection directly on the range view projection of the point cloud.

2.2. Multimodal 3D Object Detection

Multimodal 3D object detection aims to fuse the precise geometry from LiDAR with the rich semantic texture from cameras. Early methods adopted hard fusion. For instance, Shi et al. [2] proposed a LiDAR and camera data fusion framework for UAV platforms, directly connecting the 3D point cloud tensor generated by LiDAR and RGB images to the 6-channel input of the YOLO network, which improves the processing speed. However, the dynamic target processing capability is limited, the flexibility is insufficient, and it struggles with fast-moving targets. Recent research focuses on achieving more efficient and refined cross-modal fusion. Fusion strategies based on the Bird’s Eye View (BEV) have become a mainstream trend. BEVFusion [19] proposed an effective paradigm for separately extracting and fusing camera and LiDAR features within a unified BEV space. Similarly, RCBEVDet [30] designed a RadarBEVNet for radar BEV feature extraction. It also introduced a cross-attention multi-layer fusion module for effective radar-camera fusion. At the fusion architecture level, decision-level fusion methods are noted for their efficiency. CLOCs [31] encodes candidate boxes from 2D and 3D detectors into a sparse tensor. It then learns their geometric and semantic consistency. This approach significantly improves the detection of distant objects. Emerging radar-camera fusion schemes offer potential for low-cost and highly robust perception. HGSFusion [34] enhances radar point cloud density via a radar hybrid generation module. It also uses a dual-synchronization module for feature fusion. This method demonstrates strong performance under poor lighting conditions.

2.3. Depth Completion

Depth completion research focuses on recovering high-quality dense depth maps from sparse depth measurements, such as LiDAR point clouds. Early iterative spatial propagation networks, like CSPN++ [35], propagate depth information within local neighborhoods iteratively using learnable convolutional affinities. Hybrid architectures combine strengths for more accurate completion. For example, CompletionFormer [36] integrates convolutional layers for capturing local details with Transformers for modeling global context. To handle sparse and non-uniform inputs, methods like SparseDC [37] design uncertainty-based feature fusion modules. These modules dynamically adjust the contribution of local features and global information. At the representation learning level, methods like DeCoTR [38] and TPVD [39] explicitly use 3D geometric information to guide the completion process. They introduce techniques such as 2D-3D cross-attention or tri-view decomposition. Image-guided completion methods fuse RGB information to improve edge accuracy and detail integrity. PDC [40] uses superpixels as boundaries to fuse and complete RGB images with sparse depth images. SparseFuse [32] performs depth completion on image pixels, generating pseudo point clouds for training.

3. Method

3.1. Network Architecture

This section introduces our proposed pseudo-point-cloud-based 3D multi-modal object detector. The overall model architecture is shown in Figure 1. The model first converts a 2D image into a point cloud format using a depth completion module. It then inputs both the pseudo-point cloud and the LiDAR point cloud into a 3D detector for multi-modal fusion and prediction. The model incorporates our three novel modules: (a) the Meta-Weighted Fusion (MWF) module, which regulates the contributions of the point cloud and pseudo-point cloud within the feature extraction backbone; (b) the Cross-Modal Attentive Alignment (CMA) module, which aligns the two modalities at the ROI level to reduce noise; and (c) the Class-Agnostic 3D Box Fusion (C3BF) module, which refines the geometric parameters of highly overlapping boxes at the prediction layer via confidence weighting.

Figure 1. Overall Network Architecture. We design three components for feature fusion, feature alignment, and prediction refinement. A Meta-Weighted Fusion (MWF) module dynamically fuses point cloud and pseudo-point cloud features. A Cross-Modal Attentive Alignment (CMA) module aligns features within Regions of Interest. A Class-Agnostic 3D Box Fusion (C3BF) module refines the predicted 3D bounding boxes.

A popular approach for image-point cloud fusion involves extracting and aligning features from each modality separately. However, aligning these heterogeneous data types is challenging and training is costly. To address this, our method uses a depth estimation model to convert the image into a pseudo-point cloud. This eliminates the heterogeneity gap and improves model efficiency. The depth completion network employs a two-branch architecture, using color and sparse depth as guiding information for dense depth prediction.

$${\widehat{D}}_f(u,v)\&={\displaystyle \frac{e^{C_{cd}(u,v)}·{\widehat{D}}_{cd}(u,v)+e^{C_{dd}(u,v)}·{\widehat{D}}_{dd}(u,v)}{e^{C_{cd}(u,v)}+e^{C_{dd}(u,v)}}}$$

(1)

Here, $C_{cd}$ and $C_{dd}$ denote confidence maps. ${\widehat{D}}_{cd}$ and ${\widehat{D}}_{dd}$ represent the predictions from the color-dominant branch and the depth-dominant branch, respectively. The color-dominant branch aims to predict a dense depth map from a color image. The depth-dominant branch focuses on upsampling a sparse depth map.

Subsequently, the depth map is converted into a point cloud. Given a pixel coordinate $(u,v)$ in the depth map and its corresponding depth value d, the 3D coordinates $(X,Y,Z)$ of the point in the camera coordinate system can be calculated using the following formulas:

$$Z·\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{cccc}{f}_x& 0& c_x& 0\\ 0& f_y& c_y& 0\\ 0& 0& 1& 0\end{array}\right]·\left[\begin{array}{c}X\\ Y\\ Z\\ 1\end{array}\right]$$

(2)

3.2. Meta-Weighted Fusion Module (MWF)

Conventional fusion methods combine features by simple summation or concatenation. Unlike static strategies that overlook scene-dependent modality advantages, our approach dynamically prioritizes features based on their contextual reliability. Specifically, the system amplifies semantic-rich image features or reinforces precise geometric LiDAR data depending on the environmental complexity. To address the limitations of fixed fusion strategies, this section introduces a context-aware Meta-weighted fusion module, as shown in Figure 2. Specifically, the module uses a lightweight meta-network. It generates optimal feature fusion weights based on the global context of the input scene. This enables autonomous allocation of modal contributions. The fusion pathways are deliberately restricted to a convex combination of three basic operations: LiDAR-dominant, Image-dominant and Balanced Average. This design leverages the Mixture-of-Experts (MoE) paradigm. It defines three expert sub-pathways, representing the core decision strategies for multi-modal fusion. This structure ensures dynamic adaptive fusion is achieved while fundamentally minimizing computational overhead and the total parameter count. Specifically, the meta-network learns a minimal set of weights $(w_a,w_b,w_c)$ for these three strategies. This process efficiently approximates the continuous, scene-dependent fusion function, successfully addressing the limitations of conventional fixed-weight methods.

Figure 2. Meta-weighted fusion module.

Global context descriptor extraction. Given the 3D feature tensor $F_{3d}$ from the LiDAR branch and the 2D feature tensor $F_{2d}$ from the virtual point branch, we first apply global average pooling. This operation extracts a global scene representation that is independent of spatial locations.

$$g_{3d}={\displaystyle \frac{1}{H\times W}}\sum _{i=1}^H\sum _{j=1}^W{F}_{3d}(i,j)$$

(3)

$$g_{2d}={\displaystyle \frac{1}{H\times W}}\sum _{i=1}^H\sum _{j=1}^W{F}_{2d}(i,j)$$

(4)

Then, they are concatenated to form a joint scene context vector $s=[g_{3d};g_{2d}]\in R^{C_{3d}+C_{2d}}$, which serves as the input to the meta-network.

Meta-network for dynamic weight generation. This module employs a meta network $\mathcal{M}$ with minimal parameters. Its function is defined as:

$$\mathcal{M}\left(s\right)=\mathrm{Softmax}(W_2·\mathrm{ReLU}(W_1·s+b_1)+b_2)$$

(5)

Here, $W_1\in R^{64\times (C_{3d}+C_{2d})}$ and $W_2\in R^{3\times 64}$ are learnable parameters. The network maps the high-dimensional context information to a low-dimensional, normalized weight space $w=(w_a,w_b,w_c)$. These weights represent the confidence scores for three basic fusion strategies.

Weight-adaptive feature fusion. The final feature fusion is a convex combination of the three basic operations. It is formulated as:

$$F_{fused}=w_a·F_{3d}\oplus w_b·F_{2d}\oplus w_c·{\displaystyle \frac{F_{3d}\oplus F_{2d}}{2}}$$

(6)

Here, ⊕ denotes element-wise addition. This module can automatically adjust the degree of reliance on each modality based on scene complexity. Factors include the sparsity of the LiDAR point cloud and the information richness of the image texture. Thus, it learns a continuous function approximation, rather than making a simple binary choice.

3.3. Cross-Modal Attentive Alignment (CMA)

In 3D multi-modal object detection frameworks based on Region Proposal Networks (RPN), feature alignment at the Region of Interest (ROI) level critically affects detection performance. This is due to the misalignment between pseudo-point clouds and real point clouds. Pseudo-point clouds, derived from images, are dense and texture-rich but geometrically less accurate. In contrast, raw point clouds are geometrically precise but sparse. Their distributions and feature statistics within an ROI differ significantly. Simply concatenating or summing these features at the ROI level causes a modality misalignment problem. This limits the effective use of multi-modal information. To address this, we propose a Cross-Modal Attention Alignment module (CMA). It achieves cross-modal feature alignment and enhancement at the ROI level through bidirectional cross-attention and adaptive gated fusion. As shown in Figure 3, the CMA module consists of two key components. The cross-attention module establishes deep interaction between modalities. The adaptive gating fusion module performs adaptive fusion via feature concatenation and a feature selection gate. The final output is a multi-modal, aligned feature.

Figure 3. Cross-Modal Attentive Alignment.

The ROI features from the two modalities are denoted as $RO{I}_{lidar}\in R^{N\times B\times C}$ and $RO{I}_{image}\in R^{N\times B\times C}$. Here, N is the number of voxels, B is the batch size, and C is the feature channel dimension.

Cross-attention alignment. For the LiDAR-to-image attention path, $RO{I}_{lidar}$ serves as the Query, while $RO{I}_{image}$ acts as the Key and Value. The cross-attention is computed as shown in the formula. Symmetrically, for the image-to-LiDAR path, $RO{I}_{image}$ is the Query, and $RO{I}_{lidar}$ are the Key and Value. This generates $Q_2=W_{Q2}{F}_{image},K_2=W_{K2}{F}_{lidar},V_2=W_{V2}{F}_{lidar}$. The output is computed as ${\tilde{F}}_{image}=\mathrm{MultiHead}(Q_2,K_2,V_2)$. This bidirectional design ensures symmetric information flow. It allows each modality to gather complementary context from the other. This effectively mitigates the initial feature misalignment at the ROI level.

$$\begin{array}{c}\hfill Q_1=W_{Q1}{F}_{lidar}\end{array}$$

(7)

$$\begin{array}{c}\hfill K_1=W_{K1}{F}_{image}\end{array}$$

(8)

$$\begin{array}{c}\hfill V_1=W_{V1}{F}_{image}\end{array}$$

(9)

$${\tilde{F}}_{lidar}=\mathrm{MultiHead}(Q_1,K_1,V_1)$$

(10)

$${\mathrm{head}}_i=softmax\left({\displaystyle \frac{Q{\left(K\right)}^{\top }}{\sqrt{d_k}}}\right)V$$

(11)

Gated fusion. For each modality, the original ROI feature is concatenated with its cross-attention-enhanced counterpart. This yields the enriched representations: $G_A=[F_A,{\tilde{F}}_A]\in R^{B\times N\times 2C}$ and $G_B=[F_B,{\tilde{F}}_B]\in R^{B\times N\times 2C}$. The feature selection gate in this module considers inter-channel relationships. More importantly, it focuses on effectively fusing the cross-modal and original modal features. The feature selection weights are computed using a two-layer MLP and a Sigmoid activation function:

$$\begin{array}{c}\hfill W_A=\sigma (W_2·\mathrm{ReLU}(W_1·G_A+b_1)+b_2)\end{array}$$

(12)

Here, $W_1\in R^{2C\times (C/4)}$ and $W_2\in R^{(C/4)\times 2C}$ are computed similarly. The selection weights are then applied to the concatenated features:

$$H_A=G_A\odot W_A$$

(13)

$$H_B=G_B\odot W_B$$

(14)

Here, ⊙ denotes channel-wise multiplication. The gated fusion emphasizes identifying and retaining the most critical feature elements for the detection task within the cross-modal context. This achieves selective information enhancement.

Multi-modal feature integration. Finally, the enhanced features from both modalities are concatenated. This forms the final aligned feature representation $H_{\mathrm{final}}=[H_A,H_B]\in R^{N\times B\times 4C}$. It provides a more discriminative feature representation for subsequent classification and bounding box regression.

3.4. Class-Agnostic 3D Boxes Fusion

This paper employs a weighted bounding box fusion algorithm to replace the traditional Non-Maximum Suppression (NMS) method. This optimizes the initial detection results, as shown in Figure 4. NMS suppresses highly overlapping neighboring boxes by setting an IoU threshold. In dense object scenes, it can cause false suppression due to small box offsets. Furthermore, keeping only a single box discards valuable localization information. The Class-Agnostic 3D Box Fusion algorithm aims to utilize information from duplicate boxes generated by multiple modalities. It seeks to improve localization accuracy while maintaining a high recall rate.

Figure 4. Comparison of the processing pipelines of C3BF and NMS. (a) The NMS method: it only keeps the detection box with the highest confidence (yellow) and suppresses (discards) the other overlapping boxes (green and blue). (b) The proposed C3BF method: it first clusters spatially overlapping boxes into a group. Then, it performs weighted fusion of these boxes based on their confidence scores. Finally, more accurate fused bounding boxes (purple) are generated.

Given an initial set of detection boxes $\mathcal{B}={\left\{b_i\right\}}_{i=1}^N$ and their corresponding confidence scores $\mathcal{S}={\left\{s_i\right\}}_{i=1}^N$, where each box is parameterized as $b_i={[x_i,y_i,z_i,l_i,w_i,h_i,{\theta }_i]}^{\top }$, representing its center coordinates, dimensions, and orientation angle. First, the detection boxes are sorted in descending order of confidence:

$$\mathrm{argsort}\left(\mathcal{S}\right)=[i_1,i_2,\dots ,i_N],s_{i_1}\ge s_{i_2}\ge \dots \ge s_{i_N}$$

(15)

For each box $b_{i_k}$, compute its 3D IoU with existing cluster centers $c_j$:

$${\mathrm{IOU}}_{3D}(b_{i_k},c_j)={\displaystyle \frac{V(b_{i_k}\cap c_j)}{V(b_{i_k}\cup c_j)}}$$

(16)

If ${max}_j{IoU}_{3D}(b_{i_k},c_j)>{\tau }_{\mathrm{iou}}$, assign it to the cluster with the maximum IoU. Otherwise, create a new cluster. Here, ${\tau }_{\mathrm{iou}}$ is a preset threshold.

For each cluster ${\mathcal{C}}_k=\{b_1^{\left(k\right)},\dots ,b_{n_k}^{\left(k\right)}\}$ and its scores ${\mathcal{S}}_k=\{s_1^{\left(k\right)},\dots ,s_{n_k}^{\left(k\right)}\}$, the fused boxes’ parameters are calculated as:

$$b_k^{\mathrm{fused}}={\displaystyle \frac{{\sum }_{j=1}^{n_k}{s}_j^{\left(k\right)}{b}_j^{\left(k\right)}}{{\sum }_{j=1}^{n_k}{s}_j^{\left(k\right)}}}$$

(17)

The fused score is computed as the average confidence within the cluster.

$$s_k^{\mathrm{fused}}={\displaystyle \frac{1}{n_k}}\sum _{j=1}^{n_k}{s}_j^{\left(k\right)}$$

(18)

4. Experimental Procedures

4.1. Experimental Details

To validate the method’s applicability in autonomous unmanned scenarios, we employ the KITTI dataset [41], which provides realistic urban driving environments analogous to low-altitude drone navigation challenges. Our method is implemented with PyTorch 1.8.1. The model is trained on two 3090 GPUs using the Adam optimizer for 50 epochs. The initial learning rate is 0.01, and we use a one-cycle learning rate scheduler. The model uses virtual point clouds generated by PENet [42]. The model architecture is based on Voxel-RCNN. It contains four VirConv layers with feature dimensions of 16, 32, 64, and 64. The training loss function follows [20]. We use both local and global data augmentation strategies. These include GT-sampling, local transformations (rotation, translation), and global transformations (rotation, flipping). Detection performance is measured by the KITTI 3D Average Precision (AP) metric on the Car class. Objects are categorized into three difficulty levels: Easy, Moderate, and Hard. This is based on their occlusion and truncation levels. Evaluation is performed over 40 recall thresholds. The IoU threshold for the Car class is set to 0.7.

4.2. Multi-Modal 3D Detection on the KITTI Dataset

This experiment evaluates the performance of our proposed method for 3D object detection. We use the detection accuracy (reported as a percentage) under three difficulty levels: Easy, Moderate, and Hard. Competing methods are divided into two categories based on their sensor data usage: LiDAR-only and LiDAR + RGB fusion. As shown in Table 2, we compare our method against various state-of-the-art algorithms. Results show that our method achieves the best overall performance across all three difficulty levels. It achieves scores of 92.22% (Easy), 85.03% (Moderate), and 82.25% (Hard). Furthermore, the overall trend indicates that methods fusing LiDAR with RGB information (LiDAR + RGB) generally outperform LiDAR-only approaches. This highlights the effectiveness of multi-modal fusion for improving detection accuracy.

Table 2. Object detection results on the KITTI dataset.

Method	Modality	Easy (%)	Mod. (%)	Hard (%)
Btc [43]	LiDAR	90.64	82.86	78.09
Casa [44]	LiDAR	91.58	83.06	80.08
SparseDet [45]	LiDAR	90.79	81.17	78.11
GFA-SMT [46]	LiDAR	89.12	80.24	74.52
Futr3d [47]	LiDAR + RGB	90.55	82.28	77.59
Vpfnet [48]	LiDAR + RGB	91.02	83.21	78.20
Sparse [32]	LiDAR + RGB	91.73	84.76	77.92
VirConv [33]	LiDAR + RGB	90.10	84.72	80.22
VoPiFNet [49]	LiDAR + RGB	88.51	80.97	76.74
Ours	LiDAR + RGB	92.22	85.03	82.25

Visualization: Figure 5 illustrates the multi-modal data fusion pipeline of our method. The first column shows the original color images from the RGB camera sensor. The second column displays the 3D point cloud data from the LiDAR scan. The third column presents the result of projecting the LiDAR points onto the image plane. This step achieves pixel-level alignment of geometric and texture information, forming the basis for effective feature fusion. The last column shows the “virtual points” (in red) generated by the depth completion algorithm. These virtual points effectively compensate for the sparsity of the original LiDAR point cloud and enrich the scene’s geometric details.

Figure 5. Input modality visualization on the KITTI dataset.

4.3. Multi-Modal 3D Detection on the KITTI Dataset Under Noisy and Occluded Conditions

To validate the model’s performance under noisy and occluded conditions, we constructed a synthetic noisy dataset. This noisy dataset was generated from the KITTI dataset. We converted ground truth labels to the LiDAR coordinate system and identified the points within the objects. To simulate occlusion in harsh environments, 30% of the foreground points (points within the objects) were perturbed by Gaussian noise with a standard deviation of $0.2\mathrm{m}$. Figure 6 visualizes the noisy point cloud distribution in the Bird’s Eye View (BEV). The added noise within the objects manifests as localized disorder and blurring of the geometric boundaries.

Figure 6. BEV visualization on the KITTI dataset under noisy and occluded conditions. The green boxes are ground truth bounding boxes, and the arrows indicate the object orientations.

Table 3 explicitly demonstrates the detection performance of different modalities under these simulated noisy and occluded scenes. Multimodal fusion significantly outperforms single-modality methods across all difficulty levels. This confirms the robustness of the pseudo-point-based multimodal detection architecture. Our proposed method surpasses the baseline multimodal method at all difficulty levels. Specifically, our method achieves $81.92\%$ AP on the challenging Hard difficulty. This is a $1.37\%$ improvement compared to the baseline multimodal method. This demonstrates that the MWF and CMA modules effectively filter noise. They address feature misalignment caused by noise and depth estimation errors. Furthermore, performance shows a slight drop when noise is introduced compared to performance under ideal conditions (normal). We observe the smallest performance drop of 0.33% on the Hard samples. Conversely, the drop is slightly larger for Easy samples, at 0.64%. Easy samples typically feature dense original point clouds. Adding Gaussian noise causes a non-negligible perturbation to this inherently precise geometric structure. Since Easy samples rely more on precise geometric information in the absence of noise, their performance shows a relatively larger decline when noise is introduced.

Table 3. Object detection results on the KITTI dataset under noisy and occluded conditions.

Method	Easy (%)	Mod. (%)	Hard (%)
Image	85.63	73.86	68.52
Point cloud	89.70	80.33	79.34
Multimodal	90.49	83.27	80.55
Ours (noise)	91.58	84.61	81.92
Ours (normal)	92.22	85.03	82.25

Figure 7 intuitively demonstrates the robustness of our method under simulated noisy and occluded conditions. While the point cloud data was perturbed by Gaussian noise (yellow), the baseline model resulted in missed detections. In contrast, our method successfully detected and accurately localized these occluded targets.

Figure 7. Visualization of object detection results on the KITTI dataset under noisy and occluded conditions. The gray points represent the raw point cloud, the yellow points denote noise, the green bounding boxes are the predictions of the baseline model, the red bounding boxes are the predictions of our model, and the blue bounding boxes are the ground truth.

4.4. Ablation Experiments on the KITTI Dataset

We conduct ablation studies on the KITTI dataset to evaluate different modules. As shown in Table 4, each module effectively improves the baseline model (A). The MWF module (B) provides the most significant gain, demonstrating the effectiveness of its dynamic fusion weights. The CMA (C) and CA3D (D) modules show consistent improvements on the more challenging Moderate and Hard settings. They address feature alignment and bounding box refinement, respectively. The combination of all three modules (F) achieves the best performance, particularly on the Easy (92.22%) and Hard (82.25%) difficulty levels.

Table 4. Ablation detection results on the KITTI validation set by using different designed components. ✔ means “with” and - is “without”.

Model	MWF	CMA	CA3D	Easy (%)	Mod. (%)	Hard (%)
A	-	-	-	90.10	84.72	80.22
B	✔	-	-	91.96	85.45	82.51
C	-	✔	-	90.13	85.19	82.09
D	-	-	✔	90.10	85.06	81.96
E	-	✔	✔	90.56	85.20	82.34
F	✔	✔	✔	92.22	85.03	82.25

Additionally, we evaluated the parameters and efficiency of different modules using 500 test samples. As shown in Table 5, the MWF module (Model B), as a lightweight meta-network, only slightly increases the parameter count (+1.55 MB). However, due to its dynamic inference process, the runtime increases significantly (+10.12 s), and the FPS decreases accordingly from 7.04 to 6.16. The CMA module (Model C) uses an attention mechanism for feature alignment. It increases the parameter count more noticeably (+5.01 MB), but the runtime overhead is well controlled (+0.20 s). The FPS (7.02) remains almost the same as the baseline (7.04). Notably, the CA3D module (Model D) serves as a post-processing step. It does not add parameters. Its efficient box fusion algorithm slightly reduces runtime and increases FPS to 7.08, showing its advantage. The full model (E) integrates all three modules. Compared to the baseline (A), it has a controlled parameter increase (+6.56 MB). The inference time increases only slightly by 1.12 s, and the FPS remains at 6.93. Figure 8 visually illustrates the relationship between parameter count and inference time. It shows that the full model achieves a significant overall performance improvement at a very small computational cost.

Table 5. Ablation experiment of the proposed method computational efficiency. ✔ means “with” and - is “without”.

Model	MWF	CMA	CA3D	Param. (MB)	TE (s)	FPS
A	-	-	-	42.75	71.02	7.04
B	✔	-	-	44.30	81.14	6.16
C	-	✔	-	47.76	71.22	7.02
D	-	-	✔	42.75	70.61	7.08
E	✔	✔	✔	49.31	72.14	6.93

Figure 8. Visualization of parameters and runtime. The notations A–E correspond to the models in Table 5.

The modality ablation results presented in Table 6 unequivocally confirm the superiority of the multimodal fusion strategy. When employing the Image modality alone, performance is the lowest across all metrics (Easy: 85.63%, Moderate: 73.86%, Hard: 68.52%) due to the inherent lack of accurate depth information and perspective distortion. Conversely, the Point Cloud (LiDAR) modality, which provides accurate yet sparse 3D geometric structure, yields significantly higher detection accuracy (Easy: 88.93%, Moderate: 82.19%, Hard: 79.29%). Crucially, only the combination of both modalities—Multimodal fusion (LiDAR + RGB)—achieves the optimal overall performance (Easy: 92.22%, Moderate: 85.03%, Hard: 82.25%). This strongly validates the necessity of deep fusion, which leverages the precise geometric structure from LiDAR and the rich semantic texture from images to complement each other.

Table 6. Ablation results on the KITTI validation set by using different modality.

Modality	Easy (%)	Mod. (%)	Hard (%)
Image	85.63	73.86	68.52
Point cloud	88.93	82.19	79.29
Multimodal	92.22	85.03	82.25

As shown in Table 7, we explored the impact of different modality fusion methods. The feature concatenation strategy significantly outperforms simple addition (+) across all metrics. This proves the importance of preserving feature completeness for improving performance. Our proposed CMA fusion strategy achieves the best performance. It shows an improvement of 1.87% over the basic concatenation method, especially on the most challenging Hard level. This validates that the CMA module, through cross-modal attention for feature alignment and adaptive gated fusion, can more effectively utilize complementary information. It significantly enhances robustness in difficult scenarios. A key finding is that a naive cross-attention strategy performs worse than even concatenation. This indicates the necessity of the designs within our CMA module, such as the adaptive gating mechanism, rather than simply introducing an attention mechanism.

Table 7. Ablation results on the KITTI validation set by using different fusion strategy on the detection head.

Fusion Strategy	Easy (%)	Mod. (%)	Hard (%)
+	89.43	80.33	77.51
concat	90.10	84.72	80.22
cross attention	89.72	82.29	77.78
CMA	90.13	85.19	82.09

We investigated the impact of different parameter settings on module performance. As shown in Figure 9 (left), we analyzed the performance trend concerning the number of cross-attention heads. The CMA module exhibited differing optimal head counts based on difficulty level (2 for Easy, 8 for Moderate/Hard). We attribute this variation to a trade-off between feature diversity, subspace decomposition capability, and model generalization. According to Table 8, the model achieved the best performance for Moderate and Hard difficulty levels when the number of attention heads was set to 8. This is because high-difficulty scenes (e.g., high occlusion, low truncation) present sparse features and complex backgrounds, leading to more severe initial misalignment of multimodal features and necessitating the identification of finer complementary information. Employing 8 attention heads allows the cross-modal feature to be decomposed into more independent subspaces. This enables the model to learn finer-grained, multi-view alignment relationships (e.g., simultaneously attending to geometric shape and textural boundaries). This multi-granularity feature interaction significantly enhances the model’s ability to capture and utilize complementary information, thus achieving more robust feature alignment and improving performance in complex scenarios. However, performance peaked at 2 heads for the Easy difficulty level. In Easy scenes, the required cross-modal alignment relationship is relatively simple and general. An excessive number of heads (e.g., 8) may cause the model to learn overly specialized or redundant alignment patterns, slightly sacrificing generalization in simple scenes. Two heads provide sufficient capacity to capture the core alignment information for Easy scenes while maintaining good generalization. Considering overall performance across all difficulty levels, setting the number of heads to 8 is the optimal choice.

Figure 9. Comparing the effects of different parameters on CMA and MWF.

Table 8. Ablation study: number of heads in cross attention of CMA.

Heads	Easy (%)	Mod. (%)	Hard (%)
2	91.73	82.98	79.96
4	89.76	85.12	81.96
8	90.13	85.19	82.09

As shown in Figure 9 (right), we analyzed the effect of the meta-network’s hidden layer dimension on performance. According to Table 9, a hidden dimension of 64 was optimal, yielding the best overall performance across all three difficulty levels. The meta-network aims to dynamically generate optimal fusion weights w based on the joint 3D and 2D global context vector s. This process is a nonlinear function approximation, mapping high-dimensional scene features to a low-dimensional, normalized weight space. Setting the hidden dimension to 64 provides moderate model capacity. This capacity is sufficient to learn the complex nonlinear relationship between scene context and optimal modal contributions. This allows the model to accurately decide when to rely more on pseudo-point cloud information and when to trust point cloud geometry. A low dimension of 32 led to a notable performance drop. When the dimension was reduced to 32, the model capacity became insufficient, causing a significant performance drop, especially on Moderate and Hard samples where scene complexity is high and modal advantages change dynamically. The meta-network could not effectively capture subtle variations in different scene contexts, resulting in inaccurate or non-adaptive fusion weights (i.e., underfitting), which significantly decreased performance. A high dimension of 128 provided no significant gain. Increasing the dimension to 128 resulted in performance similar to 64, but with increased computational overhead. This indicates that the 64-dimension capacity reached the saturation point for effective information utilization in this task. The extra capacity (128 dimensions) primarily learned noise or redundant patterns in the scene features, offering no noticeable performance improvement.

Table 9. Ablation study: number of hidden layers in the Meta network of MWF.

Hidden	Easy (%)	Mod. (%)	Hard (%)
128	90.49	85.31	82.45
64	91.96	85.45	82.51
32	89.51	82.18	79.34

4.5. Comparative Analysis of ROI Feature Activation

To intuitively demonstrate the effectiveness of the CMA module, we visualize the detection results and the corresponding Region of Interest (ROI) feature intensities in Figure 10. The green and red boxes represent the ground truth and model predictions, respectively. The colored dots indicate the centers of ROIs, with numerical labels explicitly displaying the mean activation intensity of the features at each location. As highlighted by the black rectangles in Figure 10, the baseline model (a) exhibits relatively suppressed feature responses, with an activation value of 0.52 positioned at the lower end of the activation intensity spectrum. This weak feature representation leads to missed detections. In contrast, our optimized model (b) generates significantly stronger feature activations in the same regions, with an intensity value of 0.15, which lies at the upper end of the activation intensity range. This enhancement indicates that our method effectively extracts more discriminative semantic features from the point cloud, enabling the model to accurately localize and classify objects that were overlooked by the baseline.

Figure 10. Heatmap visualization of ROI feature responses. The points illustrate the center of Region of Interests (ROIs), and their colors (from purple to yellow) correspond to the feature activation intensity. Higher values indicate stronger model confidence in the local features. Subfigure (a) denotes the predictions of the baseline model, and (b) denotes those enhanced with the CMA module. Here, green boxes are ground truth, red boxes are model predictions, and black boxes are used to highlight our model’s superior performance.

4.6. Qualitative Analysis of the MWF Module

To intuitively demonstrate the adaptive behavior of our proposed Meta-Weighted Fusion (MWF) module, we visualize the dynamically generated fusion weights for different scenes. Figure 11 presents a qualitative analysis, showing the input camera image (top row), the front-view LiDAR point cloud with ground-truth objects marked in red (bottom left), and the corresponding fusion weights generated by the MWF module (bottom right). In scenario (a), the target vehicle is close. The LiDAR point cloud for this object is relatively dense and accurate. Consequently, the MWF module assigns a higher weight to the LiDAR-dominant pathway. In contrast, scenario (b) features a distant target. The corresponding LiDAR point cloud is sparse due to sensor limitations. In this challenging case, the MWF module intelligently increases the weights for the image-dominant and balanced-average strategies. This shift leverages the rich semantic texture from the image to compensate for the lack of geometric data. These observations confirm that our MWF module dynamically regulates the contributions of each modality. It enhances LiDAR features when the point cloud is reliable and strengthens image features when the geometry is sparse. This adaptive fusion strategy is crucial for robust performance across varied driving conditions.

Figure 11. Visualization of MWF module weights. (a,b) represent near and distant scenes, thereby showing the change in weights. The top row displays the camera image. The bottom row shows the front-view projection of the LiDAR point cloud and the corresponding fusion weights of the MWF module. Red points in the point cloud view indicate ground truth objects.

4.7. Object Detection Visualization

We further present qualitative 3D object detection results of our method on the KITTI dataset, as shown in Figure 12. For comparison, we also include results from VirConv, a representative method. From left to right, the figure shows the ground truth, the results from VirConv, and the results from our proposed method. Detected objects are indicated by green bounding boxes. Our method produces more accurate detection results compared to VirConv. It performs notably better at distinguishing foreground objects from complex backgrounds. For instance, in urban scenes, VirConv tends to generate false positives or miss detections, such as failing to detect vehicles with sparse features. This is because VirConv is more susceptible to background clutter and sensor noise. In contrast, our method effectively mitigates these issues through enhanced feature fusion and alignment. It demonstrates higher detection accuracy and robustness in challenging scenarios.

Figure 12. Visualization of object detection results on the KITTI dataset. The green boxes represent the model’s 3D predicted boxes.

5. Discussion

Compared to most existing multimodal detection methods, our proposed approach demonstrates significant performance advantages. However, despite the notable gains, our method still presents certain limitations. First, the network relies on a depth completion module (e.g., PENet) to generate pseudo-LiDAR point clouds. The resulting depth estimation errors require structural mitigation within the model. Second, the MWF and CMA modules introduce marginal increases in inference time and parameter count, respectively. The full model’s total parameter count is $49.31\mathrm{MB}$. Our analysis suggests that the model’s total size is suitable for deployment on various autonomous systems and mobile robotic platforms. For instance, the Jetson Xavier NX platform, with its 8GB memory, is capable of running medium-complexity models like ResNet-50 (with parameters $<100\mathrm{MB}$). Furthermore, the NVIDIA DRIVE Orin chip provides a computing power of 254 TOPS. In comparison, the RTX 3090 used in our experiments provides 285 TOPS at INT8 precision. This comparison indicates that our model’s resource requirement is well below the capacity limit of dedicated automotive hardware. In future work, we will further explore model efficiency optimization techniques such as knowledge distillation and quantization, prioritizing validation on power-constrained platforms.

6. Conclusions

This paper presents a pseudo-point-based adaptive fusion network for 3D object detection, an end-to-end framework for monocular camera and LiDAR fusion. We introduced three novel modules to achieve biased fusion and optimization. The MWF module uses a lightweight meta-network to dynamically balance the contributions of LiDAR and pseudo-point features. The CMA module employs bidirectional cross-attention and adaptive gating to resolve feature misalignment at the ROI level, yielding more discriminative representations. Ultimately, the C3BF strategy consolidates redundant predictions through a confidence-weighted mechanism, substantially refining both the localization precision and the structural stability of the resultant 3D bounding boxes. Extensive ablation studies and performance evaluations were conducted on the KITTI dataset. Our method achieved leading SOTA performance, reaching 82.25% AP on the challenging Hard difficulty level. Notably, the MWF module provided the most significant performance gain, specifically achieving a 2.29% improvement on the Hard level. The efficient C3BF module slightly increased inference speed without adding model parameters. For future work, we will focus on efficiency optimization techniques like knowledge distillation and quantization, emphasizing real-world validation on power-constrained platforms.