MultiDistiller: Efficient Multimodal 3D Detection via Knowledge Distillation for Drones and Autonomous Vehicles

Abstract

Real-time 3D object detection is a cornerstone for the safe operation of drones and autonomous vehicles (AVs)—drones must avoid millimeter-scale power lines in cluttered airspace, while AVs require instantaneous recognition of pedestrians and vehicles in dynamic urban environments. Although significant progress has been made in detection methods based on point clouds, cameras, and multimodal fusion, the computational complexity of existing high-precision models struggles to meet the real-time requirements of vehicular edge devices. Additionally, during the model lightweighting process, issues such as multimodal feature coupling failure and the imbalance between classification and localization performance often arise. To address these challenges, this paper proposes a knowledge distillation framework for multimodal 3D object detection, incorporating attention guidance, rank-aware learning, and interactive feature supervision to achieve efficient model compression and performance optimization. Specifically: To enhance the student model’s ability to focus on key channel and spatial features, we introduce attention-guided feature distillation, leveraging a bird’s-eye view foreground mask and a dual-attention mechanism. To mitigate the degradation of classification performance when transitioning from two-stage to single-stage detectors, we propose ranking-aware category distillation by modeling anchor-level distribution. To address the insufficient cross-modal feature extraction capability, we enhance the student network’s image features using the teacher network’s point cloud spatial priors, thereby constructing a LiDAR-image cross-modal feature alignment mechanism. Experimental results demonstrate the effectiveness of the proposed approach in multimodal 3D object detection. On the KITTI dataset, our method improves network performance by 4.89% even after reducing the number of channels by half.

1. Introduction

With smart transportation systems evolving towards an integrated air–ground–space framework, 3D detection plays a critical role in both drones and autonomous vehicles. In applications of drones, 3D object detection enables accurate identification of both aerial and ground obstacles, supporting autonomous navigation, obstacle avoidance, and target tracking, thereby enhancing flight safety and mission efficiency. In the field of autonomous vehicles, this technology aids in precise localization, real-time obstacle detection, and comprehensive environmental perception in complex traffic conditions, providing crucial data support for autonomous driving decisions and fundamentally improving road safety and operational efficiency. Autonomous drones and vehicles system consists of three main components: perception, decision-making, and control. As the first step, perception plays a critical role, as its accuracy significantly impacts subsequent navigation decisions. In the field of autonomous driving, precise 3D object detection enables vehicles to identify pedestrians, other vehicles, and obstacles while predicting their movement trajectories, leading to safer driving decisions. Therefore, the accuracy and speed of 3D object detection are directly related to the safety and reliability of autonomous navigation.

Despite current research still being primarily centered on ground-based single-platform perception, this cross-platform collaborative vision is driving the development of common technologies such as multimodal fusion and lightweight modeling [1]. Furthermore, we believe that with the evolution of air–ground integrated perception systems, an increasing number of drone-based datasets will be collected, and our methods can be effectively utilized in these applications as well. While recent studies have made progress in point cloud, camera-based, and multimodal detection, practical applications still face significant challenges [2]. The real-time fusion of heterogeneous sensor data, such as LiDAR and cameras, substantially increases computational complexity [3]. Additionally, existing lightweight methods struggle with multimodal feature coupling representation and the balance between classification and localization, leading to severe performance degradation during compression.

Knowledge distillation, an effective model compression technique [4], has been well-established in 2D vision by transferring “soft knowledge” from teacher models to enhance student models’ generalization capabilities. However, knowledge distillation in 3D scenes faces unique bottlenecks. The processing of point clouds and images through multi-view transformations such as BEV/RV and voxelization introduces complex spatial contextual relationships [5,6]. The parameter scale of 3D convolutional networks grows exponentially, while network heterogeneity exacerbates knowledge alignment difficulties. Additionally, the inherent complexity of 3D spatial data results in massive data volumes, further hindering direct migration of 2D distillation methods to 3D domains.

Existing approaches attempt to address these challenges. PointDistiller [7] improves local geometric structure learning of point clouds through dynamic graph convolution, while SparseKD [8] refines logit distillation to guide student models. However, these methods fail to enable models to adaptively learn high-quality features. Moreover, severe foreground-background imbalance in object detection is often overlooked, along with the varying importance of features across different channels and spatial positions. To mitigate this, we incorporate a bird’s-eye view (BEV) foreground mask, enabling the network to focus more on foreground learning and reduce background noise interference. Furthermore, by leveraging spatial and channel attention mechanisms, we emphasize more informative channels and spatial features, enhancing the network’s perception capabilities. While CaKDP [9] introduces student-driven class-aware knowledge distillation to mitigate classification discrepancies between heterogeneous detectors, it neglects relative confidence ranking among anchors, leading to classification errors and false positives. To address this, we propose ranking-aware category distillation, enabling the student model to learn more precise modeling relationships. Moreover, to extend our approach to multimodal 3D object detection, we introduce multimodal interactive feature supervision based on an intermediate fusion model to fully capture the correlations and potential of each modality.

The proposed cross-modal fusion knowledge distillation framework effectively reduces computational complexity while maintaining good model performance. Specifically, its key contributions are as follows: (1) We propose attention-guided feature distillation, incorporating a foreground-background mask to mitigate noise interference and leveraging attention mechanisms to enhance perception of critical channels and spatial locations. (2) We introduce rank-aware category distillation, enabling the student model to establish more accurate modeling relationships. (3) We develop multimodal interactive feature supervision, leveraging an intermediate fusion model to enhance point cloud and image information extraction.

2. Related Works

We first introduce the widely adopted methods in 3D object detection. Currently, existing algorithms can be broadly categorized into three groups based on the type of sensors used: The detection methods can be primarily divided into three categories based on the type of sensor used: (1) LiDAR-based object detection: Point-based methods, such as PointNet++ [10], extract features by using set abstraction; Voxel-based methods [11,12,13] divide point clouds into voxels or pillars [14], which are then convolved to obtain features; Point-voxel methods [15] combine both feature extraction techniques. (2) Image-based object detection: Methods like DETR3D [16] query 3D features for detection, while LSS [17] utilizes view transformations. (3) LiDAR and image-based object detection: Methods like PointPainting [18] enhance LiDAR point clouds with images for early fusion, some works [5,6,19] fuse features in the BEV (bird’s eye view) perspective, and CLOCS [20] performs late fusion to combine proposals from different modalities.

In order to significantly reduce complexity and computational requirements while maintaining network performance, many lightweight methods based on knowledge distillation have been proposed. Hinton et al. [21] first introduced knowledge distillation, the core idea of which is to enhance the performance of small models (student networks) by transferring knowledge from large models (teacher networks). In the field of 2D vision, a relatively mature methodology has been developed: feature alignment-based FitNets [22] significantly improve image classification performance by distilling intermediate-layer features; FAKD [23] introduces feature association mechanisms to enhance the effectiveness of knowledge transfer; and IntRA-KD [24] improves segmentation accuracy by transmitting scene structure information through encoding. For 2D object detection tasks, the latest methods mainly focus on improving distillation loss and soft label strategies, designing feature distillation mechanisms, and optimizing teacher–student network architectures.

In the realm of 3D object detection, research on knowledge distillation has followed two major technical routes. On one hand, methods such as LIGA-Stereo [25] enhance geometric perception through cross-modal knowledge transfer, while CRKD [26] and SRKD [27] improve detection robustness via multi-sensor knowledge fusion and weather condition simulation, respectively. On the other hand, approaches like SparseKD [8] and PointDistiller [7] focus on model lightweighting by balancing efficiency and accuracy through channel/depth compression and local geometry distillation. It is noteworthy that CaKDP [9] proposed a student-driven class-aware distillation strategy to establish a knowledge transfer bridge among heterogeneous detectors. However, existing methods still exhibit several limitations. First, most studies focus on pure point cloud modalities, and the lightweight design of multimodal fusion networks remains insufficiently explored. Second, current feature distillation methods do not effectively address the severe foreground–background imbalance in 3D detection and lack adaptive awareness of the importance of channel and spatial features. Moreover, existing class distillation strategies fail to model the confidence ranking among anchors, thereby limiting improvements in classification performance.

3. Method

3.1. Overall Framework Description

The proposed cross-modal knowledge distillation framework is illustrated in Figure 1. We design an attention-guided feature distillation module, where the foreground and background are separated based on a bird’s-eye view (BEV) foreground mask. Then, a combination of spatial and channel attention mechanisms is employed to guide the student model in focusing on more critical feature regions. Secondly, we introduce a ranking-aware response distillation strategy, which aligns the classification confidence distributions of single-stage and two-stage detectors by modeling anchor-level distributions and applying Kullback–Leibler (KL) divergence constraints. Finally, we propose an interactive feature supervision mechanism that establishes a feature distillation path from LiDAR to images during the intermediate fusion stage, leveraging point cloud data to enhance the spatial awareness of image features.

3.2. Attention-Guided Feature Distillation

Current channel compression-based distillation methods suffer from dual limitations [18,28]. On one hand, reducing feature dimensions leads to the loss of multi-level spatial semantic information, making it difficult to preserve critical geometric details. On the other hand, traditional methods treat feature map channels and spatial positions indiscriminately, neglecting the extreme foreground-background imbalance in 3D scenes, which exacerbates noise interference. To address these issues, as illustrated in Figure 2, we propose a dual-attention guided hierarchical distillation framework. First, a bird’s-eye view (BEV) mask is used to separate the foreground and background. Then, a learnable attention weight mechanism is introduced in the channel dimension to dynamically calibrate and enhance the transfer of knowledge in key channels, mitigating information degradation caused by feature compression. In the spatial dimension, spatial attention is leveraged to focus on the depth feature transmission of potential target regions.

As shown in Equation (1), the bird’s-eye view mask can be represented as $M_{l,w}$, where $r$ represents the real labels, and $\left(l,w\right)$ represents the position in the BEV. If the corresponding position contains an object, the mask is 1; otherwise, it is 0.

$$M_{l,w}=\left\{\begin{array}{c}1, if\left(l,w\right)\in r\hfill \\ 0, Otherwise\hfill \end{array}\right.$$

(1)

Additionally, to balance the size differences between object categories and avoid detection performance degradation due to small targets (e.g., pedestrians), we also use a mask $G_{l,w}$ based on object scale, as shown in Equation (2). Where $S_r$ represents the area of the ground truth bounding box in the bird’s-eye view (BEV) perspective and $N_{bg}$ is the remaining background area, as shown in Equation (3).

$$G_{l,w}=\left\{\begin{array}{c}{\displaystyle \frac{1}{S_r}}, if\left(l,w\right)\in r\\ {\displaystyle \frac{1}{N_{bg}}}, Otherwise\end{array}\right.$$

(2)

$$N_{bg}=\sum _{l=1}^L\sum _{w=1}^W(1-M_{l,w})$$

(3)

The scale-aware mask $G_{l,w}$ serves to normalize the contribution of each object based on its size. Larger objects inherently cover more pixels and would otherwise dominate the loss. By incorporating object area and normalized background compensation, $G_{l,w}$ ensures that both large and small objects are treated fairly during knowledge transfer. Many studies [29,30,31,32] have shown that using attention mechanisms to focus on key pixels and key channels helps CNN-based models achieve better results. To minimize the computational cost of the student network, we directly use simple calculations [33] to obtain spatial and channel attention masks. First, we calculate the absolute mean values for different pixels and different positions with the following Formulas (4) and (5):

$$S^S\left(F\right)={\displaystyle \frac{1}{C}}\cdot \sum _{c=1}^C \left|F_c\right|$$

(4)

$$S^C\left(F\right)={\displaystyle \frac{1}{LW}}\cdot \sum _{l=1}^L \sum _{w=1}^W \left|F_{i,j}\right|$$

(5)

where $L$, $W$, $C$ represent the length, width, and the number of channels of the feature. $S^S$ and $S^C$ are the spatial and channel attention maps. The attention masks are then expressed as Equations (6) and (7):

$$A^S\left(F\right)=H\cdot W\cdot softmax\left(S^S\left(F\right)/T\right)$$

(6)

$$A^C\left(F\right)=C\cdot softmax\left(S^C\left(F\right)/T\right)$$

(7)

where $T$ is a hyperparameter used to adjust the distribution [21]. During the training process, we use the teacher’s attention mask to guide the student. The spatial attention map $A^S\left(F\right)$ tends to assign higher weights to regions with strong activation responses, often corresponding to object centers or boundaries in 3D scenes. Similarly, the channel attention $A^C\left(F\right)$ emphasizes feature channels that are more discriminative across different object categories. This selective enhancement helps the student focus on semantically meaningful information during the distillation process. The final feature distillation loss is computed as follows:

$$\begin{gathered} L_{att\_feat}=\alpha \sum _{c=1}^C \sum _{l=1}^L \sum _{w=1}^W M_{l,w}{G}_{l,w}{A}_{l,w}^S{A}_c^C{\left(F_{c,l,w}^t-f\left(F_{c,l,w}^s\right)\right)}^2 \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+\beta \sum _{c=1}^C \sum _{l=1}^H \sum _{j=1}^W \left(1-M_{l,w}\right)G_{l,w}{A}_{l,w}^S{A}_c^C{\left(F_{c,l,w}^t-f\left(F_{c,l,w}^s\right)\right)}^2 \end{gathered}$$

(8)

where $\alpha$ and $\beta$ are hyperparameters to balance foreground and background, and $F^t$ and $F^s$ represent the teacher and student feature maps, respectively.

To verify the practicality of our dual attention guided distillation in real world settings, we designed all attention computations to rely solely on simple, highly parallelizable tensor operations (summation, normalization and softmax) without introducing any extra convolutional layers. The learnable parameters consist only of per channel scaling weights, which represent a vanishingly small fraction of the overall model size.

During training, the additional cost arises only from (1) computing two attention maps per feature tensor—one along the spatial dimensions and one along the channel dimension—and (2) applying the distillation loss between teacher and student features. Both steps reuse intermediate activations and leverage optimized library routines, so the wall clock time per epoch increases only marginally on modern GPUs. Crucially, there is zero overhead at inference, since all attention masks and loss computations are disabled once training completes.

In terms of memory footprint, only two extra tensors (one of shape L × W and one of length C) need to be stored transiently during forward/backward passes, which is negligible compared to the backbone feature maps. Overall, our framework imposes only a light training time penalty while delivering consistent gains in student accuracy, and it does not affect runtime latency or deployment size.

3.3. Ranking-Aware Class Distillation

Analysis [9] reveals significant differences in prediction distributions between heterogeneous detectors (two-stage vs. single-stage). Two-stage detectors utilize RoI selection to suppress background noise and achieve higher category confidence, while single-stage detectors, with their dense anchors, tend to induce inter-class competition. To address this issue, CAKDP [9] proposes a student-driven, category-aware knowledge distillation strategy. However, directly aligning classification scores overlooks the relative confidence ranking among anchors [34]. Particularly for challenging samples, the student network’s limited feature representation may lead to misaligned anchor selection, making it difficult to maintain spatial consistency with the teacher’s predictions. As illustrated in Figure 3, we have designed a ranking-aware category distillation framework that supervises the differences in category distributions via KL divergence and aligns the confidence ranking relationships using a ranking loss. This approach achieves a deep alignment of prediction distributions between heterogeneous detectors, effectively enhancing the modeling accuracy for challenging targets.

Specifically, we first use the student network to select the Top-N high-response anchors for each instance as representative samples. The spatial positions of the student anchors are then mapped onto the teacher network’s first-stage feature map. RoI pooling is used to extract the corresponding second-stage classification scores. The $j$-th instance of each frame corresponds to the representative sample anchor $a_i^j$, where $i\in \left\{1,\cdots ,N\right\}$, and $N$ represents the number of selected anchors. The corresponding student and teacher target class classification scores are denoted as ${p_s^{cls}}_i^j$ and ${p_t^{cls}}_i^j$. We apply the softmax function to these anchor scores to obtain the ranking distributions ${p_s^{rank}}_i^j$ and ${p_t^{rank}}_i^j$, as shown in the equations. The temperature parameter $\tau$ is introduced to soften the distribution, enhancing the ranking information transfer for difficult samples. The KL divergence between the teacher and student ranking distributions is minimized:

$${p_s^{rank}}_i^j={\displaystyle \frac{exp({p_s^{cls}}_i^j/\tau )}{\sum _{m=1}^N exp({p_s^{cls}}_m^j/\tau )}}$$

(9)

$${p_t^{rank}}_i^j={\displaystyle \frac{exp({p_t^{cls}}_i^j/\tau )}{\sum _{m=1}^N exp({p_t^{cls}}_m^j/\tau )}}$$

(10)

$$L_{rank}=\sum _{j=1}^M \sum _{i=1}^N {p_t^{rank}}_i^{j}log\left({\displaystyle \frac{{p_t^{rank}}_i^j}{{p_s^{rank}}_i^j}}\right)$$

(11)

We directly constrain the class probability distributions of the teacher and student for the same anchor points, and compute the per-class KL divergence $L_{cate}$:

$$L_{cate}=\sum _{j=1}^M \sum _{i=1}^N {p_t^{cls}}_i^{j}log\left({\displaystyle \frac{{p_t^{cls}}_i^j}{{p_s^{cls}}_i^j}}\right)$$

(12)

$$L_{cls}={\gamma L}_{rank}+\delta L_{cate}$$

(13)

where $\gamma$ and $\delta$ are weight coefficients used for policy adjustment. The ranking-aware class distillation method aligns anchor-level ranking distributions, enhancing the student’s ability to learn from the key anchors in the teacher’s decision logic. This ranking distribution supervision also alleviates the false-negative problem caused by anchor position shifts. The method achieves multi-granularity supervision: the class distribution constraint ensures global confidence accuracy, while the ranking distribution constraint enhances local ranking consistency, thus promoting the performance improvement of heterogeneous detectors.

The overall loss function for the method can be expressed as:

$$L_{loss}=L_{gt}+L_{cls}+L_{feat}$$

(14)

where $L_{gt}$ represents the loss function that calculates the difference between the network’s output and the ground truth during the student network’s training phase.

To ensure that our ranking-aware class distillation is both effective and practical, we carefully designed each component to incur minimal overhead. During training, our module reuses the student’s forward-pass scores and applies a highly optimized top-k selection to identify the N most responsive anchors, incurring only marginal latency. These anchor coordinates are then mapped to the teacher’s first-stage feature map and processed with standard RoI pooling—no new network layers or learnable parameters are introduced. Next, two softmax distributions (Equations (9) and (10)) and their corresponding KL-divergence losses (Equations (11) and (12)) are computed in a fully vectorized fashion, scaling linearly with N and leveraging backend-accelerated tensor routines. Temporarily storing only two small N-length tensors per instance imposes a negligible memory footprint compared to backbone activations. Crucially, all extra operations—anchor selection, RoI pooling, softmax and loss calculation—are active only during training, and are entirely bypassed at inference, resulting in zero impact on deployment latency or model size. Overall, our ranking-aware class distillation adds only a slight, practically imperceptible training-time cost while consistently boosting student accuracy.

3.4. Interactive Feature Supervision Distillation

Currently, there are three main approaches for LiDAR-Camera fusion in object detection: early fusion, intermediate fusion, and late fusion. The early fusion approach typically processes images first, then continues to use a 3D backbone network and detection head to perform object detection on the point clouds. Therefore, the previously proposed point-cloud-based 3D object detection knowledge distillation method can be directly applied. The late fusion approach is similar, allowing the previously introduced loss function for knowledge distillation to be directly used.

In LiDAR–camera 3D detection, three principal fusion paradigms exist: early fusion, intermediate fusion, and late fusion. Early-and late-fusion distillation methods supervise only within a single modality—either by applying point-cloud distillation losses directly to fused representations or by treating the entire multimodal feature map as a homogeneous block. Such approaches overlook the rich, asymmetric relationships between LiDAR and image features and fail to adapt when teacher and student channels do not match. By contrast, our interactive feature supervision mechanism Interactive Feature Supervision Distillation explicitly aligns and transfers knowledge both within and across modalities, before and after fusion.

The intermediate fusion approach is mainly illustrated in Figure 4. In this method, images and point clouds are processed separately by their respective backbone networks to extract features. The student camera features are denoted as $F_{Camera}^s$, while the student LiDAR features are denoted as $F_{Lidar}^s$. The teacher LiDAR features are denoted as $F_{Lidar}^t$, and the teacher camera features are denoted as $F_{Camera}^t$. The fused features of the student and teacher networks can be represented as ${F_{CL}^s}^{\prime }$ and ${F_{CL}^t}^{\prime }$, respectively. Similar to the early fusion method, the difference between ${F_{CL}^s}^{\prime }$ and ${F_{CL}^t}^{\prime }$ can be minimized using $L_{feat}$. However, in the case of intermediate fusion, the feature differences before fusion remain crucial for network performance. Therefore, we propose a multimodal interaction feature supervision mechanism to fully extract the correlations between the multimodal features and the potential of each individual modality.

Similar to the survey [28], we use an automatic channel encoder-decoder, consisting of an encoder that gradually reduces the channel dimension and a decoder that increases the channel dimension. This structure allows the teacher and student networks, which may have mismatched channel dimensions, to be adjusted to the same dimension, after which the loss calculation is performed. Similar to the attention-guided feature distillation method, we can also obtain the spatial attention mask and channel attention mask of the features. To capture global features, the feature distillation loss functions for LiDAR-to-LiDAR $L_{l-l}$, Camera-to-Camera $L_{c-c}$, and LiDAR-to-Camera $L_{l-c}$ can be represented by the following formulas:

$${L_{l-l}=L}_F\left(A^{Spatial}\left(F_L^t\right),A^{Spatial}\left(F_L^s\right)\right)+L_F\left(A^{Channel}\left(F_L^t\right),A^{Channel}\left(F_L^s\right)\right)$$

(15)

$${L_{c-c}=L}_F\left(A^{Spatial}\left(F_C^t\right),A^{Spatial}\left(F_C^s\right)\right)+L_F\left(A^{Channel}\left(F_C^t\right),A^{Channel}\left(F_C^s\right)\right)$$

(16)

$${L_{l-c}=L}_F\left(A^{Spatial}\left(F_C^t\right),A^{Spatial}\left(F_C^s\right)\right)+L_F\left(A^{Channel}\left(F_C^t\right),A^{Channel}\left(F_C^s\right)\right)$$

(17)

$$L_{intra\_feat}={{\sigma }_{1}L}_{l-l}+{{\sigma }_{2}L}_{c-c}+{{\sigma }_{3}L}_{l-c}$$

(18)

${\sigma }_1, {\sigma }_2, {\sigma }_3$ are used to balance the relationship between distillation across different modalities. By incorporating $L_{intra\_feat}$ into Equation (14), the loss function for the multimodal object detection knowledge distillation method can be computed.

4. Experiment Result

4.1. Experiment Setting

The KITTI [35] dataset is the most widely used dataset for 3D object detection in outdoor autonomous driving scenarios, comprising 7481 training samples and 7518 test samples. We adopt the same data split as SECOND [12], utilizing 3712 frames for training and 3769 frames for testing, and evaluate the mean Average Precision (mAP) across three difficulty levels (easy, moderate, and hard) using 40 recall points as per the official evaluation protocol. Additionally, we validate our approach on the nuScenes [36] dataset—a larger-scale outdoor driving dataset where each frame includes data from six surrounding cameras and a LiDAR sensor. The dataset contains over 1.4 million annotated 3D detection boxes spanning 10 classes, and the nuScenes detection score (NDS) is used as the evaluation metric.

We validate the proposed method using various distillation pairs across different datasets. For the KITTI dataset, we use several two-stage detectors, including Voxel-RCNN [11], PV-RCNN [15], and PartA2 [37], as teachers to assist in training single-stage detectors (SECOND [12] and CenterPoint [13] with different detection heads). The multimodal data input used directly references methods based on virtual point inputs (SFD [19], Virconv [38]). Specifically, we combine the virtual points generated by depth completion from images and the real LiDAR point clouds as the input data, validating the overall performance of the fusion method. Additionally, we perform network performance validation for intermediate fusion using the existing fusion method SFD [19].

In the experimental setup, for the single-stage, front fusion method with the SECOND detector, the hyperparameters are set as {$\alpha =1\times {10}^{-3},\beta =5\times {10}^{-4},\gamma =1\times {10}^{-3},\delta =5\times {10}^{-3}$}. For CenterPoint, the hyperparameters are set as {$\alpha =1.6\times {10}^{-3},\beta =8\times {10}^{-4},\gamma =3\times {10}^{-3},\delta =5\times {10}^{-3}$}. For the intermediate fusion detectors, the hyperparameters in the interactive feature supervision are set as {${\sigma }_1=3\times {10}^{-4},{\sigma }_2=1\times {10}^{-4},{\sigma }_3=5\times {10}^{-4}$}. Other training and evaluation configurations from OpenPCDet are kept as default. All experiments are deployed on a GeForce RTX 4090 GPU.

4.2. Experiment Result on KITTI

We first validate the effectiveness of our method on the KITTI dataset’s validation set. As shown in

Table 1. Illustration of the interactive feature supervision method.

	Model	Model Size	KD	Car	Pedestrian	Cyclist	mAP	Up/Down1	Up/Down2
Student	SECOND [12]	base	$\times$	82.49	53.98	67.41	67.96	-	-
	CenterPoint [13]	base	$\times$	79.97	52.93	69.16	67.35	-	-
Teacher	Voxel-RCNN [11]	base	$\times$	85.56	59.20	74.60	73.45	-	-
Student	SECOND [12]	base	$\surd$	84.48	61.78	76.29	74.18	$\uparrow$	$\uparrow$
		S	$\surd$	84.16	64.03	74.52	74.24	$\uparrow$	$\uparrow$
		XS	$\surd$	83.67	61.64	73.24	72.85	$\uparrow$	$\downarrow$
		XXS	$\surd$	80.33	54.76	67.41	67.50	$\downarrow$	$\downarrow$
	CenterPoint [13]	base	$\surd$	84.4	62.27	75.41	74.03	$\uparrow$	$\uparrow$
		S	$\surd$	83.72	60.39	74.76	72.96	$\uparrow$	$\downarrow$
		XS	$\surd$	81.71	62.04	73.08	72.28	$\uparrow$	$\downarrow$
		XXS	$\surd$	74.84	60.15	67.85	67.61	$\uparrow$	$\downarrow$
Teacher	PV-RCNN [15]	base	$\times$	85.50	59.21	73.13	72.61	-	-
Student	SECOND [12]	base	$\surd$	84.59	60.28	75.41	73.43	$\uparrow$	$\uparrow$
		S	$\surd$	83.88	60.76	72.9	72.51	$\uparrow$	$\downarrow$
		XS	$\surd$	83.33	61.57	72.29	72.40	$\uparrow$	$\downarrow$
		XXS	$\surd$	80.62	53.77	67.39	67.26	$\downarrow$	$\downarrow$
	CenterPoint [13]	base	$\surd$	84.68	58.66	71.72	71.69	$\uparrow$	$\downarrow$
		S	$\surd$	84.39	59.7	74.16	72.75	$\uparrow$	$\uparrow$
		XS	$\surd$	81.93	60.2	71.7	71.28	$\uparrow$	$\downarrow$
		XXS	$\surd$	75.35	58.92	68.26	67.51	$\uparrow$	$\downarrow$
Teacher	PartA2 [37]	base	$\times$	83.65	61.78	74.08	73.17	-	-
Student	SECOND [12]	base	$\surd$	83.89	61.51	75.12	73.51	$\uparrow$	$\uparrow$
		S	$\surd$	84.39	61.7	74.94	73.68	$\uparrow$	$\uparrow$
		XS	$\surd$	84.28	59.25	73.34	72.29	$\uparrow$	$\downarrow$
		XXS	$\surd$	78.82	54.42	67.36	66.87	$\downarrow$	$\downarrow$
	CenterPoint [13]	base	$\surd$	84.06	59.61	73.24	72.30	$\uparrow$	$\downarrow$
		S	$\surd$	83.85	59.62	74.46	72.64	$\uparrow$	$\downarrow$
		XS	$\surd$	81.18	60.86	73.67	71.90	$\uparrow$	$\downarrow$
		XXS	$\surd$	75.76	59.09	66.40	67.08	$\downarrow$	$\downarrow$

, the detection results for all three categories (Car, Pedestrian, and Cyclist) are evaluated under the moderate difficulty level of the KITTI benchmark. Our fused approach enables the training of accurate and compact student detectors through knowledge distillation from multiple diverse teacher methods.

Specifically, the first column on the right side of the table (up/down1) presents the performance comparison between channel-reduced student networks and their original counterparts. Our method demonstrates that only when the student network’s parameters are reduced toless than 0.2× of the original configuration does it exhibit marginal performance degradation compared to the baseline. In contrast, all other reduced configurations achieve superior performance after knowledge distillation.

The second rightmost column (up/down2) quantifies the performance gap between teacher networks and student networks. Remarkably, our approach even surpasses the two-stage detection teacher networks when the channel reduction in student networks remains moderate (e.g., >0.5× parameter retention). This evidence suggests that our distillation framework successfully preserves critical detection knowledge while achieving substantial model compression.

To further demonstrate the effectiveness of our multi-modal knowledge distillation method, we employ the SFD network as the teacher network and utilize channel-reduced networks with the second stage removed as student networks. Comparative results with other knowledge distillation methods are presented in

Table 2. Illustration of the interactive feature supervision method.

	Model	Size	Car	Pedestrians	Cyclist
Tea	SFD [19]	base	88.27	66.69	72.95
Stu	Reduced SFD	S	86.16	63.47	70.14
	+Vanilla KD [21]	S	86.12	63.72	71.06
	+GID [39]	S	85.79	62.46	69.27
	+SparseKD [8]	S	87.11	65.29	71.15
	+PointDistiller [7]	S	86.84	64.12	70.74
	+CaKDP [9]	S	87.09	65.48	71.57
	+Ours	S	88.34	66.41	72.73

. Our approach substantially outperforms competing methods while maintaining baseline performance levels despite significant computational cost reduction.

Our visualization results are presented in Figure 5. The first row shows the detection results of the SECOND student network obtained through our knowledge distillation approach using VoxelRCNN as the teacher, while the second row displays the original SECOND detection results. It is evident that our network significantly improves false detection reduction and maintains high accuracy.

4.3. Ablation Study

This section investigates the impact of different distillation losses through comprehensive ablation studies. We first individually evaluate $L_{cls}$, $L_{intra\_feat}$, and $L_{att\_feat}$, followed by pairwise combinations of these loss components. Following the configuration in

Table 3. Ablation studies on SFD method.

Method			Car	Pedestrians	Cyclist
${\mathit{L}}_{\mathit{i}\mathit{n}\mathit{t}\mathit{r}\mathit{a}\_\mathit{f}\mathit{e}\mathit{a}\mathit{t}}$	${\mathit{L}}_{\mathit{a}\mathit{t}\mathit{t}\_\mathit{f}\mathit{e}\mathit{a}\mathit{t}}$	${\mathit{L}}_{\mathit{c}\mathit{l}\mathit{s}}$
$\times$	$\times$	$\times$	86.16	63.47	70.14
$\surd$	$\times$	$\times$	87.14	64.51	71.07
$\times$	$\surd$	$\times$	86.87	64.37	70.86
$\times$	$\times$	$\surd$	86.94	65.14	71.24
$\surd$	$\surd$	$\times$	87.49	65.25	70.97
$\times$	$\surd$	$\surd$	87.74	65.75	71.94
$\surd$	$\times$	$\surd$	87.19	65.83	71.48
$\surd$	$\surd$	$\surd$	88.34	66.41	72.73

, we employ the SFD method as the teacher network and a channel-reduced SFD architecture as the student network. The experimental results presented in Table 3 demonstrate that each proposed computational module contributes positively to overall detection performance, with combined usage achieving optimal effectiveness.

In addition to the KITTI experiments, we further conduct validation experiments using the TransFusion method on the nuScenes dataset. The experimental results are shown in

Table 4. Ablation studies on nuScenes method.

	Model	Size	mAP	mATE	mASE	mAOE	mAVE	mAAE	NDS
Tea	Transfusion [6]	base	0.6388	0.2867	0.2555	0.2725	0.2624	0.1895	0.6927
Stu	Reduced Transfusion	S	0.6230	0.3290	0.2569	0.3124	0.2683	0.1921	0.6756
	Reduced Transfusion + Ours	S	0.6387	0.2885	0.2564	0.2870	0.2638	0.1904	0.6907

. The results similarly demonstrate the effectiveness of our approach, showing that the method maintains baseline performance levels while achieving significant model size reduction.

To investigate the characteristics of distillation performance, we conducted experiments using a two-stage to two-stage knowledge distillation framework. The experimental results presented in

Table 5. Ablation studies on homogeneous method.

	Model	Size	Car	Pedestrians	Cyclist
Tea	SFD	base	88.27	66.69	72.95
Stu	SFD	S	88.53	66.72	73.10
Tea	Voxel-RCNN	base	85.56	59.20	74.60
Stu	Voxel-RCNN	S	85.76	60.14	74.87
Tea	PV-RCNN	base	85.50	59.21	73.13
Stu	PV-RCNN	S	85.61	59.33	73.48
Tea	PartA2	base	83.65	61.78	74.08
Stu	PartA2	S	83.72	61.94	74.31

demonstrate that our method effectively preserves model performance while reducing network complexity through knowledge distillation. Notably, our approach achieves simultaneous channel compression and performance enhancement across all comparative experimental groups.

Furthermore, we conducted experiments with varying parameter configurations. Our analysis reveals that the proposed method demonstrates low sensitivity to specific hyperparameter settings, maintaining stable performance across different parameter selections. The channel-reduced student architecture retains critical feature representation capacity through our $L_{att\_feat}$, which enforces structural similarity between teacher and student feature maps. This built-in regularization reduces dependence on precise parameter tuning. The synergistic combination of $L_{cls}$, $L_{att\_feat}$ and $L_{intra\_feat}$ creates complementary learning signals. This loss landscape smoothing effect mitigates the risk of local minima entrapment caused by suboptimal hyperparameters.

5. Conclusions

This paper proposes a cross-modal knowledge distillation framework for enhancing 3D object detection performance through multi-level knowledge transfer strategies. To address critical challenges in cross-modal feature learning, we present three innovative solutions: (1) Attention-Guided Feature Distillation: An attention-driven distillation mechanism incorporating bird’s-eye-view (BEV) foreground masks and dual-attention modules substantially enhances feature learning sufficiency while suppressing background noise interference. (2) Category-Aware Response Distillation: Our anchor-level distribution modeling with KL divergence optimization effectively mitigates classification performance degradation when distilling knowledge from two-stage to single-stage detectors. (3) Interactive Feature Supervision Framework: A novel cross-modal alignment strategy with LiDAR-image feature supervision significantly strengthens the representation capability of intermediate fusion approaches. Extensive experiments on KITTI and other benchmarks demonstrate that our method maintains baseline performance levels while reducing model complexity, even achieving performance improvements over certain existing approaches. These technical advances provide new solutions for real-time 3D perception in autonomous vehicles and UAV applications, while establishing reference standards for future research in multi-sensor fusion systems.

Critical Reflection: Our framework consistently outperforms or matches full-size baselines on KITTI, nuScenes, and other benchmarks—even under substantial compression—demonstrating that carefully designed KD strategies can not only recover but sometimes exceed the accuracy of uncompressed models. While our experiments validate the effectiveness of each module, future work could explore automated hyperparameter tuning for the attention and distillation weights $(\alpha ,\beta ,\gamma ,\delta )$ and assess robustness under varying sensor noise and domain shifts.

Applications and Future Directions: The proposed methods are immediately applicable to real-time 3D perception in autonomous vehicles, UAV navigation, and mobile robotics—domains where tight compute and memory budgets demand efficient yet accurate detectors. Looking forward, we plan to extend our framework to handle (i) dynamic sensor configurations, adapting on-the-fly to sensor failures or occlusions; (ii) continual learning scenarios, where new object classes emerge post-deployment; and (iii) other modality pairs such as radar–camera or thermal–LiDAR fusion. We also envision integrating our distillation strategies with unsupervised and self-supervised pretraining to further reduce annotation requirements.

In summary, this work lays a comprehensive foundation for lightweight, high-performance multimodal detection. By critically evaluating our contributions and outlining clear paths for future enhancement, we aim to inspire continued progress in efficient 3D perception and multi-sensor fusion research.