Edge-Oriented Lightweight YOLO-World with Cross-Modal Fusion Adaptation for Multimodal Object Detection and Deployment

Abstract

Multimodal detection models enable flexible object detection through text prompts, but YOLO-World-style models still incur high computational and storage costs on edge devices. To address this problem, this paper develops an edge-oriented adaptation framework for replacing the original visual encoder of YOLOv8l-worldv2 with a compact visual branch. The framework jointly considers visual-branch compression, multi-scale interface consistency, cross-modal feature compatibility, and edge-side inference, rather than optimizing these aspects independently. Specifically, a YOLOv7-based visual branch is reconstructed using depthwise separable convolutions, enhanced by DyHead, and compressed through sensitivity-guided Filter Pruning via Geometric Median (FPGM) under multi-scale interface constraints. An identity-initialized semantic adaptation layer and multi-template text prototypes are then introduced to alleviate the feature distribution mismatch between the compressed visual branch and the original cross-modal fusion space. Finally, offline text prototype generation and TensorRT-based INT8/FP16 mixed-precision inference are used for edge deployment. Experiments on BDD100K show that the lightweight visual encoder achieves 62.45% mAP@0.5 with 22.5 M parameters and 52.8 GFLOPs. After multimodal integration, the proposed model achieves 64.9% mAP@0.5 with 31.5 M parameters and 67.2 GFLOPs, reducing GFLOPs by 67.1% compared with YOLOv8l-worldv2 while causing only a 1.6 percentage-point accuracy drop. On the Jetson Orin Nano Super, the deployed model reaches 32.6 FPS with a model size of 32 MB, demonstrating its feasibility for edge perception scenarios.

1. Introduction

Autonomous vehicles need to perceive surrounding objects in real time under complex traffic conditions, and visual object detection serves as a fundamental component of environmental understanding and driving decision-making. Road-scene images usually contain multiple categories of objects, such as vehicles, pedestrians, riders, traffic lights, and traffic signs. They are also affected by weather, illumination, occlusion, and object-scale variations, which impose high requirements on the accuracy, real-time performance, and generalization ability of detection models [1]. Owing to their favorable balance between inference speed and detection accuracy, YOLO-series detectors and real-time detection Transformers have been widely applied to real-time object detection tasks [2,3,4,5]. However, conventional detectors rely on a fixed set of predefined categories. When the target categories change, additional data annotation and model training are usually required, making them less suitable for category expansion in open traffic scenarios.

Vision–language pre-training models provide a new perspective for overcoming the limitations of closed-set detection. CLIP (Contrastive Language–Image Pre-training) establishes semantic associations between image features and textual descriptions through image–text contrastive learning, enabling models to transfer across tasks based on natural language prompts [6]. GLIP (Grounded Language–Image Pre-training) unifies object detection and phrase grounding within a single framework [7], while Grounding DINO further improves text-guided open-set object localization [8]. YOLO-World combines open-vocabulary detection with a YOLO-based one-stage detection framework and integrates multi-scale visual features with text embeddings through cross-modal fusion modules, achieving a trade-off between detection accuracy and inference efficiency [9]. This makes it a feasible foundation for flexible object detection in autonomous driving scenarios.

Although YOLO-World provides real-time open-vocabulary detection capability, its deployment on edge devices is still constrained by limited computational resources, storage capacity, and inference latency. In this study, the Ultralytics implementation of YOLOv8l-worldv2 is adopted as the YOLO-World baseline model, which has approximately 46.83 M parameters and 204.5 GFLOPs [10]. Although this model offers strong open-vocabulary detection capability, its high computational cost limits its deployment efficiency on resource-constrained edge devices.

Existing lightweighting studies mainly focus on structural optimization, pruning-based compression, and low-precision deployment. For object detection models deployed on edge devices, a careful trade-off is usually required among detection accuracy, inference latency, model size, and hardware adaptability [11,12]. Depthwise separable convolution (DSC), Dynamic Head (DyHead), and Filter Pruning via Geometric Median (FPGM) have been widely used to reduce computational cost, enhance detection features, and perform structured pruning, respectively [13,14,15,16]. Structured pruning and quantization are also commonly adopted to improve model deployment efficiency [17,18].

However, most existing lightweighting methods are designed for closed-set detectors or general visual networks. When they are directly applied to multimodal detection frameworks such as YOLO-World, two issues remain. First, a single lightweighting strategy is often insufficient to achieve both a high compression ratio and satisfactory accuracy preservation. Second, after the original visual encoder is replaced with a lightweight branch, the resulting visual features may deviate from the CLIP text semantic space and the original cross-modal fusion modules. Existing prompt learning methods mainly improve the adaptation ability of vision–language models from the text side [19,20], but they pay limited attention to cross-modal feature alignment after visual branch replacement.

DSC, DyHead, FPGM pruning, prompt templates, and TensorRT inference are established techniques and are not claimed as individually novel. The contribution of this study lies in coordinating them to address two coupled issues arising from visual-branch replacement in YOLO-World: maintaining the P3/P4/P5 interface consistency after compression and preserving compatibility between the compressed visual features and the original cross-modal fusion space. Specifically, sensitivity-guided pruning is performed under multi-scale interface constraints, DyHead compensates for weakened multi-scale representation, and an identity-initialized semantic adaptation layer together with multi-template text prototypes mitigates the cross-modal feature distribution mismatch. This coordinated design enables substantial visual-branch compression while retaining compatibility with the original text-conditioned detection pipeline.

To address these issues, this paper proposes a lightweight visual encoder construction and cross-modal fusion adaptation method for YOLO-World and further validates its edge deployment performance through TensorRT-based mixed-precision quantization. The main contributions are summarized as follows:

(1)

A coordinated visual-branch replacement methodology is developed for the YOLO-World multimodal detection framework. Instead of independently applying lightweight operators and pruning, the proposed method combines DSC-based backbone reconstruction, DyHead-based multi-scale compensation, and sensitivity-guided FPGM pruning under explicit P3/P4/P5 interface constraints. This design reduces the computational cost of the visual branch while maintaining the structural compatibility required by the subsequent cross-modal fusion modules.

(2)

A cross-modal compatibility mechanism is designed to address the feature distribution shift caused by replacing the original YOLO-World visual encoder with a compressed closed-set branch. An identity-initialized semantic adaptation layer reorganizes the compressed multi-scale features before cross-modal fusion, while multi-template category prototypes stabilize the textual semantic anchors. The ablation results distinguish this adaptation effect from the contribution of visual-branch compression itself.

(3)

An edge-oriented execution scheme is established by coupling offline text prototype generation with module-aware INT8/FP16 mixed-precision inference. The scheme removes the CLIP text encoder from the online edge-side pipeline and retains higher precision for cross-modal similarity-sensitive operations, providing an end-to-end evaluation of accuracy, computational cost, storage footprint, and inference speed on the Jetson Orin Nano Super.

2. Related Work

Open-vocabulary object detection has attracted increasing attention because it enables detection models to recognize objects beyond a fixed set of predefined categories. CLIP establishes semantic associations between images and text through image–text contrastive learning, providing a foundation for natural language prompt-driven visual recognition [6]. However, CLIP is mainly designed for image-level semantic matching and cannot directly perform region-level object localization.

To address this limitation, RegionCLIP extends vision–language pre-training to region-level representation learning [21], while OWL-ViT implements open-vocabulary object detection based on a vision Transformer and image–text pre-training [22]. Furthermore, GLIP unifies object detection and phrase grounding within a single framework [7], and Grounding DINO improves text-guided open-set object localization [8]. These methods have promoted the development of open-vocabulary detection, but their model architectures are usually complex and involve relatively high inference costs. In comparison, YOLO-World integrates visual features and text embeddings through cross-modal fusion modules, achieving a favorable balance between real-time performance and open-category detection capability [9]. Therefore, it is adopted as the multimodal detection foundation in this study. Recently, Mamba-YOLO-World and YOLOE have further advanced real-time open-vocabulary detection from the perspectives of cross-modal fusion and efficient prompting mechanisms [23,24].

Model lightweighting is another important direction for deploying detection models on resource-constrained edge devices. Depthwise separable convolution can reduce the parameter count and computational cost introduced by standard convolutions, and its effectiveness has been verified by Xception and the MobileNet series [13,14]. EfficientDet improves the accuracy–efficiency trade-off of detection models through a bidirectional feature pyramid network and compound scaling strategy [25], while DyHead enhances detection feature representation through dynamic modeling along the scale, spatial, and task dimensions [15]. In recent years, real-time detection methods such as YOLOv9, YOLOv10, and RT-DETR have further advanced the accuracy–efficiency frontier of detection models from the perspectives of gradient information utilization, end-to-end detection, and efficient encoder design [3,4,5].

In addition to structural optimization, structured pruning can directly remove channels or filters, making it easier to translate model compression into practical hardware acceleration. FPGM identifies redundant filters based on the geometric median criterion, avoiding the limitation of relying solely on the L1 or L2 norm to measure filter importance [16]. However, detection networks usually contain multi-branch connections and cross-scale fusion structures, and pruning therefore needs to maintain dimensional and interface consistency between upstream and downstream layers. DepGraph models inter-layer dependencies to enable general structured pruning for complex network architectures, providing a useful reference for channel regularization in multi-branch networks [26].

For cross-modal adaptation and edge deployment, YOLO-World fuses visual features with text embeddings through cross-modal fusion modules, enabling the model to perform open-vocabulary detection according to text prompts [9]. However, when the original visual encoder is replaced with a lightweight visual branch, the distribution of the new visual features may deviate from both the CLIP text semantic space and the original cross-modal fusion modules. Direct replacement may therefore weaken the matching between regional visual features and category-level text prototypes. Prompt learning methods such as CoOp and CoCoOp can improve the task adaptation capability of vision–language models [19,20], but they mainly optimize image–text alignment from the text side and pay limited attention to the visual feature distribution shift caused by lightweight visual encoder replacement. Post-training quantization methods such as Q-YOLO and AdaRound have shown that low-precision inference can reduce model storage and computational overhead [27,28]. However, the focus of this study is not to propose a new quantization algorithm, but to deploy a lightweight multimodal detection model on edge devices through TensorRT-based mixed-precision inference [29].

In summary, existing studies have made progress in open-vocabulary detection, visual branch compression, structured pruning, and edge deployment. Nevertheless, systematic research on cross-modal adaptation after visual encoder replacement in YOLO-World remains limited. Based on YOLOv8l-worldv2, this paper proposes a coordinated method that combines visual branch lightweighting with fusion adaptation, aiming to reduce the online inference cost of multimodal detection models while preserving their text-prompt-based detection capability.

3. Proposed Method

To reduce the computational overhead of YOLO-World-based multimodal detection models on edge devices, this paper develops a method that combines visual branch lightweighting, cross-modal fusion adaptation, and mixed-precision deployment. The overall framework is shown in Figure 1.

3.1. Overall Framework

This study adopts YOLOv8l-worldv2 as the multimodal baseline and replaces only its original visual encoder with a compact branch, while retaining the original cross-modal fusion and text-conditioned detection pipeline. To reduce edge-side overhead, category-level text embeddings are generated offline, and the CLIP text encoder is excluded from online inference. The resulting framework is illustrated in Figure 1.

As shown in Figure 1, the proposed framework consists of three main stages. First, a lightweight visual encoder is constructed based on YOLOv7. Depthwise separable convolutions are used to replace computationally expensive 3 × 3 standard convolutions in the backbone network, and DyHead is introduced at the P3, P4, and P5 multi-scale output levels for dynamic feature enhancement. Subsequently, structured channel pruning is performed by combining layer-wise sensitivity analysis with the FPGM geometric median criterion, resulting in a compact visual encoder.

Second, the compact encoder is integrated into YOLO-World as an alternative visual branch. A lightweight semantic adaptation layer is inserted between the DyHead-enhanced features and the cross-modal fusion modules to alleviate the distribution discrepancy between closed-set visual features and the CLIP text semantic space. Meanwhile, multi-template text prompts are used to construct category-level text semantic prototypes, and the text encoder is kept frozen during multimodal fine-tuning.

Finally, edge deployment optimization is performed based on offline text semantic prototypes and a TensorRT mixed-precision inference engine. During online inference, the edge device mainly executes visual feature extraction, feature adaptation, cross-modal fusion, and text-contrastive detection, while the CLIP text encoder is excluded from the online inference pipeline.

The three stages are designed as a coupled process rather than as an independent sequence of existing techniques. DSC reconstruction and FPGM pruning reduce computational cost but may weaken multi-scale representation and alter the channel-wise distribution of visual features. DyHead is therefore retained to compensate for multi-scale feature degradation, while the pruning process is constrained to preserve the P3/P4/P5 interfaces required by the original fusion modules. The semantic adaptation layer subsequently addresses the remaining distribution mismatch before visual–text interaction, and the frozen multi-template text prototypes provide stable semantic references during multimodal fine-tuning. This dependency among compression, interface preservation, and semantic adaptation constitutes the central design rationale of the proposed framework.

3.2. Construction of the Lightweight Visual Encoder

To reduce visual-branch computation while preserving the P3/P4/P5 interfaces required for cross-modal fusion, a compact encoder is constructed based on YOLOv7 [2,30] and subsequently integrated into the original YOLOv8l-worldv2 multimodal framework. YOLOv7 is selected because its ELAN-W backbone contains explicit multi-branch convolution and feature concatenation paths, which facilitate partial DSC replacement, structured channel pruning, and preservation of the P3/P4/P5 output interfaces. Specifically, the computationally expensive $3\times 3$ standard convolutions in ELAN-W can be replaced without removing its original multi-branch feature reuse structure and $1\times 1$ channel-fusion layers. These characteristics make YOLOv7 suitable for constructing a compact visual branch that can subsequently be connected to DyHead and the cross-modal fusion modules.

Other YOLO backbones could also be used in the proposed framework. However, differences in stage depth, channel dimensions, feature aggregation structures, and multi-scale output interfaces would require redesigning the DSC replacement positions, pruning dependencies, and semantic adaptation layers. Therefore, the resulting accuracy–efficiency trade-off would depend on the selected backbone and cannot be directly transferred without retraining and interface adaptation. The improved DSC-ELAN-W structure adopted in this study is shown in Figure 2.

Let the number of input channels be $C_{in}$, the number of output channels be $C_{out}$, and the convolution kernel size be $K$ × $K$. The parameter count of a standard convolution can be expressed as:

$$P_{conv}=K^2{C}_{in}{C}_{out}$$

(1)

Depthwise separable convolution decomposes a standard convolution into two stages: depthwise convolution and pointwise convolution. Its parameter count can be expressed as:

$$P_{dsc}=K^2{C}_{in}+C_{in}{C}_{out}$$

(2)

As illustrated in Figure 2, each computationally expensive $3\times 3$ standard convolution in the selected ELAN-W branches is replaced by a depthwise convolution followed by a $1\times 1$ pointwise convolution. Equations (1) and (2) quantify the parameter costs of these two alternatives. According to the two equations, when $K=3$ and the channel numbers are relatively large, $P_{\mathrm{d}\mathrm{s}\mathrm{c}}$ is substantially smaller than $P_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{v}}$. This explains the computational reduction provided by the DSC blocks shown in Figure 2 while preserving the feature map size and output-channel interface.

After reconstructing the backbone network with DSC, the basic computational cost of the model is reduced. However, lightweight convolutions and subsequent channel pruning may weaken fine-grained feature representation, thereby affecting the detection performance for small objects and complex scenes. To address this issue, DyHead is introduced at the P3, P4, and P5 multi-scale outputs of the lightweight visual encoder. In this study, DyHead is used as a visual feature enhancement module rather than as a final category prediction head. During closed-set training, the enhanced visual features are connected to the YOLOv7 detection branch for encoder training and pruning evaluation. During multimodal integration, they are fed into the cross-modal fusion modules through the semantic adaptation layer and then fused with text embeddings.

To further compress the model size, sensitivity-guided FPGM structured pruning is introduced on the basis of DSC-based backbone reconstruction and DyHead-based feature enhancement. Considering that different convolutional layers have different sensitivities to channel pruning, layer-wise sensitivity analysis is adopted to evaluate the response of each layer to pruning. For the $l$-th convolutional layer, while keeping the parameters of other layers unchanged, temporary channel pruning is applied only to the current layer, and the change in model accuracy is evaluated on the pruning strategy evaluation subset. Let the original detection accuracy of the unpruned model be A, and let the detection accuracy of the current layer under pruning ratio ($r$) be $A_l\left(r\right)$. The accuracy degradation can be expressed as:

$$\Delta A_l\left(r\right)=A-A_l\left(r\right)$$

(3)

As indicated by Equation (3), a larger $\Delta A_l\left(r\right)$ means that the corresponding layer is more sensitive to channel pruning, whereas a smaller value suggests greater channel redundancy. The values of $A_l\left(r\right)$ obtained at different pruning ratios are used to generate the layer-wise sensitivity curves. Based on these curves and the corresponding accuracy degradation defined in Equation (3), the layers are divided into sensitive, moderately sensitive, and robust groups, which are subsequently assigned different pruning ratios.

After determining the pruning ratio for each layer, the FPGM criterion is adopted to select the filters to be pruned within each layer. FPGM identifies redundant filters from the perspective of the geometric relationships among filters in the same convolutional layer and preferentially removes filters that are highly similar to others and therefore more replaceable. During pruning, the number of remaining channels is further constrained to ensure consistency among multi-branch feature concatenation, the P3/P4/P5 output channels, and the input interfaces of the subsequent semantic adaptation layer and cross-modal fusion modules. After pruning, the model is fine-tuned with a small learning rate, allowing the compressed visual encoder to adapt to the changed feature distribution.

The resulting compact encoder retains the P3/P4/P5 interfaces required for integration into the YOLOv8l-worldv2 multimodal framework.

3.3. Design of Multimodal Fusion Adaptation

After construction, the lightweight encoder replaces the original visual branch of YOLOv8l-worldv2. Because the encoder is trained and pruned under closed-set supervision, its output distribution may differ from that of the original visual branch and the CLIP text semantic space, potentially weakening visual–text matching. As shown in Figure 3, the P3, P4, and P5 features are first enhanced by DyHead, transformed by the semantic adaptation layer, and then passed to the original cross-modal fusion and text-contrastive detection pipeline. DyHead is retained solely as a multi-scale feature enhancement module rather than an additional detection head.

To alleviate the distribution discrepancy between lightweight visual features and the text semantic space, a lightweight semantic adaptation layer is inserted between the DyHead output features and the cross-modal fusion modules. This adaptation layer consists of a single 1 × 1 convolution. The input and output channel numbers are kept the same, and the spatial resolution of the feature map remains unchanged. Therefore, the layer only performs linear reorganization of visual features along the channel dimension. For the $i$-th multi-scale feature $F_i$, the adaptation process is formulated as:

$${\tilde{F}}_i={Conv}_{1\times 1}(F_i;W_i,b_i)=W_i\ast F_i+b_i$$

(4)

where $F_i$ denotes the $i$-th visual feature output by DyHead, ${\tilde{F}}_i$ denotes the adapted visual feature, and ${Conv}_{1\times 1}(F_i;W_i,b_i)$ denotes a $1\times 1$ convolution parameterized by the weight $W_i$ and bias $b_i$. The symbol $\ast$ denotes the convolution operation, and the bias $b_i$ is added channel-wise to the convolution output.

Equation (4) implements the semantic adaptation block in Figure 3, mapping each DyHead-enhanced feature to the original Re-parameterizable Vision–Language Path Aggregation Network (RepVL-PAN) interface through a learnable channel transformation without changing its spatial resolution.

To avoid introducing an abrupt perturbation to the pretrained visual features, the $1\times 1$ adaptation convolution is initialized as an identity mapping. When the input and output channel numbers are equal, its weight tensor can be regarded as a channel transformation matrix. The diagonal channel coefficients are initialized to 1, the off-diagonal coefficients are initialized to 0, and the bias $b_i$ is initialized to 0. Therefore, at the exact initialization point, Equation (4) gives ${\tilde{F}}_i=F_i$, rather than merely an approximate equality. After multimodal fine-tuning begins, $W_i$ and $b_i$ are updated by gradient descent; consequently, ${\tilde{F}}_i\approx F_i$ only during the initial training stage, after which the adaptation layer gradually learns a channel-wise transformation suitable for cross-modal fusion.

On the text side, multi-template text prompts are constructed according to the detection categories of BDD100K to improve the stability of category-level semantic prototypes. For each category, several prompt forms are designed, including direct category descriptions, image-scene descriptions, and road-scene-specific descriptions, such as “a photo of a car”, “a photo of a car on the road”, and “a car in a driving scene”. During training, different templates are randomly sampled to generate text inputs, thereby enhancing the robustness of the model to variations in prompt expressions. During validation and testing, the text embeddings generated from multiple templates for the same category are averaged and used as the semantic prototype of that category. The CLIP text encoder is kept frozen during multimodal fine-tuning to maintain the stability of the text semantic space and reduce training overhead.

3.4. Mixed-Precision Deployment on Edge Devices

To reduce the inference latency and storage footprint of the fusion-adapted multimodal detection model on edge devices, this study adopts post-training quantization (PTQ) to build a TensorRT mixed-precision inference engine [29]. Unlike the structural lightweighting strategy described in Section 3.2, mixed-precision deployment optimizes the model from the perspective of numerical precision during inference. Therefore, structural lightweighting and mixed-precision inference reduce model overhead from different but complementary aspects. The overall deployment process is shown in Figure 4.

As shown in Figure 4, the deployment process consists of offline preparation and online edge-side inference. During offline preparation, the frozen CLIP text encoder is first used to generate text semantic prototypes for each detection category. Meanwhile, 1000 images are randomly selected from the BDD100K training set to construct the quantization calibration set, and the validation set is not used for calibration. The fusion-adapted multimodal detection model is then exported to ONNX (Open Neural Network Exchange) format. During TensorRT engine construction, the calibration set is used to collect activation distributions, determine the quantization scale factors for INT8 execution, and generate the serialized inference engine.

For the quantization strategy, the lightweight visual encoder, DyHead feature enhancement module, and semantic adaptation layer are mainly executed in INT8 precision to reduce the computational cost of convolutional and linear operations. For the cross-modal fusion modules and the text-contrastive detection head, which directly interact with the text semantic prototypes, INT8/FP16 mixed-precision execution is adopted according to TensorRT operator support and calibration results. This strategy aims to reduce quantization errors in visual–text similarity computation. The CLIP text encoder is used only to generate category-level semantic prototypes offline and does not participate in online inference. The generated text embeddings are stored in FP16 precision and used as fixed constant inputs in the inference pipeline.

During online edge-side inference, the model receives only the input image and the pre-generated category-level text semantic prototypes. The complete text encoding process is no longer executed on the edge device, which reduces both memory usage and computational overhead. This deployment scheme is suitable for autonomous driving detection scenarios in which the category set is fixed or updated infrequently. If the text prompts need to be changed frequently in practical applications, the category-level text semantic prototypes should be regenerated offline or updated through a separate maintenance process. The effects of quantization on detection accuracy, model size, and inference speed are further analyzed in the experimental section.

4. Experiments and Analysis

Experiments are conducted to evaluate the overall accuracy–efficiency trade-off, lightweight visual encoder performance, component contributions, text sensitivity and image robustness, cross-domain and unseen-category generalization, and edge deployment performance.

4.1. Dataset and Experimental Settings

The BDD100K autonomous driving dataset is used as the main benchmark for experimental validation. This dataset contains driving-scene images collected from urban roads, highways, and various weather and illumination conditions. Its detection categories cover typical road objects, such as vehicles, pedestrians, riders, traffic lights, and traffic signs. Since the annotations of the official BDD100K test set are generally not publicly available, the official training set is used for model training, and the official validation set is used to report the results of ablation experiments, model comparisons, multimodal fusion adaptation, and edge-side deployment accuracy. To prevent pruning ratio selection and quantization calibration from affecting the final evaluation, both the pruning strategy evaluation subset and the quantization calibration set are split from the training set, while the official validation set is used only for final performance reporting.

To ensure fair model comparison, all compared models are trained and evaluated using the same BDD100K training/validation split, input size, and evaluation script. The YOLOv8l-worldv2 baseline model is also fine-tuned on the BDD100K training set, and its results are reported on the official validation set. Other closed-set detectors and the proposed lightweight visual encoder are compared under the same evaluation protocol. Except for differences in model architecture, the main training settings are kept consistent.

The YOLOv8l-worldv2 implementation in Ultralytics 8.2.0 (Ultralytics Inc., Frederick, MD, USA) is adopted as the original multimodal detection baseline in this study. To ensure consistency in complexity statistics, the parameter count and GFLOPs of all models are calculated under the same input size of 640 × 640 and a batch size of 1. Specifically, the complexity of YOLOv8l-worldv2 is obtained from the statistics of the corresponding model configuration, while the complexity of the proposed model is calculated according to the actual network composition of the lightweight visual encoder and the fusion adaptation structure. It should be noted that the complexity statistics in this study include only the detection network computations performed during online edge-side inference, and do not include the offline CLIP text encoding process. For both the YOLOv8l-worldv2 baseline and the proposed model, the text encoder is used to generate category-level text semantic prototypes in advance. During online inference, only visual feature extraction, cross-modal fusion, and text prototype matching are retained.

The lightweight encoder is first trained and pruned using $640\times 640$ inputs and is then integrated into YOLO-World for multimodal fine-tuning. During this stage, the CLIP text encoder is frozen, while the visual branch, DyHead, semantic adaptation layer, and cross-modal detection-related parameters are updated. The resulting model is subsequently deployed on the Jetson Orin Nano Super using TensorRT INT8/FP16 mixed-precision inference.

Model training and accuracy evaluation are conducted on an NVIDIA GeForce RTX 4090 GPU (NVIDIA Corporation, Santa Clara, CA, USA). The software environment consists of PyTorch 2.0.1 (PyTorch Foundation, San Francisco, CA, USA) and CUDA 11.8 (NVIDIA Corporation, Santa Clara, CA, USA). Edge deployment experiments are performed on an NVIDIA Jetson Orin Nano Super Developer Kit (NVIDIA Corporation, Santa Clara, CA, USA) using TensorRT 10.3 (NVIDIA Corporation, Santa Clara, CA, USA). The device is configured in the 25 W maximum performance mode, the inference batch size is set to 1, and model warm-up is performed before formal timing. In this study, mAP@0.5 is used to evaluate detection accuracy; parameter count, GFLOPs, and model size are used to evaluate model complexity; and single-frame inference latency and FPS are used to evaluate edge-side inference efficiency.

4.2. Overall Performance Comparison

To provide a more comprehensive comparison with representative open-vocabulary detectors, the proposed model is evaluated against YOLOv8l-worldv2, YOLOE-v8-M, and MM-Grounding-DINO-Tiny [31]. YOLOE-v8-M represents a recent efficient real-time open-vocabulary detector, whereas MM-Grounding-DINO-Tiny provides a non-YOLO Transformer-based baseline. All models are evaluated on the same BDD100K training and validation split, with an input resolution of 640 × 640 and the unified evaluation protocol described in Section 4.1. For consistency, the total computational cost is measured as the GFLOPs of the complete online detection pipeline for a $640\times 640$ input with a batch size of 1. For the proposed method, this cost includes the lightweight visual encoder, DyHead, semantic adaptation layer, cross-modal fusion modules, and text-contrastive detection head, while excluding only the offline CLIP text encoder. The same statistical scope is used for all compared open-vocabulary detectors. The results are presented in

Table 1. Comparison of accuracy and total online computational cost among representative open-vocabulary detectors.

Model	mAP@0.5 (%)	Parameters (M)	GFLOPs
YOLOv8l-worldv2	66.5	46.83	204.5
YOLOE-v8-M	62.0	25.9	78.9
MM-Grounding-DINO-Tiny	67.9	63.1	168.3
Ours	64.9	31.5	67.2

.

As shown in Table 1, the proposed method requires a total online computational cost of 67.2 GFLOPs, compared with 204.5 GFLOPs for YOLOv8l-worldv2, 78.9 GFLOPs for YOLOE-v8-M, and 168.3 GFLOPs for MM-Grounding-DINO-Tiny. This corresponds to computational cost reductions of 67.1%, 14.8%, and 60.1%, respectively. Therefore, the proposed method has the lowest total online computational cost among the compared open-vocabulary detectors while maintaining an mAP@0.5 of 64.9%.

To further analyze the relationship between detection accuracy and computational cost, mAP/GFLOPs is used as an auxiliary metric for comparison, as shown in Figure 5. The mAP/GFLOPs ratios of YOLOv8l-worldv2, YOLOE-v8-M, MM-Grounding-DINO-Tiny, and the proposed model are 0.325, 0.786, 0.403, and 0.966, respectively. The proposed model achieves the highest ratio, representing a 197% improvement over YOLOv8l-worldv2 and indicating higher detection efficiency per unit of computation.

4.3. Performance Analysis of the Lightweight Visual Encoder

Furthermore, to evaluate the basic detection capability of the lightweight visual encoder before its integration into YOLO-World, it is compared with mainstream closed-set detectors, including YOLOv7, YOLOv8m [32], YOLOv9c [3], and YOLOv11m [33]. The results are shown in

Table 2. Performance comparison with mainstream closed-set detectors.

Model	mAP@0.5 (%)	Parameters (M)	GFLOPs
YOLOv7	62.12	37.3	105.3
YOLOv8m	61.80	25.9	78.9
YOLOv9c	62.05	25.3	102.4
YOLOv11m	61.95	20.1	68.0
Ours	62.45	22.5	52.8

. It should be noted that this experiment is mainly intended to verify the accuracy and computational efficiency of the proposed visual branch and does not represent the final performance of the complete multimodal detection model.

As shown in Table 2, the proposed lightweight visual encoder achieves 62.45% mAP@0.5 with 22.5 M parameters and 52.8 GFLOPs. Compared with YOLOv7, its parameter count and GFLOPs are reduced by 39.7% and 49.9%, respectively. Among the compared models, it achieves the lowest computational cost while maintaining comparable detection accuracy, making it suitable as the visual branch of the subsequent multimodal framework. The accuracy–efficiency relationship among different detection models is shown in Figure 6.

Figure 6 further illustrates that the proposed encoder occupies a favorable position on the accuracy–computation plane, achieving the lowest GFLOPs among models with comparable mAP@0.5.

To provide a more intuitive comparison of different closed-set detectors, typical road-scene samples are selected for visualization analysis, as shown in Figure 7.

All compared models can detect major road objects, including vehicles, pedestrians, traffic signs, and traffic lights. The proposed encoder shows relatively stable responses to several distant small objects, densely distributed targets, and objects under complex illumination. These qualitative results indicate that DyHead partly compensates for the multi-scale representation loss introduced by DSC reconstruction and structured pruning.

4.4. Ablation Experiments

Ablation experiments are conducted to evaluate the pruning strategy, the lightweight visual encoder components, and the multimodal fusion adaptation modules.

4.4.1. Pruning

First, to determine appropriate pruning strengths for different convolutional layers, a pruning strategy evaluation subset is split from the BDD100K training set. Following the method described in Section 3.2, layer-wise sensitivity analysis is conducted on each convolutional layer to be pruned. The accuracy responses of different convolutional layers under different pruning ratios are shown in Figure 8.

As shown in Figure 8, the detection accuracy of each layer generally decreases as the pruning ratio increases. However, the degree of accuracy degradation varies across different layers, indicating that different convolutional layers have different sensitivities to channel pruning. When the pruning ratio is relatively low, the accuracy changes of most convolutional layers are small, and the differences among layers are not sufficiently distinguishable. When the pruning ratio increases to 50% or higher, the accuracy of some sensitive layers decreases markedly. In contrast, a pruning ratio of 40% can more clearly distinguish the sensitivity differences among different convolutional layers without causing severe accuracy degradation in most layers. Therefore, 40% is selected as the reference pruning ratio for layer sensitivity classification in this study.

Based on this setting, the convolutional layers to be pruned are divided into sensitive layers, moderately sensitive layers, and robust layers. A layer is classified as sensitive when the accuracy degradation exceeds 5 percentage points; as moderately sensitive when the degradation is between 2 and 5 percentage points; and as robust when the degradation is less than 2 percentage points. In the subsequent differentiated pruning strategy, sensitive layers are assigned pruning ratios lower than 40% to reduce the loss of critical features, while robust layers are assigned pruning ratios higher than 40% to sufficiently compress redundant channels. Moderately sensitive layers are assigned pruning ratios close to 40% to balance accuracy preservation and compression ratio.

To validate the effectiveness of this strategy, three pruning schemes are compared: uniform pruning, conservative pruning, and differentiated pruning. The results are shown in

Table 3. Comparison of different pruning strategies.

Strategy	Sen.	Mod.	Rob.	mAP@0.5 (%)	Parameters (M)	GFLOPs
None	—	—	—	63.10	31.4	93.6
Uniform	40%	40%	40%	61.55	19.2	54.5
SG-Con.	12%	25%	38%	62.82	22.8	64.3
SG-Ours	18%	32%	48%	62.45	22.5	52.8

Note: None denotes the unpruned DSC + DyHead encoder before FPGM pruning. Sen., Mod., and Rob. denote sensitive, moderately sensitive, and robust layers, respectively. SG-Con. denotes conservative sensitivity-guided pruning, and SG-Ours denotes the sensitivity-guided pruning strategy adopted in this study.

.

As shown in Table 3, the uniform pruning strategy does not consider the sensitivity differences among layers, resulting in a relatively large decrease in accuracy. The conservative pruning strategy can better preserve detection accuracy, but its compression ratio is limited. In contrast, the differentiated pruning strategy adopted in this study achieves more substantial reductions in parameter count and computational cost while maintaining relatively high detection accuracy. The fine-tuning recovery process after pruning is shown in Figure 9.

4.4.2. Ablation of Visual Encoder Modules

Furthermore, to verify the effects of different modules in the lightweight visual encoder, the original YOLOv7 is used as the baseline, and ablation experiments are conducted by progressively adding each module. The results are shown in

Table 4. Ablation results of the lightweight visual encoder.

Config.	DSC	DyHead	FPGM Pruning	mAP@0.5 (%)	Parameters (M)	GFLOPs
A (baseline)				62.12	37.3	105.3
B1	√			62.03	29.6	89.6
B2		√		62.71	39.1	109.3
B3	√	√		63.10	31.4	93.6
D (encoder)	√	√	√	62.45	22.5	52.8

.

As shown in Table 4, DSC reduces the parameter count and GFLOPs, whereas DyHead improves detection accuracy through multi-scale feature enhancement. Applying both modules increases mAP@0.5 from 62.12% to 63.10% while reducing GFLOPs from 105.3 to 93.6. Subsequent FPGM pruning further reduces the model to 22.5 M parameters and 52.8 GFLOPs, with an mAP@0.5 of 62.45%. These results confirm the complementary roles of computational reduction, feature compensation, and structured compression.

4.4.3. Ablation of Fusion Adaptation Modules

Finally, to validate the fusion adaptation effect after integrating the lightweight visual encoder into YOLO-World, ablation experiments are conducted on the semantic adaptation layer, multi-template text prompts, and the DyHead retention strategy. The results are shown in

Table 5. Ablation results of multimodal fusion adaptation.

Config.	Adaptation Layer	Multi-Template Prompts	DyHead	mAP@0.5 (%)	Parameters (M)	GFLOPs
E1				61.9	28.5	62.1
E2	√			63.1	29.8	63.0
E3	√	√		63.8	29.8	63.0
E4 (ours)	√	√	√	64.9	31.5	67.2

.

In Table 5, DyHead denotes retention of the multi-scale feature enhancement module during multimodal integration rather than the addition of an independent detection head. Unlike Table 2, which evaluates the closed-set visual encoder, Table 5 evaluates the complete cross-modal detection model.

As shown in Table 5, direct integration of the lightweight visual encoder yields 61.9% mAP@0.5. Adding the semantic adaptation layer increases the accuracy to 63.1%, and introducing multi-template text prototypes further improves it to 63.8%. Retaining DyHead during multimodal integration produces the best result of 64.9%. These progressive gains demonstrate that channel adaptation, text-prototype stabilization, and multi-scale feature enhancement each contribute to cross-modal compatibility after visual-branch replacement.

4.5. Text Sensitivity, Image Robustness, and Cross-Domain Validation

To evaluate text-semantic sensitivity, four representative BDD100K categories—car, person, bus, and traffic light—are selected. For each category, 200 validation images containing at least one corresponding instance are randomly sampled without replacement, yielding 800 image–category pairs.

Each pair is evaluated using three prompts: the category name, a category-consistent natural-language prompt, and the irrelevant prompt “a photo of a cat”. All inference settings are kept unchanged. The highest confidence score for the input prompt is recorded, and the response gap is defined as the difference between the mean confidence scores of the natural-language and irrelevant prompts. In total, 2400 image–prompt evaluations are performed, and the results are reported in

Table 6. Class-wise results of the expanded text sensitivity test.

Category	Category Name Prompt	Natural-Language Prompt	Irrelevant Prompt	Response Gap
Car	0.84	0.86	0.14	0.72
Person	0.76	0.78	0.12	0.66
Bus	0.81	0.83	0.11	0.72
Traffic light	0.69	0.72	0.09	0.63
Average	0.78	0.80	0.12	0.68

.

As shown in Table 6, category-consistent prompts produce substantially higher confidence scores than the irrelevant prompt across all four categories. The average scores for the category name, natural-language, and irrelevant prompts are 0.78, 0.80, and 0.12, respectively, yielding an average response gap of 0.68. The consistent trend across categories confirms that the model remains responsive to prompt semantics rather than behaving as a fixed-category detector.

To further evaluate robustness to variations in image orientation and illumination, the same 800 image–category pairs used in the text sensitivity test are additionally transformed while keeping the corresponding category-consistent natural-language prompts unchanged. In-plane rotations of $\pm 15°$ and $\pm 30°$ are applied to simulate orientation variations, while brightness factors of 0.6, 0.8, 1.2, and 1.4 are used to simulate different illumination conditions. For each transformed image–prompt pair, the highest confidence score is recorded and averaged over all four categories. No retraining or fine-tuning is performed, and all other inference settings remain unchanged. The results are presented in

Table 7. Robustness of the proposed model under image rotation and illumination variations.

Image Condition	Transformation Setting	Mean Confidence	Change from Original
Original	0°, brightness = 1.0	0.80	—
Rotation	−15°	0.76	−0.04
Rotation	+15°	0.77	−0.03
Rotation	−30°	0.69	−0.11
Rotation	+30°	0.70	−0.10
Low illumination	Brightness = 0.6	0.67	−0.13
Low illumination	Brightness = 0.8	0.75	−0.05
High illumination	Brightness = 1.2	0.78	−0.02
High illumination	Brightness = 1.4	0.72	−0.08
Average under perturbations	—	0.73	−0.07

Note: Mean confidence denotes the average highest confidence score obtained using the category-consistent natural-language prompt over the same 800 image–category pairs used in the text sensitivity test. No retraining or fine-tuning is performed. The change from original is calculated relative to the mean confidence of 0.80 under the original image condition.

.

As shown in Table 7, the proposed model maintains relatively stable semantic responses under moderate image rotation and illumination variations. The confidence degradation under $\pm 15°$ rotation and brightness factors of 0.8 and 1.2 is limited, whereas larger rotation angles and stronger illumination changes cause more noticeable decreases. Nevertheless, the model continues to produce substantially higher responses to the category-consistent prompts than to the irrelevant prompt reported in Table 6. These results demonstrate a certain degree of robustness to geometric and illumination perturbations, while also indicating that severe image transformations remain challenging.

The preceding experiments are conducted primarily on the official BDD100K validation set. To further evaluate model generalization, complementary experiments are conducted on the KITTI dataset [34] and the COCO 2017 validation set [35]. KITTI is used to evaluate cross-dataset transfer within the road traffic domain. Only the road object categories shared with or mapped to BDD100K are included in the mAP calculation. Since KITTI remains visually and semantically similar to BDD100K, five COCO categories absent from the BDD100K task-specific training labels, namely cat, dog, horse, sheep, and zebra, are additionally selected for unseen-category evaluation. Neither E1 nor E4 is retrained or fine-tuned on KITTI or COCO. The results are presented in

Table 8. Cross-domain inference results on KITTI and unseen-category results on COCO.

Evaluation Dataset/Category	E1 AP@0.5 (%)	E4 AP@0.5 (%)
KITTI mapped road categories	58.6	61.3
COCO—Cat	27.8	30.6
COCO—Dog	31.9	34.1
COCO—Horse	40.7	43.8
COCO—Sheep	34.6	36.9
COCO—Zebra	45.3	48.7
COCO unseen-category mean	36.1	38.8

.

As shown in Table 8, without retraining or fine-tuning, E4 outperforms E1 by 2.7 percentage points on KITTI, reaching 61.3% mAP@0.5. It also improves the mean AP@0.5 of the five COCO categories from 36.1% to 38.8%. These results support the contribution of the fusion adaptation design to cross-dataset transfer and text-conditioned detection beyond the BDD100K task-specific label space.

4.6. Evaluation of Edge Deployment Performance

To validate the deployment feasibility of the proposed multimodal detection model on edge devices, the fusion-adapted model is deployed on the NVIDIA Jetson Orin Nano Super platform [36], and an INT8/FP16 mixed-precision inference engine is built using TensorRT [29]. During inference testing, the input size is uniformly set to 640 × 640, the batch size is set to 1, and model warm-up is performed before formal timing to reduce the influence of first-run inference and cache initialization on the test results. During deployment, the CLIP text encoder does not participate in online edge-side inference. Instead, category-level text semantic prototypes are generated in advance during the offline stage. The main computations actually executed on the edge device include the lightweight visual encoder, DyHead feature enhancement module, semantic adaptation layer, cross-modal fusion modules, and text-contrastive detection head.

However, this design assumes that the target category set is predefined. When new categories or prompt expressions are introduced, the corresponding text embeddings must be regenerated and loaded into the inference engine. Therefore, the current deployment scheme is more suitable for fixed or slowly changing detection categories and is less flexible for applications requiring dynamic user-defined prompts.

First, the detection accuracy and model size before and after quantization are compared, and the results are shown in

Table 9. Comparison of accuracy and model size before and after quantization.

Precision	mAP@0.5 (%)	Model Size (MB)
FP32	64.9	115
INT8/FP16	62.1	32

.

As shown in Table 9, mixed-precision quantization reduces the model size from 115 MB to 32 MB, corresponding to a 72.2% reduction, while mAP@0.5 decreases from 64.9% to 62.1%. Retaining precision-sensitive cross-modal operations in FP16 therefore provides a practical trade-off between storage reduction and accuracy preservation.

Furthermore, the inference speed of the model before and after quantization is tested on the Jetson Orin Nano Super, and the results are shown in

Table 10. Inference speed comparison of the proposed model before and after quantization on the edge device.

Precision	Latency (ms)	FPS
FP32	54.9	18.2
INT8/FP16	30.7	32.6

.

As shown in Table 10, after TensorRT mixed-precision optimization, the single-frame inference latency is reduced, and the quantized model reaches an inference speed of 32.6 FPS, meeting the basic requirement for real-time inference. It should be noted that the FPS reported in this study mainly measures the forward inference process of the model. If image loading, preprocessing, NMS (Non-Maximum Suppression) post-processing, and result visualization are all included in the end-to-end pipeline, the actual system-level frame rate may be affected by I/O and post-processing overhead.

5. Conclusions

This paper addresses the coupled problems of computational reduction and cross-modal compatibility that arise when replacing the original YOLO-World visual encoder with a compact visual branch. Rather than proposing new lightweight convolution, pruning, or quantization operators, the study contributes a coordinated visual-branch replacement and adaptation methodology. The methodology integrates interface-constrained visual compression, multi-scale feature compensation, semantic feature adaptation, stable text prototype construction, and edge-oriented mixed-precision execution into a unified pipeline.

Experimental results show that the proposed lightweight visual encoder achieves 62.45% mAP@0.5 while reducing the parameter count and GFLOPs by 39.7% and 49.9%, respectively. After being integrated into the YOLOv8l-worldv2 multimodal detection framework, the proposed model achieves 64.9% mAP@0.5 with 31.5 M parameters and 67.2 GFLOPs. Compared with YOLOv8l-worldv2, the proposed model reduces GFLOPs from 204.5 to 67.2 with only a 1.6 percentage-point decrease in accuracy and improves the mAP/GFLOPs ratio from 0.325 to 0.966. These results indicate that the proposed method can effectively improve detection efficiency per unit of computation.

The expanded multi-category text sensitivity test, image perturbation robustness evaluation, and complementary evaluations on KITTI and COCO further demonstrate that the model remains responsive to text semantics and retains a certain degree of robustness, cross-domain transferability, and unseen-category generalization capability.

For edge deployment, the CLIP text encoding process is frozen and moved offline, and an INT8/FP16 mixed-precision inference engine is built using TensorRT. After quantization, the model size is reduced from 115 MB to 32 MB, and the inference speed reaches 32.6 FPS on the Jetson Orin Nano Super platform. These results verify the applicability of the proposed method on resource-constrained edge devices.

The proposed method still has several limitations. First, the semantic adaptation layer uses a single 1 × 1 convolution for linear mapping, which may have limited capacity for modeling complex distribution discrepancies between visual and textual semantic spaces. Second, the pruning ratio allocation depends on layer-wise sensitivity analysis, and future work may further optimize it by incorporating hardware-aware constraints or automatic search strategies. In addition, post-training quantization still introduces a certain degree of accuracy degradation. Future studies may explore local quantization-aware training or more fine-grained mixed-precision retention strategies. In addition, the current deployment evaluation is conducted only on the Jetson Orin Nano Super platform. Because inference latency, memory usage, and acceleration efficiency depend on the hardware architecture, inference framework, and operator support, the reported deployment results cannot be directly generalized to other edge devices. Future work will evaluate the proposed model on additional GPU-, NPU-, and CPU-based edge platforms, as well as on more diverse datasets, to further examine its generalization and hardware adaptability.