Search Results (443)

Transformer-based detectors model long-range context effectively, yet their representations remain dominated by RGB appearance and can become unreliable in cluttered, occluded, or crowded scenes. We present BMF-DETR, a pseudo-depth-guided detector that introduces RGB-derived geometric structure without requiring a depth sensor. DA3Mono-Large from Depth Anything 3 generates spatially aligned pseudo-depth maps offline, while two ResNet-50 streams encode appearance and relative geometry. Bidirectional cross-modal attention (BCMA) establishes two-way correspondence, and multi-strategy fusion (MSF) combines the streams through global calibration, channel allocation, and spatial gating before squeeze-and-excitation (SE) recalibration. On the fixed validation/evaluation split of the 2024 Roboflow-curated PASCAL VOC derivative, the complete model reaches 60.80 AP, compared with 52.80 AP for a capacity-matched dual-RGB control. BMF-DETR obtains 49.30 AP on COCO 2017. Its detector contains 58 M parameters and requires 103 GFLOPs; these figures exclude offline pseudo-depth generation. A shared-low-level variant retains 60.10 AP with 50 M parameters and 87 GFLOPs. The results show that pseudo-depth can serve as a useful auxiliary representation when its contribution is separated from capacity effects and evaluated under controlled fusion settings. Full article

Visible–Thermal (VT) object detection provides stable and all-weather perception for autonomous systems and remote-sensing intelligence. However, Tiny-Object Detection (TOD) remains a persistent bottleneck due to severe feature scarcity and distinct cross-modality distribution gaps. At tiny scales, targets offer near-zero textural details and easily vanish during standard downsampling operations. To address these challenges, we propose ProG-Net, a detection framework tailored for visible–thermal tiny-object detection. Specifically, we deploy a frozen Vision-Foundation Model (VFM) backbone to inherit highly generalized pre-trained representations. To counteract feature scarcity, we introduce an auxiliary point-prediction branch governed by a dedicated CM-Point-Head. This mechanism leverages explicit point-prompt guidance to force the frozen encoder to produce highly focused spatial priors, effectively isolating tiny targets from complex background clutter. Furthermore, we design a prompt-guided cross-modality fusion strategy via CM-Det-Neck and CM-Det-Head. This module aggregates and aligns dual-stream representations to compensate for severe modality imbalances. Extensive experiments on RGBT-Tiny and LLVIP benchmarks demonstrate that our proposed ProG-Net generally achieves superior overall performance across multiple evaluation metrics compared to state-of-the-art methods, providing a new insight for the community. Full article

With the rapid development of autonomous systems, there is a growing need for versatile yet low-cost platforms that integrate perception, decision-making, and control in real-time environments. This study presents the design and implementation of a multi-functional intelligent car system based on the STM32F103C8T6 microcontroller. The system integrates an OPENMV camera for visual recognition, a DY-SV8F voice module, a dual 4-channel grayscale sensor array for high-precision path tracking, and a PS2 remote controller. A hierarchical control architecture is adopted. The core role of the actuators, which drives the vehicle’s motion in response to visual and sensory feedback, is emphasized through a tightly coupled control loop that translates perceptual data into precise PWM signals for motor speed and direction regulation. The core contribution lies in the design of a novel path tracking algorithm featuring Gaussian-weighted sensor fusion, a finite-state machine for path feature recognition, and an adaptive PID controller with curvature-based feedforward compensation. Furthermore, this paper provides a detailed exposition of a lightweight color marker recognition algorithm operating in the hue, saturation, value (HSV) color space, including its mathematical formulation and system-level integration with the motion control loop for event-triggered multi-modal task execution. Experimental results demonstrate that the proposed system achieves a mean absolute path tracking deviation of 2.1 mm, a color detection accuracy of 100% under controlled lighting, and flawless execution of complex, event-driven task sequences. Full article

Multimodal remote-sensing classification aims to improve pixel-level land-cover recognition by exploiting complementary information from hyperspectral images and auxiliary modalities. Under limited labelled data and pronounced cross-modal discrepancies, however, existing methods often focus on feature-extraction architectures or fusion schemes, while paying less attention to whether the semantic-prior extraction branch can adapt to the current task distribution. Here we propose SGML-net (semantic-guided meta-learning network), a semantic-guided meta-learning framework that combines dual-stream feature extraction, adaptive feature fusion and a collaborative GCN-Mamba representation module. The GCN branch extracts task-relevant semantic priors from dynamic adjacency relations, whereas the Mamba branch captures long-range spatial dependencies. We further introduce a two-stage training strategy that first meta-learns task-adaptive semantic priors and then uses them as stable guidance for global semantic fusion. Experiments on the MUUFL, Houston and Berlin datasets yield average overall accuracies of 84.90%, 95.30% and 89.40%, respectively, supporting the effectiveness of SGML-net for small-sample multimodal remote-sensing classification. Full article

Accurate Land Use and Land Cover (LULC) mapping in high-resolution alpine environments is challenging due to complex terrain, heterogeneous vegetation, seasonal snow and ice cover, and the limited spectral information provided by conventional aerial imagery. Although foundation models such as the Segment Anything Model (SAM) effectively capture structural features, their class-agnostic design, limits fine-grained semantic discrimination and typically requires large annotated datasets. This study proposes a multi-modal deep learning framework for alpine LULC mapping using sparse annotations, which would fall under the category of weakly supervised learning. The framework employs a dual-encoder architecture that integrates RGB imagery, six-band multispectral imagery, and custom adapters for spectral indices, and Digital Surface Models (DSMs). A SAM-based encoder extracts geometric and contextual features from RGB imagery, while a dedicated encoder learns complementary spectral representations from multispectral data. To address the boundary uncertainty introduced by sparse supervision, we propose post inference hybrid refinement strategy that combines a Canopy Height Model (CHM) derived from the DSM to improve tree crown delineation with edge-based refinement for low vegetation classes, such as shrubs, and mathematical methods to refine road and building edge delineation with DSMs. Experimental results across fourteen alpine classes highlight that the framework achieves a validation mIoU of 0.777, macro-averaged over the fourteen classes. For the present sensor choice and classification scheme, no comparable multi-modal baseline exists. Thus, results are reported in absolute terms. The present multi-modal data fusion sets the baseline for future scalable alpine LULC mapping applications. Full article

Visible-infrared (RGB-IR) object detection leverages multimodal information to ensure reliable perception in complex environments. However, dynamic scenes pose significant challenges due to the frequent inconsistency between scene-level modality contributions and local spatial reliability. Furthermore, standard feature extraction progressively attenuates boundary-sensitive structural cues, and unified fusion strategies often fail to capture spatially varying cross-modal complementarity. To overcome these limitations, we propose a Boundary-Guided Dual-Perspective Cross-Modal Fusion Network (BDPNet) to explicitly preserve shallow geometric structures and decouple deep semantic fusion into macroscopic and microscopic perspectives. Specifically, a Geometric Boundary Enhancement Module (GBEM) embeds Sobel-based high-frequency priors into shallow dual-modal features via residual spatial modulation, preventing the loss of crucial localization cues during downsampling. In the deep semantic space, a Hybrid Dual-Perspective Adaptive Fusion Module (HDAM) employs an illumination-aware branch for global modality weighting and a spatial confidence-driven branch for local cross-modal rectification. A spatial gating mechanism then dynamically reconciles these macro-environmental and micro-signal features. Extensive experiments on M3FD, LLVIP, and DroneVehicle demonstrate the effectiveness of BDPNet. Compared with state-of-the-art methods, BDPNet improves mAP_50–95 by 0.8% and 1.0% on M3FD and LLVIP, respectively, and improves mAP₅₀ by 0.6% on DroneVehicle, while using substantially fewer parameters and lower computational cost. Full article

This paper aims to address image degradation caused by adverse weather conditions and partial occlusion in autonomous driving scenarios. The network adopts a dual-branch architecture, which utilizes a residual fusion block (ResFuse) to achieve the deep fusion of cross-modal features. A dual-path attention mechanism (DualAtt) and an integrated high-frequency attention module (HFAtt) were designed to capture key information in feature maps. Additionally, this study designs a spatial-channel downsampling module (SCDown). This module implements spatial-to-channel image downsampling. Experimental results demonstrate that the proposed network achieves a detection speed of 61 frames per second (FPS) and precision (P) of up to 87.4% on the M3FD dataset. Comparative experimental results show that MHF-Net achieves 84.3% in the mAP₅₀ evaluation metric. Compared to the baseline model, the number of parameters was reduced by 95% and the computational complexity was reduced by 88%. Compared with other mainstream object detection algorithms, MHF-Net strikes a favorable balance regarding performance in autonomous driving scenarios. Full article

Multi-modal MRI provides essential anatomical and pathological information for accurate brain tumor segmentation. However, deep learning-based segmentation methods are hampered by limited annotated data and incomplete modality acquisition in clinical MRI datasets. To address this issue, we propose a synthetic enhancement framework based on SCGAN-MultiJNet for multi-modal brain-tumor MRI. Specifically, SCGAN establishes a dual-branch disentangled latent space to independently encode images’ structural contour and textural features. Combined with the multi-scale fusion capability of MultiJNet, the proposed network realizes effective modality translation. Subsequently, synthetic samples are mixed with BraTS2020 training data to optimize the U-Net segmentation model. With the optimal real–synthetic data-mixing strategy, the segmentation metrics are improved: accuracy, Dice, precision, and IoU increase from 0.9857, 0.8919, 0.8980, and 0.8054 to 0.9864, 0.8962, 0.9205, and 0.8122. The proposed synthetic augmentation experimentally demonstrates enhancements in model robustness against MRI disturbances (including Gaussian blur, brightness shift, etc.). Finally, cross-domain generalization experiments conducted on the BraTS2025-SSA-Data demonstrate that introducing synthetic data augmentation can effectively boost the model’s cross-domain generalization capability. Overall, the proposed SCGAN-MultiJNet-based data synthesis algorithm provides a feasible technical solution to break the data bottleneck in multi-modal MRI brain tumor segmentation. Full article

Multimodal Sequential recommendation alleviates the semantic insufficiency and data sparsity of item-ID-based models by incorporating side information such as text and images. However, multimodal systems face the dual challenges of feature-space heterogeneity and modality-specific noise, in addition to the dynamic evolution of user interests over time. Existing methods still struggle to jointly handle cross-modal alignment and time-aware preference modeling. To address these challenges, we propose a multimodal sequential recommendation framework with cross-modal alignment and temporal gating, which leverages item ID, text, and image modalities to capture users’ dynamic interests. The proposed model contains three core components. First, a cross-modal alignment mixture-of-experts module preserves modality-specific features with dedicated experts and captures shared semantics with common experts, thereby mitigating the semantic mismatch inherent in direct fusion. Second, a hierarchical time-aware mixture-of-experts module uses short-term intervals, long-term spans, and periodic time encodings for expert routing, and applies a time-aware modality gate to adaptively adjust the importance of ID, text, and image modalities under different temporal contexts. Third, a sequential interest contrastive learning objective enhances the discriminability of ID-based sequential interest representations by leveraging dynamic temperature scaling, multi-scale positive samples, hard negative mining, and diversity regularization. Experiments on games, beauty, and toys demonstrate that the proposed method consistently outperforms representative sequential and multimodal recommendation baselines on Normalized Discounted Cumulative Gain (NDCG)@5, NDCG@10, Mean Reciprocal Rank (MRR)@5, and MRR@10. Furthermore, ablation results validate the effectiveness of each proposed component. Full article

Mirror segmentation aims to identify mirror pixels from a single RGB image, yet remains challenging because mirrors provide weak intrinsic texture cues and their appearance is dominated by scene-dependent reflections under varying illumination and viewpoints. While recent models improve performance by leveraging contextual contrast, symmetry priors, frequency/spectral cues, or additional modalities (e.g., depth), many cross-cue or symmetry-aware designs still rely on direct spatial-domain fusion, such as concatenation, addition, or attention. Such fusion can amplify reflection-induced high-frequency variations and lead to leakage, shape distortion, and unstable boundaries. In this paper, we propose a symmetry-aware mirror segmentation framework that stabilizes cross-branch interaction via a frequency-domain cross-conditioned fusion mechanism. We build a dual-path Siamese encoder using the original image and its horizontally flipped counterpart, and introduce Heat Conduction Operator-based Cross Fusion (HCOCF), which performs heat-conduction-inspired spectral attenuation in the DCT domain. Unlike conventional fusion, HCOCF generates a nonnegative cross-conditioned attenuation coefficient map from the opposite branch and applies it to the DCT coefficient grid of the target branch. This produces a DCT-domain attenuation mask that controls the spectral refinement strength of each target feature stream, enabling global context propagation while suppressing unstable reflection-induced high-frequency responses without aggressive direct feature mixing. For multi-scale decoding, we adapt the cross-scale decoder of the baseline symmetry-aware architecture by replacing simple addition with conditional feature aggregation, which refines the HCOCF-enhanced features and improves boundary recovery. Extensive experiments on MSD, PMD, and RGBD-Mirror demonstrate competitive performance against representative supervised mirror segmentation methods. In particular, our RGB-only model achieves 88.47% IoU on MSD and 73.72% IoU on PMD, and remains competitive on RGBD-Mirror without using depth input. Full article

Retrofitting an internal combustion engine (ICE) dynamometer test bench for electric motor testing introduces significant measurement and control challenges. The two modes differ fundamentally in torque dynamics, communication protocols, and energy flow characteristics. This study presents a dual-mode measurement and control system for a retrofitted 250 kW test bench. The system retains ICE testing capability while adding motor testing through a shared LabVIEW interface. Three key technical problems are addressed: (i) a cross-mode PID controller with gain scheduling and bumpless mode switching; (ii) a wavelet-based dual-sensor torque fusion method using complementary filtering at a 50 Hz crossover frequency; and (iii) active front end (AFE) energy feedback optimization with quadrant-dependent DC bus voltage setpoints. Validation was performed on an 80 kW (240 kW peak) interior permanent magnet synchronous motor (IPMSM) across 823 operating points. The system achieved a peak system efficiency of 94.88%. Energy recovery efficiency improved from 58% to 82% through AFE parameter optimization. The proposed methodology offers a cost-effective template for extending existing ICE test infrastructure to support electric vehicle powertrain development. Full article

Short-term multimodal travel demand forecasting must reconcile shared mobility patterns with mode-specific dynamics and uneven spatial coverage. This paper proposes Multimodal Disentangled Dynamic Fusion (MDDF), which combines three mechanisms: dual-branch encoding with an orthogonality constraint to separate shared and modality-specific representations; Jensen–Shannon divergence regularization with coverage-aware masking to align shared features without introducing structurally absent observations; and attention-based dynamic fusion with an auxiliary ranking loss to adapt cross-modal information exchange. We evaluate MDDF over five random seeds on two multimodal datasets with distinct settings: 10 min bus, metro, and taxi demand over 491 traffic analysis zones in Shenzhen, and hourly bus, metro, and bike demand over 69 zones in Manhattan. MDDF achieves the lowest mean MAE in all nine mode–horizon settings on both datasets and the lowest mean RMSE in eight of nine settings on each dataset. On Shenzhen, MDDF improves upon the strongest competing baseline in all nine mode–horizon settings; at the 60 min horizon, the mean MAE reductions are 19.5% for bus, 18.3% for metro, and 5.9% for taxi. Ablation and representation analyses indicate broad, though not universal, benefits from disentanglement, alignment, and dynamic fusion. Full article

Motion deblurring is an essential capability for high-speed vision applications such as autonomous driving and unmanned aerial vehicles. Existing frame-based deblurring methods often struggle with rapid motion, whereas event cameras offer a promising alternative. However, irregular motion patterns and event noise remain challenging. To address these issues, we propose ALNet, an adaptive lateral-interaction spiking neural network for event-driven motion deblurring. ALNet adopts a dual-branch image–event architecture. In the event branch, we develop a Lateral Deformable Interaction Spiking Neural Network (LDSNN), which introduces lateral spike interactions and adapts the offset-and-modulation mechanisms of deformable convolution to the lateral interactions of spiking neurons. The design dynamically adjusts the interaction region, aiming to enhance motion boundary modeling and reduce sensitivity to event perturbations. For temporal feature extraction from event streams, we design an Image-Guided Spiking Transformer (IGST), which uses image-domain spatial context to guide the temporal processing of event spikes. For image–event feature fusion, we introduce a Cross-Modal Attention Fusion module (CMAF) to align and selectively fuse multi-modal features, thereby improving edge reconstruction. Experiments on the GoPro, REBlur, and Ev-REDS datasets show that ALNet contains only 6.08 M parameters while achieving competitive PSNR and SSIM performance, providing a favorable trade-off between restoration accuracy and model compactness. Full article

Reliable human target detection in smoke-filled environments is essential for firefighting robots and rescue perception systems. However, conventional RGB cameras are severely degraded by dense smoke, while a single infrared or thermal imaging sensor cannot simultaneously provide sufficient structural details and reliable target-related thermal information. To address these challenges, this paper proposes a local variance-guided adaptive infrared–thermal sensor fusion framework for human target detection in smoke-filled environments, aiming to alleviate smoke-induced degradation in multimodal perception through improved infrared representation and adaptive cross-modal information utilization. An improved dark channel prior-based infrared desmoking algorithm is designed, where guided filtering is employed to refine the transmission map, suppress halo artifacts, and enhance infrared image quality. Furthermore, a local variance-guided adaptive fusion strategy is proposed, which utilizes local variance as an information saliency metric to generate pixel-level adaptive modality weights for fusing desmoked infrared and thermal images. In addition, a lightweight YOLO11n detector is adopted to achieve efficient human target recognition while maintaining a favorable balance among detection accuracy, computational cost, and inference efficiency. Experimental results on the self-built dense-smoke dual-modal dataset demonstrate that the proposed framework achieves high detection performance with low model complexity and efficient detector-stage inference. The ablation results demonstrate the contribution of infrared–thermal multimodal fusion to reliable smoke perception and indicate that the proposed local variance-guided adaptive fusion strategy maintains comparable detection accuracy while providing a better detector-stage speed–accuracy balance than fixed-weight fusion. With a low parameter count, the adopted YOLO11n detector shows potential for future deployment on resource-constrained firefighting robotic platforms. Full article

Sentiment analysis of multimodal social media data is of great importance, not only for recognizing objective information but also for capturing subjective emotional states. While single-modal sentiment analysis has achieved notable progress, existing multimodal approaches still face two key challenges: (1) inadequate modeling of subjective emotional characteristics and (2) insufficient handling of cross-modal inconsistencies. To address these limitations, we propose an image-text multimodal sentiment analysis framework (CogSent) grounded in the psychological Dual-System Theory. First, inspired by human fast and slow thinking, we develop a Hierarchical Dual-Channel Cognition (HDCC) architecture to extract intuitive and rational affective features, respectively. Second, we introduce an Intuition-Guided Cognition Refinement (IGCR) module that uses System-I-inspired intuitive representations as affective priors to retrieve and refine System-II-inspired contextual representations via cross-attention. Third, we propose a Dynamic Cross-Modal Cognition Fusion (${\mathrm{DC}}^2$F) network that predicts sample-adaptive thresholds from modality discrepancy, agreement, and attention statistics, dynamically regulating directional image–text interaction to mitigate interference from conflicting cross-modal sentiment signals. Extensive experiments on public benchmarks demonstrate that CogSent achieves improved performance compared to existing methods and yields competitive results on multiple evaluation settings. Full article