Search Results (151)

Background: Integrating human neural signals with computational vision systems offers a promising route toward more robust visual recognition, yet supporting mixed-granularity recognition, where both coarse- and fine-grained categories must be distinguished within a unified system, remains challenging due to the heterogeneous feature spaces of electroencephalography (EEG) and visual data. Methods: We propose “Align and Fuse,” a two-stage Transformer-based framework. Stage 1 constructs a shared semantic space using a hardness-aware multimodal supervised contrastive loss with Hard Negative Weighting to explicitly target confusable class pairs. Stage 2 employs a multimodal Transformer with co-attention to fuse the aligned features for classification. Results: On the 80-class EEG-ImageNet benchmark, our framework achieved 91.12% Top-1 accuracy under a temporally separated control protocol, improving over the corresponding vision-only (89.08%) and Standard Transformer (89.95%) baselines. Under the original stratified random split, it achieved 92.56% Top-1 accuracy; on the 40-class EEGCVPR dataset, accuracy reaches 95.82%. Cross-subject experiments yield 90.92% average Top-1 accuracy on four unseen subjects, and Grad-CAM analysis suggests that aligned EEG signals shift the model’s attention toward semantically relevant regions. Conclusions: Coupling hardness-aware alignment with decoupled multimodal fusion supports EEG-augmented recognition by leveraging complementary stimulus-related information under the evaluated protocols. Because EEG features are required at inference time, the framework is positioned as a human-in-the-loop EEG-augmented recognition system rather than a standalone vision model. Full article

Hyperspectral image (HSI) classification remains a challenging task due to high spectral redundancy, complex spectral–spatial correlations, and the limited availability of labeled samples. To address these issues, this paper proposes a novel framework termed M³-Mamba, which integrates language-guided, multi-level, and Mamba-based spectral modeling for hyperspectral image classification. The proposed M³-Mamba leverages high-level semantic priors derived from multimodal representations to guide discriminative spectral modeling, enabling effective interaction between semantic information and fine-grained spectral features. In addition, a frequency-aware Mamba-based state space module is introduced to efficiently capture long-range spectral dependencies while avoiding the quadratic computational complexity of conventional attention mechanisms. Meanwhile, a text-guided modulation strategy is designed to adaptively reweight spectral responses under semantic guidance, suppressing redundant or noisy bands and enhancing class-relevant spectral responses without compromising spectral fidelity. This semantic-to-spectral modulation allows M³-Mamba to better cope with spectral variability and inter-class confusion. Extensive experiments conducted on four widely used benchmark datasets, including Indian Pines, Pavia University, Salinas, and Houston datasets, demonstrate thatM³-Mamba achieves competitive overall accuracy, average accuracy, and Kappa coefficient under the adopted benchmark settings. Ablation studies further validate the effectiveness of each key component, confirming that the proposed framework demonstrates promising effectiveness for hyperspectral image classification. Full article

Multimodal 3D object detection benefits from the complementary nature of camera images and LiDAR point clouds. However, existing voxel–pixel fusion methods typically rely on relatively coarse cross-modal interactions, which limit fine-grained structural modeling and degrade performance on small safety-critical objects. To address this issue, we propose ETA-SDQF, an edge-texture-aware semantic dual-query fusion framework designed to enhance 3D perception of vehicles, cyclists, and pedestrians. The proposed method first introduces an edge-texture-aware image backbone (ETAIB) based on the discrete wavelet transform (DWT), which improves the representation of multi-scale fine-grained image features. Then, we design a dual-query-guided attention fusion (DQGAF) module, which leverages deformable attention to adaptively aggregate voxel-aligned multi-scale image features under joint semantic and edge-texture guidance. Finally, we adopt a hybrid 3D feature learning strategy inspired by PV-RCNN, combining voxel-based feature learning with PointNet-style feature abstraction for processing fused features. This design improves the utilization of voxel features enriched with image semantics, thereby facilitating more reliable 3D object proposal generation. Experimental results on the KITTI dataset demonstrate that the proposed framework achieves better performance compared to existing baseline methods. It consistently improves pedestrian and cyclist detection, while maintaining competitive performance on car detection across different difficulty levels, showing potential benefits on challenging KITTI samples. Full article

Although multimodal large language models (MLLMs) have achieved remarkable progress in visual question answering, they remain limited in tabular tasks that require fine-grained structured information perception and complex logical reasoning. This limitation primarily stems from the high density of structured information inherent in tables and the scarcity of high-quality instruction tuning data. To address these challenges and improve the model’s reasoning accuracy in tables, we propose MMTR, a strategy-guided multimodal table reasoning method with reflective self-correction. Mechanistically, we design a dual-LoRA architecture: the Strategy LoRA is responsible for generating structured reasoning steps, while the Reflection LoRA verifies and self-corrects these initial outputs. Their synergy empowers the model with a closed-loop capability of “reasoning–reflection–correction”. On the data front, we construct StrTab-QA, a large-scale dataset comprising question-answering, negative, and reflection samples, providing diverse supervision signals. During training, we further introduce a progressive “reasoning-to-reflection” fine-tuning strategy to gradually achieve cross-modal alignment and structural adaptation. Furthermore, coupled with an adaptive resizing and padding scheme, our approach effectively preserves table structures and minimizes information distortion during visual encoding. Extensive experiments demonstrate that MMTR consistently outperforms strong baselines across multiple table reasoning benchmarks. Full article

Ensuring safe coexistence between human operators and industrial robot manipulators is a critical challenge in collaborative manufacturing environments. Existing approaches rely either on dedicated safety-rated hardware, which is expensive and difficult to retrofit, or on purely vision-based classifiers that discard the precise kinematic state available from the robot controller, leading to unresolved visual ambiguities when different joint configurations produce similar appearances from fixed camera viewpoints. Kinematics-only approaches, while precise, lack the spatial context needed to disambiguate configurations near workspace boundaries. We propose RGBJointsNet, a late-fusion multimodal deep learning classifier that combines RGB visual features extracted by a frozen EfficientNet-B2 convolutional backbone with a compact kinematic stream processing the 12-dimensional joint angle vector of a dual-UR5 robotic cell. The model maps each observation to one of five mutually exclusive workspace zones: rest ($C_0$), nominal ($C_1$), extended ($C_2$), shared/collision-risk ($C_3$), and joint-limit/singularity ($C_4$). A dedicated simulation environment built on ROS 2 Humble Hawksbill and Gazebo Classic 11 was used to generate a labelled dataset of 54,309 frames and 162,927 RGB images from three calibrated overhead cameras, with analytic ground-truth labels derived from closed-form forward kinematics. Training on a CPU with a feature-caching strategy brings the per-epoch wall-clock time to seconds, making the approach tractable without GPU hardware. On the held-out test set, the model achieves 87.1% overall accuracy and a macro-averaged F1 score of 90.0%, with near-perfect recall of 99.3% for the safety-critical shared zone $C_3$. The trained classifier is integrated as an ROS 2 inference node capable of running at 10$\mathrm{Hz}$ on a standard workstation. Our results demonstrate that joint angle information is a decisive complement to RGB imagery for fine-grained, safety-oriented workspace classification in simulation-derived settings. Full article

The worldwide spread of semiconductor devices has driven a surge in electronic waste (e-waste), which reached 62 million metric tons in 2022 and is projected to exceed 80 million metric tons by 2030. E-waste contains hazardous substances such as cadmium and mercury, yet also represents a $57 billion annual opportunity through the recovery of valuable and critical raw materials (CRMs). However, formal recycling rates remain stagnant at 22.3%, largely due to limitations of current automated sorting methods. These systems primarily rely on visible-light (RGB) imaging, which lacks the spectral resolution needed to distinguish chemically similar polymers, complex metal alloys, and composite substrates on printed circuit boards (PCBs). This paper presents a multidisciplinary synthesis of AI-driven detection and classification for e-waste, bridging materials science and computer vision through three interconnected themes. 1. Material and Economic Context: The toxicological risks and economic drivers of semiconductor recycling are characterized, framing fine-grained material identification as essential for a circular economy. 2. Multispectral Sensing & Fusion: Sensing modalities such as near-infrared (NIR), hyperspectral imaging (HSI), and X-ray fluorescence (XRF) are assessed, and sensor fusion strategies, including early, late, and intermediate fusion, are reviewed for high-throughput industrial settings. 3. Deep Learning Benchmarking: 11 publicly available PCB datasets are analyzed, and the YOLO series (YOLOv3–YOLOv12) is compared with leading non-YOLO detectors, including Faster R-CNN, RT-DETR-L, and RetinaNet. The results show that while YOLOv9s achieves a peak mAP@0.5 of 56.5% and YOLOv11s offers an optimal industrial profile (37.2% mAP@0.5:0.95 at 115 ms edge inference), all RGB-based models fail to detect visually ambiguous surface-mount devices (SMDs), with mAP values below 12%. This confirms a performance ceiling for purely visual systems. The review concludes that transitioning from RGB-centric to multispectral fusion architectures is the primary research frontier and proposes a roadmap for standardized multimodal datasets and edge-deployable fusion models to enable next-generation, high-recovery automated recycling. Full article

Text-to-motion generation aims to synthesize realistic human motion sequences that accurately reflect natural language descriptions. While recent approaches have improved motion quality, achieving strong semantic alignment between text and motion, especially for fine-grained articulations, remains a significant challenge. In this work, we propose a latent-guided text-to-motion generation framework that strengthens the interaction between textual representations and motion latent sequences. The proposed method integrates a structured motion latent space with a text-conditioned variational generation module, enhanced by a cross-modal attention mechanism. This design enables the model to effectively capture both global motion dynamics and detailed semantic information from text. Extensive experiments on the Motion-X dataset demonstrate that the proposed approach achieves strong semantic alignment, as reflected by improved R-precision and competitive matching performance. In addition, the model improves multi-modality, indicating its ability to generate diverse motion patterns under the same textual condition. Qualitative results further show that the generated motions preserve core action semantics and exhibit coherent temporal dynamics across different motion categories. Overall, the proposed framework provides an effective solution for improving text–motion alignment in high-dimensional motion spaces, highlighting the importance of latent-guided modeling for realistic and semantically consistent motion generation. Full article

To address the limitations of vision-only defect detection, image–semantic misalignment, and spatial-logic conflicts in complex substation inspection scenarios, this paper proposes a camera-sensor-based multimodal defect sensing framework with cross-modal feature mapping for substation equipment monitoring. The proposed framework integrates field inspection images acquired by camera sensors, defect textual descriptions, and equipment topology knowledge and establishes a unified domain-adaptive pre-training–bidirectional cross-modal mapping–hierarchical reasoning workflow. First, a Contrastive Language–Image Pre-training (CLIP)-based domain-adaptive pre-training strategy is developed to enhance the representation of equipment categories, defect attributes, and inspection-scene semantics. Second, a bidirectional cross-modal feature mapping network is constructed to model fine-grained interactions between candidate visual regions and textual semantics, where uncertainty-aware fusion and prototype constraints are introduced to improve semantic alignment and defect discrimination. Third, a hierarchical neuro-symbolic reasoning module incorporates equipment topology and spatial rules for posterior verification, logical consistency checking, and false-positive suppression. Experiments on a substation inspection image dataset demonstrate that the proposed method achieves 90.8% mAP@0.5, 68.7% mAP@0.5:0.95, and 89.4% F1-score, outperforming mainstream and recent detection models. Full article

Myocardial scar and edema segmentation from multi-sequence cardiac magnetic resonance (MS-CMR) is important for myocardial infarction assessment, but remains challenging due to heterogeneous modal characteristics, severe class imbalance, and the small, ambiguous nature of pathological regions. To address these issues, a dynamic synergistic segmentation network (DSS-Net) is proposed for myocardial pathology segmentation. The framework adopts a coarse-to-fine strategy, in which a coarse stage first segments the myocardium to provide anatomical priors and region constraints, and a fine stage then delineates scar and edema within the myocardium-aware space. In addition, a Modality Dynamic Fusion Module (MDFM) is designed to adaptively emphasize pathology-relevant modal information, and a Stage Feature Aggregation Module (SFAM) is introduced to enhance cross-stage feature interactions and fine-grained lesion representation. Experiments on the MyoPS 2020 and MyoPS 2024 datasets demonstrate that DSS-Net achieves competitive and balanced performance, reaching Dice scores of 0.706 for scar and 0.753 for edema on MyoPS 2020. Additionally, compared with SOTA methods in the MyoPS 2020 Challenge, the proposed method attains comparable scar segmentation performance while maintaining a more balanced trade-off between sensitivity and specificity. These findings suggest that combining anatomical guidance with pathology-aware multi-modal learning is a promising strategy for robust myocardial pathology segmentation in MS-CMR images. Full article

Accurate identification of pathogenic yeasts is essential for clinical diagnosis and effective antifungal therapy. However, current approaches predominantly rely on microscopy-based models, which require large-scale annotated datasets and exhibit limited generalization across morphologically similar species. In contrast, light-scattering (LS) imaging captures the diffraction patterns generated by internal cellular structures, providing volumetric biophysical cues that extend beyond surface morphology, yet its indirect representations pose major challenges for feature discrimination. Our objective is to develop fast and accurate methods to detect various species of yeasts. We propose FPA-YeastNet, which is a frequency-enhanced single-modality deep learning architecture that improves yeast classification in LS images by leveraging discriminative frequency-domain features. Building upon this enhanced modality, we further propose FGCA-YeastNet, a frequency-guided cross-attention network designed to integrate LS and microscopy information for complementary representation learning. The proposed multimodal model facilitates synergistic interactions between volumetric scattering structures and fine-grained cellular textures through adaptive fusion and bidirectional attention, leading to improved robustness and interpretability. Comprehensive classification experiments conducted on a multimodal yeast dataset demonstrate that FGCA-YeastNet effectively bridges the performance gap between LS and microscopy modalities, achieving significant improvements over both unimodal and multimodal baselines. The FPA-YeastNet yields an average accuracy improvement of 6.26% compared with LS-only models, and FGCA-YeastNet further provides mean gains of 19.97% and 7.67% over unimodal and multimodal baseline models, respectively. Experimental results demonstrate the diagnostic potential of light scattering and microscopic imaging and underscore the effectiveness of frequency-guided multimodal collaboration for reliable and interpretable yeast classification in clinical microbiology. Full article

Multimodal image registration is a fundamental yet challenging task, particularly in remote sensing scenarios involving cross-platform, multi-temporal, and cross-modal data. The primary difficulty arises from the coexistence of large-scale geometric distortions and complex local appearance variations across modalities, which makes it difficult for a single-stage model to achieve both global alignment and fine-grained correspondence simultaneously. To address this issue, we propose MMARNet, a task-driven coarse-to-fine registration framework that explicitly decomposes multimodal registration into global geometric alignment and local correspondence refinement. Instead of treating registration as a unified problem, the proposed framework sequentially resolves distinct sources of error, leading to improved robustness and accuracy under challenging conditions. In the first stage, MMARNet learns geometry-aware global alignment by identifying structurally reliable regions across modalities and estimating large-scale transformations, effectively reducing the initial misalignment and normalizing the geometric space. In the second stage, the model focuses on residual local discrepancies by learning context-enhanced feature representations, enabling robust keypoint-level matching even under severe modality differences and nonlinear distortions. The two stages are designed to work in a complementary manner, where global alignment significantly simplifies the subsequent local matching process. Extensive experiments on three challenging multimodal datasets demonstrate that MMARNet achieves superior performance in both accuracy and robustness compared to existing methods. The results validate the effectiveness of the proposed problem decomposition and highlight the advantage of the coarse-to-fine optimization strategy for multimodal remote sensing image registration. Full article

Manual classification of breast cancer is resource-intensive, slow, and subject to inter-observer variability, motivating automated deep learning solutions. Most current methods rely on unimodal imaging data and struggle with domain generalization (DG) across varied clinical environments. We propose a Dual-Stage Multimodal Alignment approach that integrates breast ultrasound (US) imagery with clinical text reports to improve diagnostic stability. The method proceeds in two stages: (1) Local Correlation Alignment (LCA), which aligns fine-grained visual features with textual embeddings to capture localized lesion attributes, and (2) Global Attention Alignment (GAA), which applies multi-head self-attention to the joint visual–textual sequence to encourage domain-invariant representations. We evaluate the approach on a harmonized, leakage-free repository of 6880 images aggregated from six public US datasets (BUS-CoT, BrEaST, BUS-BRA, BUS-UCLM, BLUI, BUSI) under three protocols: independent benchmarking on BUS-CoT, pooled cross-dataset evaluation, and zero-shot domain generalization on unseen unimodal target domains. On the BUS-CoT benchmark, the 198M-parameter model reaches 0.8177 accuracy and 0.8852 AUC, on par with the 7-billion-parameter Qwen2.5-VL-7B with chain-of-thought reasoning (0.8064 accuracy, 0.8354 AUC) while using roughly 1/35 the parameter count. In the pooled setting, it is competitive with single-domain state-of-the-art methods on individual subsets (e.g., 0.9576 AUC on BUSI, 0.8741 accuracy on BUS-BRA). Under zero-shot transfer without clinical text, per-domain AUC ranges from 0.7360 to 0.8060 across four unseen targets, providing a lower bound under cross-scanner shift. These results indicate that task-specific multimodal alignment can rival large vision-language models in breast US diagnosis at a fraction of the parameter count. Full article

Brain tumor segmentation from 3D MRI presents significant challenges due to small lesion sizes, ambiguous boundaries, arbitrary spatial distributions, and heterogeneous morphological properties. To tackle these issues, this paper presents a fully automatic 3D brain tumor segmentation network that integrates morphological and anatomical information under a multi-task learning framework for whole tumor, tumor core, and enhanced tumor segmentation. We propose a multimodal feature fusion module to adaptively weight features from four MRI modalities (T1, T1ce, T2, FLAIR), enabling discriminative information integration and helping reduce modality intensity discrepancy and data imbalance. Furthermore, a ConvReXt downsampling module is introduced to preserve fine-grained semantic details by reducing information loss caused by conventional pooling. A dense parallel global attention module is also developed to capture both local details and long-range dependencies, addressing the limited receptive field of standard convolutions. Extensive experiments on the BraTS2020 dataset show that the proposed model obtains average Dice coefficients of 92.54%, 89.21%, and 86.54% for whole tumors, tumor cores, and enhanced tumors. The proposed model achieves competitive performance compared with state-of-the-art methods including nnFormer, validating that it can effectively fuse multimodal and multi-scale features and improve brain tumor segmentation accuracy. Full article

Object detection in complex environments remains challenging due to illumination variations, background clutter, and the presence of small objects. Multimodal detection methods based on RGB and infrared (IR) data have shown promising potential by leveraging complementary information across modalities. However, existing approaches still suffer from cross-modal feature misalignment, loss of fine-grained details, and insufficient semantic interaction. In this work, we introduce a novel dual-stream framework called DMNet, specifically tailored for visible and IR multimodal object detection. The architecture integrates four core components designed to tackle these challenges: surface detail fusion (SDF) for shallow feature alignment, wavelet feature extraction (WFE) for frequency-domain enhancement, context-guided enhancement (CGE) for semantic refinement, and adaptive spatial fusion (ASF) for multi-scale feature aggregation. We conduct extensive evaluations on three benchmark datasets, including M3FD, LLVIP, and VEDAI, demonstrating that DMNet achieves superior detection performance compared with existing methods. Experimental results confirm that DMNet outperforms existing approaches, achieving an mAP@0.5 of 78.4% on M3FD, 94.4% on LLVIP, and 59.0% on VEDAI. Notably, the model maintains a relatively compact parameter scale (5.72 million parameters) while achieving superior detection performance, making it suitable for practical deployment. These findings highlight DMNet as an effective and efficient solution for multimodal object detection under challenging conditions, especially in low-light and small-object scenarios. Full article

Background: Rare neurological diseases are challenging to diagnose from brain MRI because of their low prevalence, heterogeneous imaging patterns, and limited annotated datasets. Deep learning may support image-level recognition, but results from curated datasets without complete patient-level identifiers require cautious interpretation. Objectives: This study proposes RareNeuroXNet, a frequency-aware multi-branch attention framework for image-level classification of rare neurological diseases from brain MRI. The objective was to assess whether combining global anatomical, local fine-grained, and frequency-domain representations improves benchmark performance, calibration, and interpretability. Methods: RareNeuroXNet uses three complementary branches: a global branch for whole-image representation, a local branch for regional feature extraction, and an FFT magnitude-based frequency branch. Features are refined using CBAM attention, fused, and classified through a fully connected head. The model was evaluated on a balanced curated dataset with five rare neurological disease classes using five-fold cross-validation, ablation analysis, calibration metrics, internal baseline comparison, paired testing against DenseNet121 local-only, and Grad-CAM visualization. MCND was also used as a complementary cross-dataset neurological MRI benchmark, not as same-task external validation. Results: RareNeuroXNet achieved strong image-level internal benchmark performance, with accuracy of $0.9924\pm 0.0061$, macro F1-score of $0.9924\pm 0.0061$, macro AUROC of $0.9998\pm 0.0002$, and macro AUPR of $0.9992\pm 0.0007$. Calibration was favorable, with ECE of $0.0052\pm 0.0029$ and NLL of $0.0276\pm 0.0159$. Ablation results showed that the local branch was the dominant contributor, while FFT and CBAM provided supportive refinement. Compared with DenseNet121 local-only, RareNeuroXNet showed modest classification gains and clearer calibration improvements. Conclusions: RareNeuroXNet demonstrated strong controlled image-level benchmark performance with high discrimination, stable cross-validation behavior, favorable calibration, and Grad-CAM interpretability. However, possible correlated slices, duplicate images, or subject overlap cannot be excluded. Future work should use patient-level, same-task, multi-center external validation and 3D multimodal MRI analysis. Full article