Search Results (872)

Autonomous semantic indoor navigation requires robust low-level control and high-level understanding of objects and spatial context in cluttered and partially occluded environments. While deep reinforcement learning (DRL) methods such as twin delayed deep deterministic policy gradient (TD3) enable reactive obstacle avoidance, they typically struggle with long-horizon semantic navigation, where object-location memory and language-level reasoning are required. We present a lightweight hierarchical two-stage framework that, to the best of our knowledge, is introduced for the first time to integrate a locally deployed vision–language model (VLM), semantic object coordinate memory, and a TD3-based DRL controller for language-conditioned indoor navigation. In Stage 1, the robot performs semantic exploration using odometry, 2D LiDAR, and VLM-based object recognition to build a geometric map and store detected object categories with their estimated world coordinates in a structured javaScript object notation (JSON) semantic memory. In Stage 2, a natural language query is used to retrieve the target object coordinates from memory and pass them to a TD3 target point navigation policy, which performs mapless navigation using odometry and RealSense RGB-D perception. The proposed framework combines open-vocabulary VLM-based object coordinate estimation, LiDAR mapping, RGB-D perception, and language grounding within a unified semantic memory representation. Experiments in a ROS-integrated realistic simulation demonstrate consistent goal-reaching performance and improved navigation efficiency compared with an Artificial Potential Field baseline using the same VLM and a DRL + GPT-4o mini configuration. We also compare the proposed VLM-based recognition module with YOLO-World v2.6 and Grounding DINO, showing that the VLM-based approach provides more reliable semantic grounding and target-coordinate estimation in the tested indoor navigation scenarios, particularly for flexible natural language object queries. Full article

Referring remote sensing image segmentation (RRSIS) aims to segment a target instance from remote sensing imagery according to a natural-language expression. It provides a flexible way to retrieve and localize specific objects in remote sensing scenes, benefiting intelligent Earth observation applications. Although existing methods have achieved promising progress by strengthening vision–language alignment, most of them still represent the expression as a holistic language feature and rely on convolution-dominated decoding for mask prediction. Such a paradigm tends to entangle target category, inter-object relation, and spatial position cues, making it difficult to distinguish the intended instance from multiple same-class distractors in complex remote sensing scenes. To address this limitation, we propose SRGFormer, a graph reasoning framework for RRSIS. Specifically, a semantic role decomposition (SRD) module decomposes the referring expression into target, relation, and position semantics, providing explicit linguistic priors for instance-level localization. Guided by the decomposed relation semantics, a semantic-relational graph transformer (SRGT) performs relation-aware graph reasoning over fused multi-scale visual features, enabling long-range dependency modeling among spatially distributed candidate instances. Furthermore, a progressive mask refinement (PMR) module continuously injects the decomposed semantic priors into semantic modulation, query initialization, and iterative mask decoding, thereby alleviating semantic fading during mask generation. Extensive experiments demonstrate that SRGFormer achieves substantial improvements on RefSegRS, attaining 66.08% mIoU and 76.93% oIoU (surpassing the prior state of the art by 3.96% and 2.83%, respectively) along with a notable 15.95% gain in Pr@0.7. Experiments on the additional RRSIS-D benchmark further demonstrate the general applicability of our approach, where SRGFormer maintains competitive performance (65.87% mIoU and 24.61% Pr@0.9) against existing methods. These results demonstrate that the proposed framework improves target localization and fine-grained mask prediction in complex remote sensing scenes. Full article

Offline action prediction does not by itself guarantee reliable closed-loop flight for unmanned aerial vehicle (UAV) vision-language-action (VLA) models. We study a controlled AirSim Blocks waypoint task using 100 expert episodes and 3385 RGB-D, instruction, state, and action records. Checkpoints are selected only on val-seen data, after which val-unseen is evaluated once. A 132,840-parameter policy reaches $0.9278\pm 0.0019$ final-test action accuracy across three training seeds, yet raw VLA control fails in closed loop. We therefore embed its action proposals in progress-aware hybrid execution with explicit recovery and near-goal precision. The strongest checkpoint reaches 59/60 goals, but crossing three independently trained checkpoints with three target seeds yields a more conservative 147/180 successes (81.7%) with zero recorded collisions and marked checkpoint sensitivity. Substantial overrides and fallback-tagged steps further show that the reported closed-loop outcomes are properties of the hybrid system, not of the learned policy alone. These findings are restricted to the controlled AirSim Blocks benchmark and do not demonstrate real-UAV deployment, sim-to-real transfer, or field robustness. Full article

UAV-based remote sensing excels in rapid response, high timeliness, simple operation, and high degrees of automation, and has been widely applied for geological hazard monitoring. Deep learning methods based on unitemporal UAV images can only analyze the static appearance of a scene, while bitemporal change detection can capture the dynamic evolution of hazards; however, due to diverse geological landforms and topography, environmental noises such as vegetation cover, and dynamic weather conditions, change detection of geological hazards from UAV images based on traditional deep learning technology is not always effective. Therefore, there is an urgent need to utilize large vision-language models (LVLMs) to further improve the accuracy and robustness of the change detection model. Motivated by this, this paper proposes a novel remote sensing model for geological hazard monitoring, referred to as CLIFF (CLIP-BIT-EfficientNet), based on the multi-modal LVLM Contrastive Language–Image Pre-training (CLIP), the change detection network Bitemporal Image Transformer (BIT), and the classification network EfficientNet, along with corresponding datasets and model fine-tuning strategies. The proposed transfer fusion module bridges the CLIFF and BIT networks by aligning their feature distributions and dimensions, allowing the general knowledge of the LVLM and the task-specific knowledge of the learnable branch to reinforce each other. Furthermore, this integrated pipeline addresses the scarcity of labeled hazard data by allowing the BIT to train on larger public datasets, while fine-tuning EfficientNet on smaller hazard-classification datasets within the change area, making the approach more efficient and reliable than direct classification methods. Experimental results show that the proposed CLIFF algorithm outperforms state-of-the-art deep learning algorithms such as LightCDNet and ChangeFormer, with an IoU of 75.74% and an F1 score of 0.8689 for change detection. Meanwhile, CLIFF has an overall accuracy rate of 86.89% in identifying geological hazards along gas pipelines, such as crude oil spills, collapses, landslides, and floods, with per-class accuracies of 87.32% and 86.17% for crude oil spills and landslides, respectively. Full article

Multimodal large language models (MLLMs) face substantial memory bottlenecks when processing long visual contexts, such as videos and high-resolution images. Existing methods that allocate visual KV cache budgets using cross-modal attention entropy mainly estimate the distributional breadth of text–vision interaction and may overlook how visual relevance varies across query positions in the encoded multimodal context. We propose a cross-modal query-position variance-aware KV cache optimization method for efficient multimodal long-context inference. The proposed method combines cross-modal attention entropy with prefill-stage query-position variance computed from cross-modal attention to estimate layer-wise visual KV cache preferences. Based on this preference score, visual KV cache budgets are allocated across layers, and variance-aware token pruning is applied to retain high-importance KV states while directly evicting redundant visual tokens without feature merging. Experiments on the MileBench benchmark using LLaVA-v1.5-7B show that, under deterministic single-run evaluation and while retaining only 20% of the visual KV cache, the proposed method produces point-estimate performance close to the full-cache reference and higher point estimates on several fine-grained reasoning and retrieval subtasks. Additional representative-subtask evaluations under different visual cache budgets and on InternVL2.5-8B further provide preliminary point-estimate evidence that the proposed allocation signal is not restricted to a single cache ratio or backbone. System profiling further shows that the 20% cache setting reduces measured KV cache GPU memory from 1.28 GiB to 0.26 GiB and decoding latency from 100.28 ms/token to 92.85 ms/token. These results suggest that cross-modal query-position variance may help preserve sparse, query-dependent visual cues under low-cache-budget multimodal inference. Full article

Open-vocabulary semantic segmentation (OVSS) of unmanned aerial vehicle (UAV) imagery aims to recognize arbitrary text-specified categories in aerial scenes, but existing OVSS models often suffer from UAV-domain shifts. To provide a reproducible testbed, this paper constructs a unified UAV OVSS benchmark by reorganizing multiple UAV segmentation datasets into cross-dataset transfer settings with explicit category harmonization and seen/unseen vocabulary analysis. Based on this benchmark, we propose UAV-OVSeg, a Cost Aggregation-style dense matching framework enhanced in two complementary directions: an LLM-guided Category Expansion Module that converts raw category names into structured UAV-aware descriptions, and a DINO-enhanced Geometric Feature Fusion Module that injects local structure into dense visual–text matching. Under SynDrone training, UAV-OVSeg achieves 65.5% mean intersection over union (mIoU) and 78.0% mean accuracy (mACC), improving upon the CAT-Seg baseline by 2.6 mIoU and 3.2 mACC. Under Aeroscapes training, it achieves 64.5% mIoU and 76.8% mACC, improving upon CAT-Seg by 2.6 mIoU and 3.1 mACC. Additional analyses of LLM variants, object scales, prompt sensitivity, boundary quality, and computational cost further verify the effectiveness and reproducibility of the proposed UAV-oriented text–visual enhancement strategy. Full article

Generative data augmentation based on diffusion models has emerged as a promising approach for few-shot image classification. Existing methods, such as DA-Fusion, typically follow a “generate-once, use-directly” paradigm, which often suffers from uncontrollable generation quality, unstable semantic consistency, insufficient global diversity, and high sample redundancy. To address these limitations, we propose a two-stage Propose-and-Select framework for controllable data augmentation. This framework curates high-quality synthetic data offline, ensuring that no additional training overhead is introduced to downstream models. For selector optimization, our method eliminates the need for additional human annotations by leveraging the zero-shot prior knowledge of a vision–language model (CLIP) to construct relative-quality pseudo-labels. Furthermore, we develop an adaptive-temperature listwise ranking distillation objective to transfer quality-aware supervision effectively. We also introduce a multi-objective consistency regularization strategy to stabilize training and improve convergence. Under a strictly controlled augmentation budget, where all methods are provided with the same number of synthetic samples, the proposed approach consistently outperforms existing diffusion-based augmentation baselines across both few-shot classification benchmarks, achieving an accuracy of 79.58% on PASCAL VOC and 80.74% on the fine-grained Oxford 102 Flowers dataset. These results demonstrate the effectiveness of the proposed generation-selection paradigm in improving the quality, diversity, and semantic relevance of synthetic samples, thereby enhancing downstream few-shot classification performance. Full article

Electroencephalography is emerging as a promising conditioning modality for generative visual models. However, existing representation learning approaches often rely on high-capacity masked autoencoders and complex projection networks. When constrained to compact embedding dimensions to match vision-language models, these heavy transformer-based bottlenecks frequently suffer from representation collapse and lose critical signal dynamics. To address this, we propose a lightweight and projection-free autoencoder that directly outputs compact, Contrastive Language–Image Pre-training (CLIP)-scale latent vectors trained toward the CLIP embedding space. Our model adopts a U-Net-style architecture combining one-dimensional convolutional residual blocks for temporal dynamics and inter-channel attention modules for spatial dependencies, alongside skip connections to ensure stable reconstruction. Extensive experiments on visual perception datasets demonstrate that our approach successfully tracks complex signal amplitudes without collapsing. Under strict dimensional constraints, the proposed model achieves superior signal reconstruction fidelity across time and frequency domains using significantly fewer parameters than traditional masked autoencoder baselines. Furthermore, latent space visualizations and zero-shot retrieval tasks reveal that while the baseline collapses toward unstructured, near-chance representations, our architecture preserves emerging, partial semantic organization and retrieves several times above chance. This indicates that the proposed design preserves signal structure while exhibiting preliminary, above-chance semantic alignment, enabling integration into brain-driven generative pipelines. Full article

Egocentric multi-view image analysis refers to the processing of utilizing synchronized video streams captured from multiple wearable cameras worn on the head or body, providing complementary first-person perspectives of dynamic, real-world interactions. Unlike single-view egocentric vision, which may suffer from severe occlusions, motion blur, and limited field-of-view or traditional fixed-camera multi-view setups (assuming static geometry and controlled environments), egocentric multi-view systems leverage body-worn rigs to enable a more robust and flexible 3D understanding in open-world, mobile scenarios. In this work, we present a systematic survey of advancements in cross-view feature fusion, geometric consistency enforcement, open-world detection, human–object interaction (HOI) modeling, action segmentation, 3D reconstruction, and novel-view synthesis specifically tailored to wearable multi-camera platforms. Key datasets released between 2024 and 2026—including HOT3D (833 min of synchronized multi-view hand/object interactions from Project Aria and Quest 3), MultiEgo (first multi-egocentric dataset for 4D social scene reconstruction), and Ego-1K (large-scale 12-camera rig for dynamic 3D video synthesis) are thoroughly examined alongside an analysis of integrations with large language models (LLMs) and vision–language models that drive performance gains, typically in the 15–30% range over single-view baselines in hand tracking, HOI recognition, and reconstruction fidelity, although we show through a consolidated meta-analysis that this gain is task-dependent: larger for geometry-bottlenecked tasks such as in-hand object lifting, and smaller, method-dependent, or occasionally negative for semantic-recognition tasks such as keystep recognition under naive view fusion. These methods cover work in multi-view stereo, cross-view learning, and novel-view synthesis while addressing several real-time wearable constraints. Practical applications such as immersive Augmented Reality/Virtual Reality (AR/VR), assistive robotics, and healthcare monitoring are also discussed together with the challenges in motion calibration, benchmark diversity, and edge deployment ability. Thus, in this review, we attempt to fill a critical gap by focusing exclusively on wearable multi-view systems in an open-world setting, synthesizing the latest literature to chart future directions toward more embodied and continual learning agents. Full article

The generalization of safety mechanisms from text-only LLMs to Large Vision–Language Models (LVLMs) remains poorly understood, especially concerning their vulnerability to typographic adversarial prompts. In this paper we ask where, inside a model, the safe-versus-unsafe decision is causally made. Focusing on decoder-only LVLMs, we show that safety is encoded as a non-linear, distributed representation that bifurcates early at a distinct Safety Bifurcation Point (SBP)—e.g., Layer 9 in LLaVA-1.5 and Layer 5 in Qwen-VL—and we expose a Correlation–Causality Gap: late layers are highly linearly separable yet causally inert. To leverage this insight we introduce a multistage, causal (interpretability-guided) framework to locate and verify safety representations, and we propose CPH (Coupled Path Hijacking), a white-box attack that jointly patches MLP and self-attention activations at the SBP to deterministically hijack the safe computational path. On the two models for which we report full tabulated results, CPH attains high oracle-scored attack success rates of 99.43% (LLaVA-1.5) and 98.57% (Qwen-VL); we additionally observe 99.71% on InstructBLIP-Vicuna, whose Vicuna backend is likewise decoder-only. We contrast this decoder-only vulnerability with encoder–decoder designs (e.g., a T5 backend or a Q-Former front-end), in which cross-modal grounding precedes decoding; we advance the greater robustness of such architectures as an architectural hypothesis for future validation, not as a tabulated result of this study. These findings indicate that VLM safety is architecture-dependent and motivate architecture-aware defenses for trustworthy multimodal AI. Full article

Transparent objects such as glass containers, test tubes, and plastic bottles are common in robotic manipulation scenarios, but their refractive and reflective surfaces produce incomplete RGB-D geometry and make task-specific grasp selection unreliable. This paper presents an integrated vision-language system for transparent object perception and task-oriented grasping. First, we construct VLM-DRE, a transparent object image instruction dataset with 12,700 images and 38,100 image-instruction-bounding-box triplets. LoRA fine-tuning of Molmo-7B improves target click accuracy from 86.4% to 91.5% and IoU@0.75 from 57.5% to 69.1%. Second, MSR-Net performs monocular depth completion and mask prediction using multi-scale adaptive feature fusion and progressive feature refinement, achieving RMSE 0.066, mAP 98.61%, and IoU 94.12% on Syn-TODD, and RMSE 0.118, mAP 99.02%, and IoU 87.95% on ClearPose. Third, LMF-Net combines RGB-D cross-modal fusion with learnable multi-factor matching to rank AnyGrasp 6-DoF candidates, reaching 77.8% Top-1 and 90.5% Top-3 accuracy on TaskGrasp-Image and improving PRISM-Real success from 61.1% to 68.5%. On a RealSense D435i–Unitree Z1 Pro platform, the complete system obtains 85.4% success with manual clicks and 71.3% with VLM-predicted clicks, supporting perception-to-grasping integration while highlighting target localisation and runtime as deployment bottlenecks. Full article

Deepfakes generated by advanced AI models pose growing challenges to digital media authenticity. Large Vision–Language Models (VLMs) have recently been explored for image forensics due to their multimodal representation ability. However, many existing VLM-based deepfake detection methods keep the visual encoder fixed to preserve pre-trained knowledge, which may limit the model’s sensitivity to low-level inconsistency artifacts that are important for deepfake detection. To address this issue, we propose IncoTune, an inconsistency-aware tuning framework that introduces trainable vision-side adaptation into the visual encoder and applies Directional Low-Rank Adaptation (DoRA) to selected linear projection layers in both the visual encoder and the language model. We further report an empirical observation in the ablation study: under the evaluated adapter configuration, replacing LoRA with DoRA in the frozen-vision setting does not improve the average AUC, whereas DoRA provides additional gains when combined with vision-side adaptation. Experimental results on FaceForensics++, DFD, Celeb-DF, DFDC, and DFDCP show that IncoTune improves cross-dataset frame-level detection performance over the frozen-vision baseline and achieves competitive performance among representative frame-level methods, while updating only 27.0M adapter parameters during training. Robustness experiments further evaluate the model behavior under common image degradations. Full article

Accurate detection in the early stages of a fire is a crucial prerequisite for the efficient implementation of fire suppression and emergency rescue operations. Its accuracy and timeliness directly affect the control of disaster loss severity. Traditional fire detection methods mainly include three categories, which are manual inspection, sensor detection, and visual recognition. However, manual inspection is restricted by labor costs and time efficiency, making it difficult to achieve large-scale, high-frequency and real-time fire monitoring. Sensor detection is easily interfered by environmental factors such as temperature, humidity, and dust, leading to frequent false alarms and missed alarms. Visual recognition technology has shortcomings in aspects such as detailed feature perception, dynamic scene modeling, and reasoning robustness in complex environments, making it difficult to meet the requirements of high-precision detection. To address these issues, this study innovatively proposes a lightweight fire and smoke detection model based on semantic enhancement and frequency domain perception modeling, which is named the SemaFire you only look once (SemaFire-YOLO) model. The model constructs a large language and vision assistant (LLaVA) semantic guidance module, which uses a large language model to understand and guide the semantic features of images, thereby enhancing the saliency representation intensity of small and weak target regions. Then, a Haar wavelet-based downsampling module is adopted, which compresses spatial information while preserving high-frequency features such as flame edges and smoke textures, improving the accuracy of target recognition. Next, the convolution modulation mechanism is introduced to replace the traditional attention mechanism, enhancing the overall modeling efficiency and reducing computational overhead. Finally, a Dynamic Tanh normalization module is adopted to replace the batch normalization module in the traditional YOLO algorithm, strengthening the model’s representation stability and reasoning robustness under unstable input distributions. Experimental results show that the SemaFire-YOLO model achieves a mean average precision (mAP@0.5) of 64.30% on the fire image dataset, which is 0.8, 2.0, 0.6, and 3.8 percentage points higher than that of mainstream models such as YOLOv5n, YOLOv8n, YOLOv11n, and YOLOv12n, respectively. It exhibits better boundary detection capability and practical deployment potential. Through visual analysis, the results indicate that the improved SemaFire-YOLO model achieves more accurate detection and higher confidence in actual complex scenarios, further verifying the model’s robustness and accuracy in complex scenarios such as low contrast and dynamic fire conditions. Full article

Although unsupervised anomaly detection has shown promising performance in industrial visual inspection, many architectures still struggle with variable input resolutions, and anomaly scores are often difficult for end users to interpret. This study proposes a multi-stage hybrid workflow for pixel-level defect localization and structured reporting in bolt head images. The proposed SA-PatchCore framework customizes PatchCore by extracting multi-scale representations from a frozen deep feature extractor and supporting resolution-adaptive anomaly-map reconstruction through dynamic feature-map sizing. After anomaly detection, a Qwen3-VL-32B-based reporting module, adapted with GRPO, uses both the original image and the anomaly overlay as visual evidence. It generates structured JSON outputs containing defect presence, a 3 × 3 location label, and a concise textual description. On the industrial bolt dataset, SA-PatchCore achieved 98.69% pixel-level AUROC, 29.73% Pixel-AP, and 39.24% oracle Pixel-F1max. Compared with PatchCore, PaDiM, DRÆM, and CS-Flow, the method delivered strong results, especially in Pixel-AUROC. In the reporting stage, defect presence/absence accuracy improved from 76.63% to 96.41%, while defective-sample recall increased from 74.23% to 96.14% over the baseline Qwen3-VL-32B. Exact location match rose from 28.15% to 53.78%, and mean partial location score improved from 32.29% to 65.34%. Overall, the framework combines accurate anomaly localization with structured reporting, improving interpretability and usability. Full article

Personal voice activity detection (PVAD) identifies whether each detected speech frame originates from a designated target speaker. Modern PVAD systems are typically trained offline and then deployed with frozen model parameters and a fixed, pre-enrolled speaker embedding, leaving them unable to adapt to distribution shifts at inference time such as unseen acoustic environments, changing speaking styles, or mismatches between enrollment and test conditions. Test-time training (TTT) and test-time adaptation have shown promise in language, vision, and several speech tasks, yet their behavior on PVAD has not been studied. In this work, we present an empirical study of two complementary test-time adaptation mechanisms built on top of the recently proposed FDE-Mamba backbone. The first is a VAD-gated TTT adapter, which instantiates the TTT-Linear formulation within the personalization pathway and augments it with a VAD-probability gate and exponential moving-average stabilization, adapting an internal weight matrix on the speaker-conditioned feature stream of each test utterance. The second is TEA (Test-time Embedding Adaptation), a scheme that keeps all model parameters frozen and instead adapts the target speaker d-vector itself via self-supervised objectives at inference time, directly targeting enrollment–test mismatch. We evaluate both mechanisms on the LibriSpeech PVAD benchmark across two backbones (LSTM-based FDE-RNN and Mamba-based FDE-Mamba), reporting category-wise average precision, mean average precision (mAP), accuracy, recall, precision, and real-time factor. We further isolate the effect of a post-hoc Gaussian smoothing step and report that, of all the components we examine, this task-agnostic smoothing accounts for the largest single accuracy gain on the FDE-Mamba backbone (accuracy $89.87\%\to 90.47\%$); the test-time adaptation mechanisms contribute a separate, smaller gain that is concentrated on speaker-discrimination metrics (mAP, precision) rather than on accuracy. Overall, the proposed test-time adaptation yields a consistent but modest improvement over the FDE-Mamba baseline (mAP $0.9605\to 0.9641$, precision $0.881\to 0.899$), while slightly reducing recall and increasing inference cost when TEA is enabled. Through ablation studies, we quantify the independent and combined contribution of each component and characterize the recall–precision trade-off introduced by adaptation. These findings, together with a discussion of their limitations and cost–benefit profile, provide a measured baseline and design insights for future work on adaptive PVAD, particularly under stronger acoustic and enrollment mismatches than those captured by the LibriSpeech protocol. Full article