Vision-Based Navigation and Perception for Autonomous Robots: Sensors, SLAM, Control Strategies, and Cross-Domain Applications—A Review

Abstract

Camera-centric perception has matured into a cornerstone of modern autonomy, from self-driving cars and factory cobots to underwater and planetary exploration. This review synthesizes more than a decade of progress in vision-based robotic navigation through an engineering lens, charting the full pipeline from sensing to deployment. We first examine the expanding sensor palette—monocular and multi-camera rigs, stereo and RGB-D devices, LiDAR–camera hybrids, event cameras, and infrared systems—highlighting the complementary operating envelopes and the rise of learning-based depth inference. The advances in visual localization and mapping are then analyzed, contrasting sparse and dense SLAM approaches, as well as monocular, stereo, and visual–inertial formulations. Additional topics include loop closure, semantic mapping, and LiDAR–visual–inertial fusion, which enables drift-free operation in dynamic environments. Building on these foundations, we review the navigation and control strategies, spanning classical planning, reinforcement and imitation learning, hybrid topological–metric memories, and emerging visual language guidance. Application case studies—autonomous driving, industrial manipulation, autonomous underwater vehicles, planetary rovers, aerial drones, and humanoids—demonstrate how tailored sensor suites and algorithms meet domain-specific constraints. Finally, the future research trajectories are distilled: generative AI for synthetic training data and scene completion; high-density 3D perception with solid-state LiDAR and neural implicit representations; event-based vision for ultra-fast control; and human-centric autonomy in next-generation robots. By providing a unified taxonomy, a comparative analysis, and engineering guidelines, this review aims to inform researchers and practitioners designing robust, scalable, vision-driven robotic systems.

1. Introduction

Vision has become a primary sensing modality for autonomous robots because cameras are compact, low-cost, and information-rich. When coupled with modern computer vision and machine learning algorithms, camera systems deliver a dense semantic and geometric understanding of the environment, enabling robots to localize, map, plan, and interact with their surroundings in real time. Rapid gains in the hardware (e.g., high-resolution global-shutter sensors, solid-state LiDAR, neuromorphic event cameras) and software (deep learning, differentiable optimization, large-scale SLAM) have pushed vision-based autonomy from laboratory prototypes to production systems across domains such as self-driving vehicles, smart factories, underwater inspection, and planetary exploration [1,2,3,4]. Yet operating robustly in the wild remains difficult: lighting variations, weather, dynamic obstacles, textureless or reflective surfaces, and resource constraints all challenge perception and control pipelines. Understanding the capabilities, limitations, and integration trade-offs for diverse vision sensors and algorithms is thus essential for engineering reliable robotic platforms.

1.1. Scope and Contributions

This review targets an engineering readership and offers a holistic synthesis of vision-based navigation and perception for autonomous robots. This review offers a panoramic overview of the literature by organizing the design space around a unifying taxonomy of vision sensors, SLAM algorithms, navigation and control strategies, and domain-specific deployments. It distills the canonical approaches, milestone results, and key engineering trade-offs while signposting specialized surveys and seminal papers for readers who need deeper detail. Building on the rich body of work reviewed in the recent literature, we contribute

• A unified taxonomy of vision sensors—from monocular RGB to event and infrared cameras, stereo, RGB-D, and LiDAR–visual hybrids—highlighting their operating envelopes, noise characteristics, and complementary strengths [2,5,6];
• A comparative review of localization and mapping algorithms, spanning sparse and dense visual SLAM, visual–inertial odometry, loop closure, semantic mapping, and the current LiDAR–visual–inertial fusion frameworks [1,7,8];
• An analysis of navigation and control strategies that leverage visual perception, including classical motion planning, deep reinforcement and imitation learning, topological memory, and visual language grounding [9,10,11];
• Cross-domain case studies in autonomous driving, industrial manipulation, autonomous underwater vehicles, and planetary rovers, illustrating how the sensor–algorithm choices are tailored to the domain constraints [4,12,13,14];
• Engineering guidelines and open challenges covering the sensor selection, calibration, real-time computation, safety, and future research directions such as generative AI, neural implicit maps, event-based control, and human-centric autonomy [15,16].

We found that the existing surveys each concentrate on a narrow slice of the field—e.g., vision-based manipulation [17], multi-task perception for autonomous vehicles [18], or single-sensor SLAM pipelines. None simultaneously covers the full spectrum of (i) vision sensor modalities (RGB, stereo, RGB-D/ToF, LiDAR, event, thermal); (ii) geometry-, learning-, and fusion-based SLAM algorithms; (iii) learning-based controllers (DRL, IL, visual servoing, VLN); and (iv) cross-domain deployments spanning ground, aerial, underwater, and planetary robots. This review provides a single reference for both algorithm developers and practitioners integrating vision into real robotic systems by covering the aforementioned subjects under an engineering lens.

1.2. Methodology

This article is written as a narrative engineering review; thus, it provides a broad, integrative overview of the field, sacrificing exhaustive, protocol-driven coverage of systematic or scoping reviews for conceptual synthesis across diverse subtopics [19]. Particularly, a narrative review was chosen because vision-based robotics is a broad and rapidly evolving field that spans many areas (e.g., sensors, algorithms, and application domains) that are often studied separately. A systematic review would require many narrow sub-queries or risk missing relevant developments, especially from applied research. In contrast, a narrative approach allows us to connect these diverse elements into a unified engineering perspective, linking sensor configurations to algorithmic performance and identifying trends that are relevant across different robotic platforms. Three relevant questions guide our synthesis:

RQ1 

What vision sensor modalities and fusion schemes are employed across robotic domains?

RQ2 

How do the current localization, mapping, and navigation algorithms exploit vision data, and what quantitative performance do they achieve?

RQ3 

What outstanding technical challenges remain, and what promising future trends are emerging?

These three questions were chosen to trace the full perception–action pipeline on which contemporary vision-based robots rely. RQ1 targets sensing: by cataloging the range of camera modalities and fusion schemes, it establishes the upstream hardware and signal characteristics that bound system performance. RQ2 addresses the algorithmic core, asking how localization, mapping, and navigation methods convert these raw signals into actionable state estimates and what measurable accuracy they achieve on standard benchmarks. Finally, RQ3 turns to forward-looking gaps, describing unsolved problems and emergent trends that will shape the next generation of autonomous platforms.

Our expert-guided search—conducted by two researchers in computer vision and robotics—reviewed flagship venues and arXiv preprints spanning 2000 to 2025 in a forward and backward reference exploration process, retrieving approximately 300 records. After duplicate removal and abstract screening, 160 papers underwent full-text appraisal. Of these, 150 publications met our engineering-focused inclusion criteria (i.e., reporting substantive engineering results, providing a comprehensive insight of sensors and algorithms, etc.), and 3 others were retained to contextualize the review methodology. The final 150 sources form the basis of this narrative synthesis, which is chosen over a protocol-driven systematic literature review because it allows us to integrate rapidly evolving cross-domain results that fall outside any single, strict taxonomy. A machine-readable spreadsheet summarizing the entry types, publication years, and top venues is provided as Supplementary Material.

1.3. Organization of the Review

Figure 1 illustrates the overview of the review, and the remainder of this study is structured as follows. Section 2 reviews the vision sensor modalities and compares their performance envelopes. Section 3 reviews the depth and scene understanding techniques, including classical geometry- and learning-based estimation. Section 4 details the visual localization and mapping algorithms, while Section 5 discusses navigation and control strategies that exploit visual feedback. Section 6 presents domain-specific case studies, and Section 7 summarizes benchmark datasets and evaluation protocols. Section 8 details the engineering design guidelines, followed by emerging trends and open challenges in Section 9. Finally, Section 10 concludes the review.

Figure 1. Schematic illustration of review.

2. Vision Sensor Modalities for Robotics

Robust autonomy hinges on selecting and fusing the right sensor suite. This section reviews the principal camera-centric modalities used in contemporary robots, summarizing their operating principles, advantages, and engineering trade-offs. Figure 2 illustrates a variety of vision sensors employed in mobile robotics, ranging from widely adopted technologies such as monocular and RGB-D cameras to cutting-edge innovations like event-based sensors.

Figure 2. Vision sensors used in mobile robotics.

2.1. Monocular and Multi-Camera RGB Systems

A single RGB camera supplies high-resolution appearance cues for object recognition and dense semantics but cannot infer metric depth without motion or priors, leading to scale ambiguity in monocular SLAM [5,20]. Multi-camera rigs expand the field of view to 360°, mitigating occlusions and improving place recognition and loop closure [21,22]. Although inexpensive and lightweight, pure RGB systems struggle under poor lighting, overexposure, or textureless surface conditions; additional sensors (e.g., IMU, depth, and LiDAR sensors) are typically integrated to recover the depth, scale, and robustness. The range of these sensors widely varies from a few meters to 500 m and can be expanded to a few kilometers when using specific algorithms [23,24].

2.2. Stereo Vision Rigs

Stereo cameras triangulate depth from the pixel disparities between two spatially separated lenses, yielding real-time metric range data [25]. The range of these sensors and the FoV can widely vary while maintaining a low to moderate cost. In the case of the ZED 2 system, the range varies from 5 to 40 m, depending on the baseline resolution, achieving a FoV of up to 120°. The performance depends on the baseline length, calibration accuracy, and scene texture; repetitive or low-texture regions, glare, and low light impair matching [26]. Nevertheless, stereo depth is passive (no active illumination) and immune to sunlight interference, making it a popular navigation sensor indoors and outdoors, often fused with indirect monocular cues for enhanced robustness [20].

2.3. RGB-D and Time-of-Flight Cameras

RGB-D sensors such as Microsoft Kinect or Intel RealSense project structured light or emit time-of-flight (ToF) pulses to obtain the per-pixel depth aligned with color imagery [5]. They excel for low-texture surfaces where stereo fails, supporting dense mapping and obstacle avoidance at video rates [26]. Their limitations include a short range (typically from 0.3 to 5 m), susceptibility to sunlight or infrared interference, and increased power draw. Recent ToF chips show improved range and outdoor tolerance, yet their cost and power remain higher than those for passive cameras [27]. Moreover, the FoVs of some commercial sensors are from 67° to 108°.

2.4. LiDAR and Camera–LiDAR Hybrids

Spinning or solid-state LiDAR emits laser pulses to build high-resolution 3D point clouds with a centimeter accuracy over tens of hundreds of meters, independent of the lighting [8]. It underpins long-range obstacle detection and drift-free SLAM, but the performance degrades in rain or fog or with transparent/specular objects, and the data are sparse at a distance [6,28]. Moreover, depending on the sensor, the power consumption is high, varying from 8 to 15 W [29]. Camera–LiDAR fusion combines the precise geometry of LiDAR with cameras’ semantics and dense textures, enabling robust perception in mixed lighting and weather [8]. Falling component prices [29] and solid-state designs are accelerating its adoption in mobile robotics.

2.5. Event-Based (Neuromorphic) Cameras

Event cameras asynchronously report per-pixel brightness changes with a micro-second latency and a >120 dB dynamic range [2,30] with a very low power consumption (0.25 mW) [31] and a narrow FoV (75° [32]). They incur no motion blur and sparsely encode only scene changes, yielding high signal-to-noise ratios at extreme speeds or in extreme lighting, ideal for agile drones or high-dynamic-range (HDR) scenes. However, event streams differ radically from conventional frames, demanding specialized algorithms for SLAM and object detection; the resolution and cost also lag mainstream CMOS sensors. Hybrid frame–event perception pipelines are an active research topic.

2.6. Infrared and Thermal Imaging Sensors

Near-infrared (NIR) cameras extend vision into low-light conditions using active IR illumination, while long-wave thermal cameras passively capture emitted heat (7.5–14 μm) and thus operate in complete darkness [33,34]. Thermal imaging penetrates moderate smoke, fog, and dust, supporting search-and-rescue, firefighting, and night-time driving [35,36,37]. These sensors can generate thermal images from a large distance [38] with a FoV from 12.4° to 45° [34]. Their cost and power consumption widely vary, from approximately 200 USD to approximately 7900 USD and from 0.15 W to 28 W, respectively [34]. Trade-offs include a lower spatial resolution, higher noise, a lack of direct depth, and opacity for glass or water. Calibration must account for sensor self-heating and emissivity variations.

2.7. A Comparative Summary

Each modality offers distinct operating envelopes (in terms of range, lighting, and latency), noise profiles, and integration costs, as detailed in Table 1. The attribute “Contributions from 2020” counts the number of conferences and journal articles published from 2020 to the date (2025), according to Scopus, in the context of robotic navigation. Cameras provide rich semantics but limited depth; LiDAR delivers accurate geometry yet sparse color; event sensors excel at high speeds; and IR cameras ensure visibility in darkness. Consequently, state-of-the-art robots increasingly adopts multi-modal perception, fusing complementary sensors through probabilistic or learning-based frameworks to achieve robust, all-weather autonomy.

Table 1. Comparison of vision sensor modalities.

Attribute	Description
Monocular RGB/Multi-Camera
Range	Algorithm-dependent; reliable within a few meters
FoV	Up to 360° (multi)
Power	Low
Cost	Low
Typical Use Cases	Appearance-based recognition, SLAM with priors, place recognition, loop closure
Contributions from 2020	128 (e.g., [20,21])
Stereo Vision
Range	5–40 m
FoV	120°
Power	Low
Cost	Low to moderate
Typical Use Cases	Real-time metric depth, indoor/outdoor navigation, robust passive depth sensing
Contributions from 2020	528 (e.g., [27])
RGB-D/ToF Cameras
Range	0.3–5 m (typical)
FoV	67°–108° horizontal
Power	Higher than passive
Cost	Higher than RGB
Typical Use Cases	Dense mapping, obstacle avoidance, low-texture surface handling
Contributions from 2020	743 (e.g., [5])
LiDAR/Camera–LiDAR Fusion
Range	Tens of hundreds of meters
FoV	360°
Power	8–15 W
Cost	Decreasing
Typical Use Cases	Long-range 3D mapping, drift-free SLAM, robust perception in varied lighting/weather
Contributions from 2020	1449 (e.g., [6,28,29])
Event-Based Cameras
Range	1–5 m
FoV	75°
Power	Very low (0.25 mW)
Cost	High
Typical Use Cases	High-speed motion scenes, HDR environments, drones, low-latency SLAM
Contributions from 2020	94 (e.g., [2,30])
Infrared/Thermal Imaging
Range	Long-distance
FoV	12.4°–50°
Power	Moderate–high but decreasing
Cost	Low to high
Typical Use Cases	Low-light/night vision, search-and-rescue, firefighting, night-time driving
Contributions from 2020	78 (e.g., [34,38])

3. Depth and Scene Understanding Techniques

Accurate depth perception is fundamental for collision avoidance, mapping, manipulation, and high-level scene reasoning. Robots derive metric structures either from geometric principles, learning-based inference, or combinations thereof. This section reviews the core techniques and their engineering trade-offs, and Figure 3 illustrates its content.

Figure 3. A schematic overview of the topics covered in “Depth and Scene Understanding Techniques”.

3.1. Geometry-Based Depth Recovery

Geometry-based depth recovery exploits projective geometry to infer range. We therefore begin with monocular motion, where the structure is triangulated but the global scale is ambiguous. Then, we cover stereo rigs whose fixed baseline converts pixel disparities into metric depth. Later, we conclude with active sensors that obtain range directly via structured light or time-of-flight. This sequence shows how adding calibrated hardware progressively resolves geometric ambiguities while introducing new practical trade-offs.

3.1.1. Structure from Motion and Monocular SLAM

In purely monocular setups, depth is inferred by triangulating feature tracks across successive frames (structure-from-motion). Modern monocular SLAM systems such as ORB-SLAM3 employ keyframe bundle adjustment to refine camera poses and 3D landmarks, but the reconstruction is determined only up to an unknown global scale [1,20]. The scale can be resolved through inertial fusion, known object sizes, or ground-plane constraints, yet meter-level drift may accrue without loop closure or external references.

3.1.2. Stereo Correspondence

Stereo rigs yield metric depth by converting pixel disparities into range:

$$z={\displaystyle \frac{fb}{d}},$$

(1)

where f is the focal length, b the baseline (i.e., the distance between the centers of the two camera lenses), and d the disparity. Classical block matching, semi-global matching, and more recent CNN-based cost volumes perform disparity estimations in real time on embedded GPUs [25]. The accuracy hinges on a well-calibrated baseline and textured surfaces; low-texture or reflective areas and occlusions lead to outliers [26]. Despite these limits, stereo is sunlight-immune and passive, making it attractive for outdoor drones and rovers.

3.1.3. Active Depth Sensors

RGB-D and time-of-flight (ToF) cameras bypass correspondence by directly projecting structured light or measuring photon round-trip times, delivering dense depth aligned with color at video rates [5]. They enable real-time dense SLAM (e.g., KinectFusion) but operate over short ranges and suffer in bright IR backgrounds; calibration drift between color and depth streams impacts their accuracy further.

3.2. Learning-Based Monocular Depth Estimation

Deep convolutional and transformer models can predict dense depth maps from a single RGB frame by learning geometric and semantic cues [39]. Supervised variants train on ground-truth LiDAR or stereo disparity, whereas self-supervised approaches exploit the photometric consistency between adjacent frames or stereo pairs, obviating costly depth labels [20]. Recent networks achieve sub-meter errors at 30 Hz on embedded devices, effectively augmenting monocular cameras with pseudo-range for obstacle avoidance and SLAM initialization.

3.3. Self-Supervised and Hybrid Approaches

Self-supervised depth methods jointly estimate depth and pose by minimizing image reprojection errors, often incorporating explainability masks and edge-aware smoothness losses to handle occlusions and textureless regions. Hybrid pipelines integrate CNN depth priors into geometric SLAM, yielding denser reconstructions and mitigating scale drift; CNN-SLAM fuses network predictions with multi-view stereo for robustness in low-parallax segments [40]. Similar ideas infill sparse LiDAR with monocular cues, producing high-resolution depth maps at minimal additional cost.

3.4. Sensor Fusion for Dense 3D Perception

No single sensor excels in all conditions; modern robots therefore fuse complementary depth sources. Visual–inertial odometry provides scale and high-rate motion, while LiDAR offers long-range geometry resilient to lighting. Camera–LiDAR depth completion networks align sparse point clouds with dense image edges, achieving a centimeter-level accuracy even under rain or fog [8]. Probabilistic occupancy fusion—either using a Truncated Signed Distance Field (TSDF) or a Euclidean Signed Distance Field (ESDF)—integrates depth from stereo, RGB-D, or LiDAR into consistent maps for planning. Emerging work has coupled event cameras’ optical flow with stereo’s disparity to boost the depth at high speeds and with a dynamic range.

3.5. The Quantitative Impact of Sensor Fusion on 3D Perception

Recent benchmarks confirm that fusing complementary sensing modalities can measurably enhance the 3D perception across a variety of operating conditions. On the KITTI depth completion benchmark [41], RGB + LiDAR methods such as PENet [42] reduced the root mean square error in the depth by roughly 30% compared with that under sparse LiDAR interpolation alone while maintaining real-time inference at 20–25 FPS [42]. Visual–inertial odometry evaluated on TUM VIE [43] and EuRoC MAV [44] shows that coupling a monocular camera with an IMU limits the translational drift to below 0.5% of the trajectory length, whereas vision-only baselines drift by 1–2% under fast motion [45]. In high-speed outdoor scenes from the DSEC dataset [46], stereo + event fusion yields an accurate dense depth at temporal resolutions unattainable using frame cameras alone, sustaining the performance when the motion blur is severe [46]. Indoors, RGB-D fusion on NYUv2 improves the reconstruction completeness by approximately 10 IoU points relative to that for RGB networks, particularly on low-texture surfaces [47]. Ablation trials further indicate that learning-based attention modules can down-weight unreliable inputs (e.g., LiDAR under fog), preserving the detection recall within 5% of the nominal conditions [48]. Collectively, these studies highlight that carefully designed fusion pipelines raise depth’s accuracy and robustness with a modest computational overhead, although latency trade-offs appear for the most complex architectures.

3.6. Discussion

Geometric methods guarantee physical consistency but falter on ambiguous or low-texture scenes; learning methods generalize the appearance priors yet risk domain shifts. Fusion strategies blend complementary strengths, trending toward multi-modal, self-supervised systems that adapt online. Remaining challenges include robust depth under adverse weather conditions, for transparent or specular objects, and in real-time inference on power-constrained platforms. Advances in neural implicit representations and generative diffusion models (cf. Section 9) promise to close these gaps further by unifying depth, semantics, and appearance into a single, continuous scene model.

4. Visual Localization and Mapping (SLAM)

Vision-based Simultaneous Localization and Mapping (SLAM) enables robots to estimate their six-degree-of-freedom (6-DoF) pose while incrementally reconstructing the surrounding scene. Today’s SLAM pipelines power everything from consumer Augmented Reality (AR) to planetary rovers, combining geometric estimation, probabilistic filtering, and, increasingly, deep learning. This section reviews the algorithmic spectrum, organized from low-level visual odometry to multi-sensor, semantic, and learning-augmented systems. Some of the key properties of the milestone SLAM techniques are displayed in comparison in Table 2. Figure 4 provides a schematic that illustrates the topics covered in this section, and Figure 5 displays a wheeled robot ready to explore a structured environment.

Figure 4. A schematic overview of the topics covered in “Visual Localization and Mapping”.

Figure 5. A wheeled robot exploring an industrial facility.

4.1. Visual Odometry Foundations

Visual odometry (VO) estimates frame-to-frame motion without enforcing global consistency. Early monocular VO tracked sparse features using RANSAC-based pose estimation, while recent direct methods minimize the photometric error over selected pixels. Though computationally lightweight, VO suffers from unbounded drift and scale ambiguity in monocular cases [49,50]. Consequently, most modern SLAM pipelines build upon VO for local tracking and add global optimization for drift correction, as exemplified by the ORB-SLAM family [1].

4.2. Sparse Feature-Based SLAM

Feature-based (indirect) SLAM detects salient points (e.g., ORB, FAST, SuperPoint) and refines the camera trajectories via bundle adjustment. The ORB-SLAM lineage epitomizes this paradigm: ORB-SLAM2 supports monocular, stereo, and RGB-D sensors, whereas ORB-SLAM3 extends further to multi-camera, fisheye, and visual–inertial configurations with multi-map Atlas management, becoming a default engine for mixed-reality headsets and service robots [1]. Advantages include its real-time speed, robustness to photometric changes, and efficient data association through binary descriptors; limitations arise in texture-poor scenes and under significant motion blur. Authors have reported a good real-time performance on a Core i7-7700 at 3.6 GHz with 32 GB RAM and CPU-only execution.

4.3. Direct and Dense SLAM

Direct approaches bypass explicit feature detection, optimizing photometric or depth residuals at the pixel level. LSD-SLAM pioneered large-scale semi-dense monocular mapping [50], while DSO improved the accuracy via sparse photometric optimization. On the dense end, KinectFusion and its successors fuse RGB-D frames into Truncated Signed Distance Fields (TSDFs) for surface reconstruction. BAD-SLAM exploits the dense depth stream of RGB-D cameras and runs a GPU-accelerated direct bundle adjustment that produces centimeter-level indoor reconstructions in real time [7]. This technique runs on a GTX 1080/1070 (CUDA ≥ 5.3) since the GPU front-end is mandatory for 30 Hz tracking. More recently, DROID-SLAM has replaced the classical geometry with a learned dense-correspondence network and recurrent bundle adjustment, yielding state-of-the-art accuracy on highly dynamic sequences, albeit with heavy GPU use [51]. Direct methods capture richer geometry than indirect pipelines but require photometric calibration and remain sensitive to lighting changes.

4.4. Visual–Inertial SLAM and Factor Graphs

Integrating inertial measurement units (IMUs) mitigates the scale ambiguity and improves the robustness under fast rotations or motion blur. Tightly coupled optimization frameworks, such as VINS-Mono, fuse pre-integrated IMU factors with visual landmarks in a sliding-window bundle adjustment. VINS-Fusion extends this idea by supporting stereo, monocular, and GPS factors; its solver has become a staple on micro-air-vehicles where the mass and power are constrained [52]. Open-source builds demonstrate 40 Hz VIO on the Jetson TX2/Xavier NX as well as desktop i7 CPUs. ORB-SLAM3 also offers a visual–inertial mode that benefits from IMU pre-integration [1]. Factor--graph formulations leverage the sparse matrix structure for real-time nonlinear optimization, achieving centimeter-level accuracy on aerial and handheld platforms.

4.5. LiDAR–Visual–Inertial Fusion

LiDAR provides geometry immune to lighting yet lacks semantics; cameras offer dense texture but struggle in adverse weather. Recent frameworks, such as LIR-LIVO, combine spinning or solid-state LiDAR with cameras and IMUs to achieve drift-free, all-weather odometry [8]. LIO-SAM departs from vision-centric design and tightly couples LiDAR, inertial measurement unit (IMU), and Global Navigation Satellite System (GNSS) measurements within a factor graph, delivering meter-accurate mapping at the city scale while still running in real time on commodity CPUs [53]. Learned cross-modal feature matching (SuperPoint + LightGlue) further enhances the alignment between LiDAR scans and images [54,55], boosting the robustness for large-scale mobile robots.

4.6. Loop Closure, Relocalization, and Multi-Session Mapping

Detecting loop closures is key to correcting accumulated drift. ORB-SLAM employs a bag-of-words place recognition system (DBoW2) to propose loop candidates; pose-graph or full bundle adjustment then enforces global consistency [1]. Learned global descriptors such as NetVLAD or HF-Net boost the recall under drastic viewpoint or lighting changes [56]. ORB-SLAM3’s Atlas merges multiple submaps on the fly, enabling lifelong, multi-session mapping. Robust relocalization allows for rapid recovery from tracking failures and supports teach-and-repeat navigation.

4.7. Semantic, Object-Level, and Neural Implicit Mapping

Augmenting geometric maps with semantics enhances the navigation and manipulation. Kimera integrates real-time metric–semantic SLAM with 3D mesh reconstruction and dynamic scene graphs, producing maps directly usable for high-level planning [57,58,59]. DynaSLAM masks dynamic objects via Mask R-CNN, yielding stable trajectories in crowded scenes [9]. Object-level frameworks such as SLAM++ represent scenes as recognized CAD models, producing compact, interpretable maps [60]. Recent neural implicit systems (iMAP, NeRF-SLAM) jointly optimize camera poses and a continuous radiance field, unifying the geometry and appearance into a learned representation [61,62]. While computationally demanding, these approaches foreshadow integrated metric–semantic mapping.

4.8. NeRF-Based Real-Time 3D Reconstruction for Navigation and Localization

Neural radiance fields (NeRFs) have rapidly progressed from offline novel-view synthesis to online dense mapping that can close the perception–action loop. The pioneering work iMAP showed that a small multi-layer perceptron (MLP) trained incrementally on RGB-D keyframes can already sustain ≈10 Hz camera tracking and 2 Hz global map updates, rivaling TSDF-based RGB-D SLAM on indoor sequences while producing hole-free implicit surfaces [61]. Building on this idea, NICE-SLAM introduced a hierarchical grid encoding: a coarse hash grid stores the global structure whereas per-cell voxel MLPs refine the local detail. The structured latent space yields sub-centimeter depth errors on ScanNet at real-time rates and scales to apartment-sized scenes without catastrophic forgetting [63].

Subsequent systems tighten the speed/accuracy trade-off along three axes.

• Hash-grid accelerations. Instant-NGP’s multiresolution hash encoding cuts both the training and rendering latency by two orders of magnitude—minutes of optimization fall to seconds, enabling live updates at $>15\mathrm{Hz}$ on a single RTX-level GPU [64];
• Dense monocular pipelines. NeRF-SLAM couples a direct photometric tracker with an uncertainty-aware NeRF optimizer to achieve real-time ($30\mathrm{Hz}$ tracking, $3\mathrm{Hz}$ mapping) performance using only a monocular camera, outperforming classical dense SLAM on TUM-RGB-D by up to $86\%$ in L1 depth [65];
• System-level engineering. Recent SP-SLAM prunes sample rays via scene priors and parallel CUDA kernels, reporting a $\times 10$ speed-up over NICE-SLAM while keeping the trajectory error below $1.5\%$ on ScanNet [66].

These real-time NeRF SLAM variants share a common structure: a fast keyframe-based pose front-end (often ORB- or DROID-style) supplies stable camera estimates; a parallel back-end incrementally fits a hash-grid NeRF in windowed regions; and a lightweight ray marcher renders the depth/color for both the reprojection loss and downstream planners. Compared with TSDF or voxel occupancy grids, implicit fields (i) fill unobserved regions plausibly, (ii) compress the geometry at $\mathcal{O}(10-100)\mathrm{MB}$ per room, and (iii) support photorealistic relocalization for loop closure and teach-and-repeat tasks.

Limitations and Outlook

The current systems still lag behind classical SLAM in pure CPU settings, and GPU memory constrains the outdoor range (tens of meters). Open challenges include (i) lifelong consistency across dynamic or revisited scenes, (ii) multi-modal fusion with LiDAR/radar for all-weather reliability, and (iii) real-time uncertainty quantification for safety-critical control. Despite the current limitations, the fact that real robots can already fly or walk using real-time NeRF maps shows that this kind of neural mapping is becoming a serious, primary approach to robot navigation.

4.9. Robustness in Dynamic and Adverse Environments

Real-world deployment faces moving objects, lighting shifts, rain, dust, and sensor degradation. Robust front-ends combine learned features (SuperPoint [54], LOFTR [67]), outlier-resilient estimators, and motion segmentation to isolate dynamic regions. DROID-SLAM demonstrates that learned dense correspondence and recurrent optimization can maintain the accuracy even in highly dynamic indoor and outdoor sequences at the cost of at least 11 GB of VRAM [51]. Multi-body SLAM jointly tracks camera and object motion [68]. Event cameras enhance tracking under high dynamic ranges or fast motion. Despite progress, fully reliable SLAM in all weather and day/night cycles remains an open challenge, motivating continued research into adaptive sensor fusion and self-supervised model updating.

Table 2. The key properties of certain milestone SLAM systems. The metrics are the EuRoC MAV Absolute Trajectory Error (ATE) RMSE averaged over the sequences reported in the original papers (NR: not reported).

System	Year	Sensor Suite	EuRoC ATE (Avg.)	Loop Closure	Semantic	Hardware
VINS-Fusion [52]	2018	Mono/Stereo + IMU + GPS	0.18 m	Yes	No	CPU
BAD-SLAM [7]	2019	RGB-D	NR	Yes	No	GPU
LIO-SAM [53]	2020	LiDAR + IMU + GNSS	NR	Yes	No	CPU
Kimera [58]	2020	Mono/Stereo/RGB-D + IMU	NR	Yes	Yes	CPU
DROID-SLAM [51]	2021	Mono/Stereo/RGB-D	0.022 m	Implicit	No	GPU
ORB-SLAM3 [1]	2021	Mono/Stereo/RGB-D/IMU	0.036 m	Yes	No	CPU

Table 2. The key properties of certain milestone SLAM systems. The metrics are the EuRoC MAV Absolute Trajectory Error (ATE) RMSE averaged over the sequences reported in the original papers (NR: not reported).

System	Year	Sensor Suite	EuRoC ATE (Avg.)	Loop Closure	Semantic	Hardware
VINS-Fusion [52]	2018	Mono/Stereo + IMU + GPS	0.18 m	Yes	No	CPU
BAD-SLAM [7]	2019	RGB-D	NR	Yes	No	GPU
LIO-SAM [53]	2020	LiDAR + IMU + GNSS	NR	Yes	No	CPU
Kimera [58]	2020	Mono/Stereo/RGB-D + IMU	NR	Yes	Yes	CPU
DROID-SLAM [51]	2021	Mono/Stereo/RGB-D	0.022 m	Implicit	No	GPU
ORB-SLAM3 [1]	2021	Mono/Stereo/RGB-D/IMU	0.036 m	Yes	No	CPU

5. Navigation and Control Strategies

Reliable robot autonomy couples visual perception with decision making and actuation. This section reviews control paradigms that exploit camera input, ranging from geometry-aware path planners to end-to-end learned policies, topological memories, and multi-modal instruction-following. Figure 6 displays the content of this section, while Figure 7 illustrates a commonly used hierarchical structure for robotic navigation, highlighting in Figure 7a the low-level control for actuator commands, in Figure 7b path planning for computing safe and feasible trajectories, and in Figure 7c high-level navigation for mission-oriented autonomy.

Figure 6. A schematic overview of the topics covered in “Navigation and Control Strategies”.

Figure 7. Hierarchical organization of robotic navigation and control.

5.1. Classical Vision-Aided Planning and Obstacle Avoidance

Early vision-based navigation relied on geometry reconstructed through visual odometry or SLAM to populate local cost maps, over which sampling- or optimization-based planners (e.g., D* Lite, MPC) generated collision-free trajectories. Monocular systems infer the scale via inertial fusion or scene priors [20,69]; stereo and RGB-D sensors provide the metric depth directly, simplifying reactive avoidance. Visual–inertial odometry supplies high-rate pose estimates for feedback linearization or nonlinear MPC, enabling agile flight and ground locomotion.

Recent RGB-D SLAM pipelines have incorporated explicit volumetric grids and neural scene representations (e.g., NeRF) to obtain photorealistic reconstructions and dense semantics [70,71]. Although these approaches improve the map fidelity and rendering speed [72,73], they intensify the CPU/GPU demands and increase the relocalization latency, especially when the scene evolves [74]. Point cloud registration remains a bottleneck: classical Iterative Closest Point (ICP) scales poorly with the map size, motivating faster optimization-free solvers [75] or feature-based variants that exploit Fast Point Feature Histogram (FPFH) descriptors and RANSAC pruning for large environments [76,77].

Overall, deterministic geometric stacks excel when the depth is abundant and the dynamics stay within the model assumptions, but they degrade when perception falters or when computational budgets are tight.

5.2. Visual Servoing for Robotic Navigation

Visual servoing is a fundamental technique in robotics and computer vision that enables a robot’s motion to be controlled by using visual information obtained directly from one or more vision sensors, typically cameras [78,79]. The principal objective of visual servoing is to minimize the error between a set of visual features measured in the current image and a desired set of these same features which corresponds to the desired configuration of the robot with respect to its environment or a target [80,81]. By maintaining this error close to zero, the robot is guided toward the desired position or pose [78], following a reactive navigation approach.

5.2.1. Techniques and Control Laws

Within the field of visual servoing, various techniques and approaches exist to achieve this goal. One of the most widespread techniques is Image-Based Visual Servoing (IBVS), which directly uses the visual features measured in the image plane in the control loop without requiring an explicit 3D reconstruction of the environment [78,79,81].

The control law in visual servoing is derived to generate the robot’s velocities (linear and angular) that minimize the error between the current and desired visual features [78,80,81]. The relationship between the temporal variation in the visual features and the camera velocities is described by an interaction matrix or a feature Jacobian [78,79,80,81]. A common control law is based on error minimization through optimization methods such as Gauss–Newton [80,82,83], which seeks to apply velocities that drive the visual features toward their desired values. For a robot following crop rows, this could involve controlling only the angular velocity while keeping the linear velocity fixed, adjusting it to center the row in the image [78].

An alternative to traditional IBVS—often dependent on extracting and tracking sparse geometric features such as points or lines—is Photometric Visual Servoing (PVS) [80,82]. Instead of geometric features, PVS uses the pixel intensities of the entire image as the visual feature. Its objective is to minimize the difference—often quantified by the Sum of Squared Differences (SSD)—between the current image and the desired image [82]. Variations such as Extended PVS or Enhanced PVS [83] incorporate preprocessing steps like zero-mean normalization (ZN) to improve the robustness against illumination changes and Gaussian filtering [82]. Because these methods use dense image information, they tend to offer a higher positioning accuracy than that with sparse-feature approaches [80].

More recently, advancements in deep learning have led to Learned Visual Servoing (LVS) [79]. This approach formulates visual servoing as a reinforcement learning problem, where a deep neural network learns a control policy directly from visual observations. Such methods often employ Deep Q-Networks (DQNs) to approximate the value function and can learn to predict discrete navigation actions [79]. The network input can be a visual representation that captures the relationship between the current and desired views, such as a dense correspondence map. These learning-based approaches aim to overcome the fragility of the classical feature-based methods under difficult visual conditions or when explicit information such as depth is unavailable.

Within the NeRF-IBVS framework, image-based visual servoing is used not only for navigation but also as a pose optimization module. Starting from a coarse initial pose, IBVS drives the virtual camera’s motion so that the features projected in the image rendered by the NeRF align with those in the query image, iteratively refining the pose estimate [84].

5.2.2. Sensors

The primary sensor for visual servoing is a camera [80,85]. Different types of cameras are employed depending on the application: monocular cameras [82,83], stereo pairs [78], and omnidirectional cameras [80].

Beyond on-board cameras, some visual servoing navigation systems may complement visual information with other sensors—such as a laser scanner—for tasks like obstacle avoidance [81].

A noteworthy aspect is that in certain approaches, the “desired sensor output” does not come directly from a physical measurement but rather from an environment model. This model may be pre-acquired—such as an orthophoto or a textured mesh [82,83]—or learned, such as a neural radiance field (NeRF) representation [84]. These models allow for the generation of images or visual information (e.g., pseudo-3D labels or depth maps) that act as the desired reference for robot control [84].

5.2.3. Path Planning and Navigation Strategies

Visual servoing is inherently a local control technique, designed to steer a robot toward a visually nearby desired pose [79,81,85]. To achieve long-range navigation or cover large areas, visual servoing is integrated into broader navigation strategies. These strategies often concatenate local visual–servo tasks to follow a route or sequence of waypoints [78,80,82]. The desired route may be defined by a visual memory—that is, an ordered set of key images [78,80,82]. The robot navigates by moving from one key image to the next, using visual servoing to converge to each image in the sequence.

In structured environments such as row-crop fields, the strategy may consist of following a continuous row using visual servoing and at the end of the row executing a maneuver or switching mechanism to move to the next row and follow it in the opposite direction [78]. A visual servoing system can integrate these behaviors—for example, by using a stereo pair and switching between the cameras to facilitate the transition [78].

Topological-map-based strategies combine visual servoing with an environment representation in the form of a graph [83,85]. Graph nodes represent significant places (such as intersections or doorways), while edges represent navigable segments—often corridors or paths—where visual servoing can be applied. A supervisory or planning algorithm organizes the sequence of elemental tasks (visual servoing, obstacle avoidance, feature estimation) to navigate between nodes [81].

Appearance-based techniques complement this approach by enabling the robot to monitor its progress along the route’s edges, comparing the current image with a set of reference images stored at the nodes or along edges [85]. Matching a reference image can indicate the robot’s approximate position and trigger the transition to a new task or route segment [83].

5.2.4. Advantages and Limitations

A major advantage of visual servoing is its ability to operate directly on visual information captured by on-board cameras, without necessarily requiring a global localization system (such as RTK-GPS) or the construction and maintenance of detailed environment maps [78]. This can result in more economical systems that are less susceptible to issues such as drift or the need for frequent map updates in dynamic environments [78]. Approaches that use dense information—such as photometric visual servoing (PVS) [80,82,83]—tend to offer a higher positioning accuracy than that with sparse-feature-based methods. PVS, particularly with techniques like zero-mean normalization (ZN) [82,83], shows robustness to global illumination changes, which typically affect geometric-feature-based methods. By using pixel intensities, these methods avoid the fragility associated with reliable feature detection, tracking, and matching [79,80].

Despite its advantages, visual servoing has significant limitations. The classical feature-based methods critically depend on the reliability of the feature detection and tracking process [79,82,83]. In environments with visual aliasing, constant changes (such as plant growth), or a lack of distinctive points, this dependence can be a source of fragility [78]. Feature quality and depth accuracy are crucial for IBVS’s performance, making it sensitive to depth-estimation errors [79]. Visual servoing is fundamentally a local control technique whose convergence domain is limited [80]; the robot can converge to the desired pose only if the initial visual difference is not too large. Therefore, it is not suitable for long-range navigation on its own and requires integration with trajectory planners or higher-level navigation strategies to cover significant distances [81,85]. A loss of overlap between the current and desired images can cause failures in classical IBVS methods.

5.3. Learning-Based Controllers

Learning-based controllers replace handcrafted rules with policies optimized from data. We begin with deep reinforcement learning, which learns navigation end to end but demands large-scale simulation. Next, we cover imitation learning, which leverages expert demonstrations to initiate policy training and reduce the risks associated with unguided exploration. Finally, we conclude with hybrid schemes that fuse classical planners with learned residuals or fallback controllers. This progression shows how successive techniques trade autonomy for efficiency and safety while preserving the ability to generalize.

5.3.1. Deep Reinforcement Learning

DRL formulates navigation as a Markov decision process, training policies to maximize task-specific rewards directly from pixels or latent embeddings. Progress in domain randomization and photorealistic simulators allows agents to acquire millions of samples before transfer, leading to champion-level drone racing and indoor navigation [3]. Nevertheless, high sample complexity and limited data efficiency remain open issues [86]. Hybrid SLAM-RL schemes mitigate this cost by re-using maps to bootstrap value functions or by constraining exploration with semantic priors [87]. Unsupervised topological SLAM combined with memory networks further enables DRL agents to generalize to unseen layouts without pose supervision [88], yet transfer under novel lighting or textures is still fragile.

5.3.2. Imitation Learning and Behavior Cloning

Imitation learning (IL) sidesteps unsafe exploration by cloning expert demonstrations. Behavior cloning provides strong priors for lane keeping or social navigation with limited data [89]. Dataset aggregation and inverse reinforcement learning address covariate shift and recover latent cost functions, respectively. In practice, IL often bootstraps policies that are later refined with DRL for improved robustness.

5.3.3. Hybrid-Model-Based and Learning Approaches

Hybrid pipelines blend the guarantees of classical planners with the adaptability of learned modules. Examples include learning residual policies that correct model mismatch or switching to DRL controllers when the perception degrades [10]. Such systems maintain safety via rule-based constraints while handling long-tail corner cases and distribution shifts.

5.4. Topological Visual Memories and Teach-and-Repeat

Topological navigation represents the workspace as a graph of key images with traversable edges, emulating human spatial memory. During the teach phase, robots store images; later, they repeat the route by matching the live view to the closest node and following edge cues [90,91]. Compared with metric SLAM, topological graphs are lightweight, adapt gracefully to dynamic obstacles, and can be built on resource-constrained platforms [92]. Learned place recognizers such as NetVLAD improve loop detection across viewpoint and lighting changes [56]. Graph neural enhancements predict reachability (Neural Topological SLAM) and accelerate planning [93]. NavTopo couples a topological map with global planning and local obstacle avoidance, obtaining an order-of-magnitude reduction in memory relative to that for metric grids while sustaining a centimeter-level accuracy [94]. Path planning from aerial imagery also falls into this family: Rodríguez et al. converted the photometric similarity into traversable visual paths using topological graphs [82,83]. Recent cognitive models have fused neural odometry with perceptual similarity and multi-factor attention to extract task-specific memory features, boosting the success under severe appearance changes [95]. Key limitations include the dependence on representative keyframes and sensitivity to persistent scene alterations.

5.5. Visual Language Navigation (VLN)

VLN agents interpret free-form human instructions to reach described goals, fusing vision, language, and sometimes memory. Structured scene memories (SSMs), bird’s-eye scene graphs (BSGs), and volumetric environment representations (VERs) ground linguistic tokens in a 3D context [96,97,98]. Multi-task backbones such as VIENNA and LANA couple instruction-following with map summarization or route explanation, enhancing transparency [99,100]. Modular schemes that combine large language models with visuomotor primitives (e.g., LM-Nav, VLMaps) reduce task-specific fine-tuning and scale to long-horizon commands [11,101]. Depth-aware agents such as SpatialBot leverage RGB-D question-answering to reason over spatial relations and support manipulation [102]. Open challenges include grounding under partial observability, mitigating hallucination, efficient long-term planning, and extending language-guided control to non-static, open-world settings [103,104].

5.6. Vision-Guided Manipulation, Visual Servoing, and Inspection

In industrial contexts, cameras enable pick-and-place, assembly, and quality inspection. Deep grasp networks infer the suction or gripper poses for cluttered objects [105]. Closed-loop visual servoing adjusts robots’ motion to minimize the image error; recent work has achieved $0.25$ mm peg-in-hole insertions using high-rate 6D tracking with passive compliance [13]. AI-driven inspection stations employ convolutional networks to surpass human defect detection accuracy [106]. Vision–force fusion further handles contact-rich tasks, generalizing across part geometries. Classical feature-based servoing is being supplanted by end-to-end policies that learn control directly from images while retaining provable stability margins [107].

5.7. Discussion and Outlook

No single control paradigm dominates across all domains. Geometry-based planners remain indispensable for verifiable safety, while learned policies excel in unstructured scenarios and tight human interaction loops. Recent evidence suggests that hierarchical stacks—combining metric maps for global reasoning, topological memories for efficient search, and end-to-end visuomotor modules for local, perception-aware control—achieve the best trade-off between computational cost and robustness. Future work must address (i) the reliability under distribution shifts and severe appearance changes; (ii) data efficiency through self-supervised and simulation-augmented learning; (iii) long-term memory management for lifelong operation; and (iv) tighter integration with emerging sensor modalities such as event cameras, neural radiance fields, and generative scene priors.

Quantitative results on standardized simulators illustrate both progress and remaining gaps. On the CARLA benchmark, the “Learning by Cheating” agent attains a perfect $100\%$ route completion (RC) score, whereas the current leaderboard topper TransFuser achieves $86\%$ RC on the MAP track [108,109]. For Habitat PointNav, the distributed Decentralized Distributed Proximal Policy Optimization (DD-PPO) policy reaches a $99.9\%$ success rate (SR) with a success weighted by path length (SPL) of $0.96$ [110]; OccAnt, which foresees the occupancy to shorten paths, still secures a $93\%$ SR and a $0.80$ SPL with far fewer training frames [111]. Habitat’s more challenging ObjectNav task exposes the headroom that remains: the 2020 baseline manages only a $6\%$ SR, yet CLIPFields lifts this to $53\%$ under the same protocol [112]. We report the SR and the SPL following the definitions of [113]. A comprehensive and recent summary of CARLA’s results is provided in [114], while the leaderboards for the Habitat Challenges 2019 and 2020 can be consulted online: https://eval.ai/web/challenges/challenge-page/254/leaderboard/839 (accessed on 6 July 2025), https://eval.ai/web/challenges/challenge-page/580/leaderboard (accessed on 6 July 2025). These public numbers underscore the rapid advances in closed-world simulation yet also reveal a sizable performance gap between point-goal navigation and object-goal reasoning—highlighting the need for a more holistic understanding of perception, memory and language in future autonomous systems.

6. Cross-Domain Application Case Studies

Vision-based autonomy manifests differently across engineering domains, each imposing unique environmental constraints, safety requirements, and cost structures. Figure 8 illustrates the content of this section. We highlight four representative case studies plus emerging platforms.

Figure 8. A schematic overview of the topics covered in “Cross-Domain Application Case Studies”.

6.1. Autonomous Road Vehicles

Camera-centric stacks in self-driving cars fuse multi-camera perception with LiDAR and radar for robust 360° situational awareness [115]. Geometric SLAM supplies ego-motion, while deep networks perform 3D object detection from lifted BEV features [116]. Benchmarks such as KITTI, Cityscapes, and 4Seasons expose challenges ranging from dynamic traffic to night-time and seasonal variations [12,41,117]. The commercial trends favor solid-state LiDAR–camera fusion for long-range detection, with learned depth completion and NetVLAD-style loop closure to handle sparse urban loops.

6.2. Industrial Robots and Smart Factories

Eye-in-hand RGB-D cameras guide grasp networks for bin picking and flexible assembly [105]. Closed-loop visual servoing combined with passive compliance achieves sub-millimeter insertions without force sensors [13]. AI-powered vision inspectors surpass human accuracy for defect detection and dimensional checks, leveraging synthetic data augmentation to address data scarcity [106]. Safety standards drive redundancy: dual-camera setups validate each other, and depth gating prevents inadvertent collision with human co-workers.

6.3. Autonomous Underwater Vehicles (AUVs)

In GPS-denied oceans, AUVs rely on monocular or stereo cameras fused with Doppler velocity logs and pressure sensors for visual–inertial SLAM [14]. Photogrammetric pipelines create high-resolution seabed mosaics for ecological reviews, while visual docking systems detect LED patterns to achieve centimeter-level alignment [118]. Event cameras and specialized de-hazing networks mitigate low-light and turbid conditions, but color distortion and refraction remain open research problems [119].

6.4. Planetary and Lunar Rovers

Mars and lunar rovers employ mast-mounted stereo pairs for terrain mapping, hazard avoidance, and slip prediction [120]. NASA’s Perseverance drove most of its first $17.7$ km via on-board vision-based AutoNav, outperforming earlier generations [4]. Field tests combine visual–inertial odometry with crater-based absolute localization and LiDAR flash sensors for shadowed regions [121]. The Chinese Zhurong similarly demonstrated autonomous navigation on the Martian surface [122].

6.5. Emerging Platforms: Drones and Humanoids

Agile UAVs leverage event cameras and GPU-accelerated VIO for high-speed flight through clutter, achieving >50 m/s in racing scenarios [2]. Next-generation humanoids integrate multi-camera rigs, depth sensors, and large vision language models for manipulation in human environments [123]. Sim-to-real transfer via reinforcement learning and domain randomization is critical for these high-DoF platforms.

Recent quantitative benchmarks across representative air and legged platforms show that multirotors still dominate in specific power while humanoids excel in autonomy per payload. The AscTec Firefly peaks at $225\mathrm{W}{\mathrm{kg}}^{-1}$ and the DJI Matrice 600 at $155\mathrm{W}{\mathrm{kg}}^{-1}$, whereas the hydraulic Atlas delivers $93\mathrm{W}{\mathrm{kg}}^{-1}$ and the electrically driven Valkyrie $47\mathrm{W}{\mathrm{kg}}^{-1}$ [44,124,125,126]. Endurance paints the opposite picture: despite its miniature scale, the Crazyflie 2.1 sustains only 6 min of hover, the Firefly 12 min, and the M600 18 min, yet Atlas and Valkyrie can operate for roughly 50 and 60 min of level walking, respectively [125,126,127]. Normalizing the endurance by deliverable payload reveals an efficiency gap: the M600 achieves about $78\min {\mathrm{kg}}^{-1}$, whereas Atlas and Valkyrie exceed $1000\min {\mathrm{kg}}^{-1}$, reflecting the energetic advantages of periodic legged gaits when the payload mass dwarfs the locomotor output. Reliability, however, remains the chief obstacle for humanoids—Valkyrie records a mean time between falls of just 18 min on DARPA-style terrain—while mature UAV avionics enable mission-critical MTBF figures approaching 10 h on the M600 [124,126]. Collectively, these metrics underscore a trade space in which aerial robots offer an unmatched peak power density and maneuverability, whereas humanoids trade speed for superior payload economy and terrestrial reachability.

7. Benchmark Datasets, Simulators, and Evaluation Metrics

Benchmarking is central to progress in robot perception and navigation. This section surveys the most widely used datasets, simulators, and evaluation metrics across SLAM, visual odometry, autonomous navigation, and task-specific domains. By standardizing the experimental conditions and reporting protocols, these resources support reproducible comparisons and accelerate system development. Figure 9 provides a schematic overview of the topics covered.

Figure 9. A schematic overview of the topics covered in “Benchmark Datasets, Simulators, and Evaluation Metrics”.

7.1. SLAM and Odometry Benchmarks

• KITTI Odometry—Stereo/mono sequences with a LiDAR ground truth for driving scenarios; reports translational and rotational RMSEs over 100 m/800 m segments [41];
• EuRoC MAV—Indoor VICON-tracked stereo+IMU recordings for micro-aerial vehicles [44];
• TUM VI—A high-rate fisheye+IMU dataset emphasizing photometric calibration and rapid motion [45];
• 4Seasons—Multi-season, day/night visual–LiDAR sequences for robustness testing under appearance changes [12].

7.2. Navigation and VLN Test Suites

• CARLA—An open-source urban driving simulator with weather perturbations; metrics include the success, collision, and lane invasion rates [128].
• AI Habitat—Photorealistic indoor simulation for point-goal and visual language tasks; success weighted by path length (SPL) is the standard metric [93,129,130,131].

7.3. Domain-Specific Datasets

City-scale perception benchmarks (Cityscapes, nuScenes, Waymo) supply dense labels across multiple sensors for autonomous driving [117,132,133]. Meanwhile, underwater datasets (e.g., RUIE [134], Aqualoc [135], etc.) capture turbidity variations, while lunar/planetary analog datasets provide stereo and thermal imagery with the inertial ground truth.

7.4. Evaluation Metrics

Pose Estimation: Absolute trajectory error (ATE) and relative pose error (RPE) quantify drift; KITTI uses segment-level drift percentages. Mapping: The quality of surface reconstruction is measured via completeness and accuracy against dense laser scans. Navigation: Success/collision rates, the SPL, and dynamic obstacle violations dominate. Detection and Segmentation: The mean average precision (mAP) and panoptic quality (PQ) are used to evaluate 2D/3D perception. Industrial Manipulation: Pick success rates, cycle times, and force thresholds are used to benchmark grasp and assembly.

7.5. Open-Source Software and Reproducibility Resources

Widely adopted frameworks such as ORB-SLAM3, VINS-Mono, OpenVINS, and Kimera provide reference implementations with ROS interfaces, facilitating comparative studies. Public leaderboards (e.g., KITTI Vision, the nuScenes detection challenge) encourage transparent reporting and rapid iterations. Table 3 displays some of the metrics used in robot perception, mapping, navigation, and robotic manipulation.

Table 3. Frequently used metrics for evaluating robot perception, mapping, navigation, and manipulation.

Task	Metric	What It Measures
Pose estimation	ATE, RPE	Absolute and relative drift of estimated trajectory with respect to ground truth
Mapping	Completeness, Accuracy	Point-/surface-level recall and deviation versus dense laser scans
Navigation	Success rate, Collision rate, SPL	Goal reached, safety violations, efficiency normalized by path length
Detection and segm.	mAP, PQ	Precision–recall area (2D/3D) and unified panoptic quality
Manipulation	Pick success, cycle time, force $\Delta$	Grasp completion ratio, throughput, and applied force thresholds

8. Engineering Design Guidelines

Drawing on the cross-domain lessons above, we distil the practical recommendations for designing, integrating, and maintaining vision-based robotic systems. Figure 10 displays a schematic overview of the main aspects discussed in this section.

Figure 10. A schematic overview of the topics covered in “Engineering Design Guidelines”.

8.1. Sensor Selection and Placement

• Matching the modality to the environment: This favors passive stereo or LiDAR in outdoor sunlight; RGB-D or structured light indoors; and thermal cameras for night or smoke [6,35].
• Maximizing the complementary overlap: Arrange multi-modal rigs with overlapping fields of view to ease extrinsic calibration and redundancy. For 360° coverage, stagger cameras with a $\ge {20}^{\circ }$ overlap to maintain feature continuity [21];
• Minimize parallax in manipulation: Mount eye-in-hand cameras close to the end-effector to reduce hand-eye calibration errors while ensuring a sufficient baseline for stereo depth.

8.2. Calibration and Synchronization

Accurate extrinsic calibration underpins multi-sensor fusion. Spatiotemporal calibration boards or online photometric–geometric optimization (e.g., Kalibr) is used to jointly refine time delays and poses. Recalibration occurs after mechanical shocks or temperature excursions; thermal cameras require emissivity compensation [34].

8.3. Real-Time Computation and Hardware Acceleration

Latency-critical front-ends (feature extraction, CNN inference) are embedded onto GPUs or specialized accelerators; global bundle adjustment is offloaded to CPU threads or the cloud if the bandwidth allows. The worst-case compute loads are profiled and ≥50% headroom is maintained for unexpected dynamics. Event camera pipelines benefit from FPGAs or neuromorphic hardware to process asynchronous streams at μs scale [2].

8.4. Safety, Reliability, and Redundancy

Watchdogs that trigger a safe stop when the perception confidence drops are implemented based on the feature count, reprojection error, or network uncertainty. Multi-modal outputs (e.g., LiDAR range vs. stereo) are cross-checked to detect sensor faults. For collaborative robots, depth or IR gating enforces separation zones and complies with ISO 10218-1 and ISO 10218-2 [136,137].

8.5. System Integration, Testing, and Deployment

Modular architectures (ROS 2, DDS) with simulation-in-the-loop testing (CARLA, Habitat) are adopted before field deployment. Hardware-in-the-loop rigs with lighting and weather variations are used to expose edge cases early. Continuous integration pipelines should replay recorded bag files against regression tests and dataset benchmarks (KITTI, EuRoC). Exhaustive logs (sensor, pose, control) should be recorded for post-processing analysis and dataset curation.

9. Future Trends and Open Research Challenges

Robotic perception is rapidly evolving: generative models now supply physics-faithful synthetic data to reinforce training sets, neural implicit maps target city-scale 3D reconstructions, and emerging sensors provide denser or faster measurements for demanding applications. Simultaneously, multi-modal fusion, human-aware reasoning, and all-weather reliability have moved from exploratory studies to core design goals. These advances promise autonomous systems that learn from richer signals, adapt online, and cooperate naturally with people, yet they create pressing challenges—ensuring domain realism, scaling the representations, fusing heterogeneous sensors under adverse conditions, and embedding strong ethical and security safeguards—which the following subsections address. A schematic illustration of the subjects discussed in this section is provided in Figure 11.

Figure 11. Schematic overview of the topics covered in “Future Trends and Open Research Challenges”.

9.1. Generative AI for Synthetic Data and Scene Completion

Recent advances in deep generative models—especially diffusion models coupled with 3D scene representations such as neural radiance fields (NeRFs)—are transforming the data generation for robotics and autonomous driving. Systems like NeRFiller iteratively inpaint missing geometry in captured scans, producing multi-view-consistent and simulation-ready completions without user prompts [138]. Hierarchical latent diffusion models (e.g., XCube) extend this capability to high-resolution voxel grids, enabling both single-scan scene completion and text-to-3D generation [139]. In autonomous driving, hierarchical Generative Cellular Automata can synthetically extend realistic LiDAR beyond a sensor’s field of view, enriching the training corpora for long-range perception [140]. These techniques supply abundant, diverse data and explicitly infer occluded regions, narrowing the sim-to-real gap [141].

Open challenges include (i) ensuring domain realism—synthetic data must match real sensor physics to avoid negative transfer [141]; (ii) embedding physical plausibility (support, lighting, dynamics) to prevent the generation of unrealistic objects; (iii) achieving real-time inference through model distillation or efficient generative back-ends; and (iv) establishing standardized metrics for 3D generative quality. Addressing these issues will move generative pipelines from offline augmentation to on-board perception.

To minimize the sim-to-real gap, researchers pair generative pipelines with unsupervised domain adaptation—e.g., style transfer networks such as CycleGAN, feature-level alignment via Maximum Mean Discrepancy, or domain randomization curricula that expose action policies to wide visual variations [142,143,144]. These methods consistently cut the performance drop from synthetic to real data by 20–40% across detection and depth completion benchmarks.

9.2. Neural Implicit Representations and Dense 3D Mapping

Neural implicit maps promise unified geometry, appearance, and semantics at an arbitrary resolution. Grid-based encodings inspired by Instant-NGP, where NGP stands for Neural Graphics Primitives, accelerate optimization by orders of magnitude; Uni-SLAM uses decoupled hash grids and uncertainty-weighted updates to deliver real-time, dense reconstructions while capturing thin structures [145]. For city-scale scenarios, Block-NeRF partitions the environment into independent NeRF blocks that can be trained, streamed, and updated separately, achieving neighborhood-sized neural maps from millions of images [146].

Open challenges center on (i) scalability, representing ever-larger or dynamic worlds without prohibitive memory; (ii) fast incremental training so that maps can adapt at $>30$ Hz; (iii) probabilistic uncertainty estimation for safety-critical localization; and (iv) robust fusion of multi-modal data (RGB, depth, LiDAR) within a single implicit field.

9.3. High-Density 3D Sensing and Sensor Fusion

Emerging solid-state LiDARs, panoramic ToF, and wide-baseline stereo dramatically increase the point density while lowering cost [147]. Learning-based early fusion networks now exploit these rich signals: SupFusion leverages teacher-guided feature supervision to improve the LiDAR–camera detection accuracy [144], while weather-conditional radar–LiDAR fusion dynamically re-weights modalities to maintain the performance in rain, fog, and snow [148].

Research directions include efficient representations (e.g., bird’s-eye-view tensors) that scale to millions of points per frame, online calibration and temporal alignment of heterogeneous sensors, and graceful degradation under sensor failure.

9.4. Event-Based Vision for Ultra-Fast Control

Event cameras provide a $\mu$s latency and a high dynamic range, enabling kilohertz feedback for agile drones and manipulation. A hybrid event+frame detector recently demonstrated the motion-capture capability of a 5000 FPS camera at a bandwidth of 45 FPS video, validating the benefit of marrying asynchronous events with the conventional frames [149].

Key gaps remain in developing high-resolution event sensors, establishing standard benchmarks, and unifying frame–event SLAM pipelines that can localize and map at high speeds.

9.5. Human-Centric Autonomy and Humanoids

Human-centric systems increasingly embed large-language or multi-modal models to understand intent and follow natural instructions. In manufacturing, LLM-mediated “symbiotic” robots adjust their plans based on real-time human feedback [150], while pedestrian action and trajectory co-prediction boosts social compliance in driving [151]. Humanoid research is progressing in terms of whole-body manipulation and dynamic locomotion, with surveys highlighting advances in lightweight structures, reinforcement-learning-based control, and safety considerations [152].

Open problems include fluid, context-aware teamwork; safe physical interactions; energy-efficient actuation; and lifelong skill acquisition in unstructured homes.

9.6. All-Weather, All-Condition Operation

Robust autonomy demands perception that endures rain, fog, snow, night, and glare. Multi-modal suites now integrate imaging radar and thermal IR, with condition-aware fusion boosting the detection when vision degrades [148]. Benchmarks such as ACDC provide the adverse-weather ground truth yet remain limited in scale [153]. Simulation with physics-based weather effects and diffusion-generated augmentations further enlarges the rare-condition data [141].

The research must tackle adaptive sensor scheduling, self-supervised domain adaptation, and resilient SLAM that tolerates partial sensor failure, while expanding public datasets covering extreme conditions.

9.7. Ethical, Security, and Societal Considerations

Public acceptance hinges on privacy-preserving perceptions, secure update pipelines, and ethically aligned decision making. Empirical studies show that hybrid ethical algorithms—balancing utilitarian, deontological, and relational principles—are viewed as more acceptable for AV dilemma scenarios than single-rule strategies [154]. Dataset bias auditing and fairness enforcements remain critical to preventing unsafe behaviors, and governance frameworks must evolve alongside technical safeguards.

10. Conclusions

Vision has evolved from a supplemental sensor to a central pillar of robotic autonomy. This review has mapped the engineering landscape from the photon to the actuator, unifying the advances in the sensing hardware, depth inference, SLAM, navigation, and cross-domain deployment. We showed how a diverse sensor palette—of monocular RGB, stereo, RGB-D, LiDAR, event, and infrared—provides complementary trade-offs in range, latency, cost, and robustness. Modern SLAM systems integrate visual, inertial, and LiDAR cues with learned features and semantic understanding to achieve a centimeter-level accuracy over kilometer-scale trajectories, even in dynamic environments. The navigation strategies now span classical geometry-aware planning, deep reinforcement and imitation learning, topological visual memories, and visual language grounding, enabling robots to operate from factory floors to Martian plains.

Case studies in autonomous driving, industrial manipulation, underwater exploration, and planetary rovers have illustrated how the domain constraints shape the sensor–algorithm choices, while benchmark datasets and simulators facilitate reproducible evaluations. Our engineering guidelines distilled the best practices in the sensor placement, calibration, compute budgeting, safety monitoring, and system integration—critical for translating research prototypes into reliable products.

By analyzing 150 relevant studies in robotic navigation and computer vision, five consistent findings emerged that cut across sensors, algorithms, and application domains:

• Fusion is mandatory. Across the benchmarks, combining complementary sensors markedly improves the accuracy: RGB + LiDAR depth completion cuts the RMSE by about 30% on KITTI, and adding an IMU reduces the visual drift on EuRoC/TUM VI from 1 to 2% to below 0.5% of the trajectory length (see Section 3.5).
• Learning is the new front-end. Learned feature extractors (SuperPoint + LightGlue) and dense networks such as DROID-SLAM consistently outperform handcrafted pipelines, maintaining high recall and robust tracking under severe viewpoint, lighting, and dynamic scene changes (Section 4.2 and Section 4.9).
• Implicit mapping is now real-time. Hash-grid NeRF variants—e.g., Instant-NGP and NICE-SLAM—reach >15 Hz camera tracking with a centimeter-level depth accuracy on a single consumer GPU (Section 4.8).
• Topological representations slash memory costs. Systems such as NavTopo achieve an order-of-magnitude (∼10×) reduction in storage relative to that for dense metric grids while retaining centimeter-level repeatability for long routes (Section 5.4).
• Safety relies on redundancy and self-diagnosis. The guideline section (Section 8.4) shows that cross-checking LiDAR, stereo, and network uncertainty signals—coupled with watchdog triggers—detects most perception anomalies before they propagate to the controller, underscoring the importance of built-in fault detection.

While a narrative review is well-suited to providing a broad, integrative panorama of a fast-moving field, it inevitably trades methodological rigor for breadth. It is worth noting that the expert-guided search, although systematic within our team, does not guarantee exhaustive coverage and may introduce selection bias toward well-known venues. Future work could mitigate these issues by coupling our conceptual framework with a protocol-driven scoping or systematic review—following the PRISMA-ScR or Joanna Briggs guidelines. Looking forward, generative AI, neural implicit mapping, high-density 3D sensing, and event-based vision promise to blur the line between perception and cognition further, pushing autonomy toward all-weather, human-aware operation. Yet open challenges remain: maintaining a robust performance in extreme conditions, secure and ethical data handling, and scalable benchmarks for emerging humanoid and aerial platforms. Addressing these issues demands interdisciplinary collaboration across optics, machine learning, control, and system engineering—hallmarks. We hope that this review equips researchers and practitioners with a cohesive roadmap for advancing vision-based robotics from niche deployments to ubiquitous, trustworthy partners in everyday life.