Deep Reinforcement Learning for Navigation via Multi-Modal Belief State Representation from LiDAR and Depth Sensors

Abstract

This paper presents a deep reinforcement learning framework for autonomous navigation based on multi-modal belief state representation learned from LiDAR and depth sensors. To address the challenges posed by partial observability and sensor-specific uncertainty, we propose a probabilistic representation module that models belief states as Gaussian distributions over latent environmental features. Sensor-specific encoders extract structured features from raw LiDAR and depth inputs, which are fused using a Q-value-guided weighting scheme derived from the policy critic. A motion-prediction pretraining strategy and a cross-modal coherence loss are introduced to enhance the alignment and reliability of the learned belief states. The resulting representation is integrated into a Soft Actor–Critic (SAC) framework to enable policy-driven decision-making under uncertainty. Extensive experiments in simulated environments demonstrate that the proposed method improves success rate, navigation efficiency, and generalization. Real-world experiments further validate these findings, with the proposed method outperforming a classical navigation baseline by reducing average travel time by 16% and path length by 4%. These results support the use of probabilistic multi-modal belief modeling for autonomous navigation under partial observability.

1. Introduction

Autonomous navigation in complex environments remains a central challenge in the field of robotics. This complexity arises from the dynamic and unpredictable nature of real-world settings, where autonomous systems must interpret incomplete information, avoid obstacles, and adapt to constantly changing conditions without human intervention. Such capability is essential in a wide range of autonomous mobile robot applications, including warehouse logistics, service delivery, industrial inspection, agricultural monitoring, and search-and-rescue operations. In these scenarios, reliable navigation is a prerequisite for safe and efficient task execution. Traditional navigation methods often depend on predefined maps and limited sensor inputs, which lack the flexibility needed to manage such variability effectively [1].

The integration of robotics with artificial intelligence aims to equip robots with the capacity to perceive, understand, and interact with their environments in a manner similar to biological agents [2]. In this context, Deep Reinforcement Learning (DRL) has emerged as a powerful tool for enabling navigation systems to learn decision policies directly from interaction-based experience [3,4,5]. By continuously adjusting their policies based on environmental feedback, DRL-based agents can attain higher levels of autonomy and adaptability, especially in complex scenarios where explicit modeling or hand-crafted planning rules are insufficient.

Despite its potential, deploying DRL for autonomous navigation presents specific difficulties. Among the most prominent is partial observability, where the agent receives only fragmentary and noisy observations of the true environmental state at each time step. This issue is compounded by sensor uncertainty. For example, LiDAR provides geometrically reliable range measurements and is effective for obstacle boundary perception and free-space estimation, but it may suffer from angular sparsity, missing returns, or limited structural descriptiveness in cluttered scenes. Depth cameras provide denser local geometric information and richer spatial structure, but their observations are more sensitive to occlusion, surface properties, and sensing conditions. As a result, either sensing modality alone may yield incomplete or ambiguous environmental understanding [6]. These challenges degrade the agent’s ability to make informed decisions and can lead to suboptimal navigation behavior.

A promising solution to these limitations lies in the use of belief state representations, which model the agent’s knowledge of the environment as a distribution over latent variables. By estimating a belief over possible true states, an agent can make more informed decisions even when observations are noisy or incomplete. Inspired by this principle, we propose a deep learning-based approach for constructing multi-modal belief state representations from LiDAR and depth camera inputs. These belief states are modeled as Gaussian distributions that capture both the expected environmental state and the associated uncertainty, and are adaptively fused through Q-value-guided weighting within the Soft Actor–Critic (SAC) reinforcement learning framework.

To address these challenges, we formulate navigation under partial observability and learn a probabilistic belief representation from LiDAR and depth observations. Sensor-specific belief representations are regularized through motion supervision and cross-modal consistency learning, and are then adaptively fused according to their decision relevance before being used by the SAC agent for policy learning. The proposed method is evaluated in both simulated and real-world environments. The main contributions of this paper are as follows:

(1)

We propose a probabilistic multi-modal belief state representation model that estimates the latent environment state from LiDAR and depth sensors under partial observability.

(2)

We design a sensor-specific encoding and Q-value-guided fusion mechanism that dynamically integrates information from heterogeneous sensors based on learned action utility.

(3)

We validate the proposed framework through experiments in simulated and real-world environments, demonstrating improved navigation performance and learning efficiency compared to single-sensor and non-fused baselines.

2. Related Work

Autonomous navigation has traditionally relied on LiDAR-based Simultaneous Localization and Mapping (SLAM) [7,8], path planning [9], and obstacle avoidance techniques [10]. While these methods are effective in structured environments, they often struggle in dynamic or unknown settings due to their reliance on pre-built maps and static assumptions. Survey studies have highlighted the importance of robust perception, uncertainty handling, and sensor fusion in autonomous navigation systems, particularly when robots operate in cluttered and partially observable environments [11,12].

To overcome such limitations, Deep Reinforcement Learning (DRL) has been widely adopted for its ability to learn navigation policies through trial and error in unstructured environments. Representative DRL-based approaches initially relied primarily on LiDAR observations. For instance, Shi et al. [4] utilized the Asynchronous Advantage Actor–Critic (A3C) algorithm with intrinsic rewards for end-to-end navigation. To improve learning efficiency and sim-to-real robustness, Wang et al. [5] later introduced a heuristic dense reward shaping strategy for map-free navigation. In addition, a hierarchical framework combining DRL for high-level decision-making with fuzzy logic for low-level control was proposed in [13], improving navigation stability and physical feasibility in industrial scenarios. Nevertheless, these LiDAR-based methods still suffer in visually complex environments due to their reliance solely on distance measurements.

Visual-based DRL navigation has attempted to mitigate these issues. Mirowski et al. [14] introduced Nav A3C for dynamic 3D mazes using visual inputs and auxiliary tasks. Zhu et al. [15] proposed a target-driven DRL model using high-fidelity 3D scene simulations, and Zeng and Wang [16] developed an asynchronous PPO method for visual navigation stability. Devo et al. [17] improved generalization in target-driven visual navigation using a dual-network architecture. Nguyen et al. [18] further incorporated uncertainty-aware visual attention into navigation, showing that explicit uncertainty modeling can improve action selection in complex environments. These methods generally rely on RGB or depth-dominant visual perception with first-person observations as the target reference [1], differing from classical navigation tasks that use explicit goal-point destination coordinates provided by localization systems.

In environments with limited observability, modeling the environment as a Partially Observable Markov Decision Process (POMDP) offers a principled framework. Recurrent models such as LSTMs have been employed to manage temporal data and improve decision-making under uncertainty [19,20]. Choi et al. [21] introduced LSTM with Local-Map Critic (LSTM-LMC) for enhanced navigation, and Wang et al. [22] proposed Fast-RDPG for efficient learning in continuous action spaces. Francis et al. [23] combined Probabilistic Roadmaps (PRMs) with RL to enhance long-range navigation in POMDPs. In addition to recurrent memory, explicit belief-space reasoning has also been investigated. Elimelech and Indelman [24] showed that decision-making in belief space can be simplified while preserving solution quality, and Huang et al. [25] demonstrated the value of online belief prediction for efficient POMDP planning under uncertainty.

Multi-modal sensor fusion has also been explored to improve environmental understanding. Huang et al. [26] developed a DRL-based method using multi-modal perception, combining camera and LiDAR data. Nguyen et al. [27] introduced NMFNet for complex environments using laser, 2D images, and 3D point cloud data. Xie and Dames [28] presented DRL-VO, integrating LiDAR and camera data to use velocity obstacles for proactive collision avoidance and dynamic navigation. Related studies have further shown the importance of reliability-aware sensing and fusion. Park and Chung [29] demonstrated that explicitly modeling LiDAR uncertainty improves localization robustness for outdoor mobile robots, while Tang et al. [12] provided a comparative review of deep learning-based multi-modal sensor fusion and discussed the strengths, limitations, and open challenges of fusion across heterogeneous sensors. These findings suggest that multi-modal perception is beneficial, but deterministic fusion alone may be insufficient when sensor quality varies over time.

Although the above approaches leverage multi-modal inputs or uncertainty-aware reasoning, they generally do not integrate probabilistic latent belief modeling and task-relevant adaptive fusion within a unified DRL navigation framework. In contrast, our method introduces an explicit probabilistic belief state representation, where LiDAR and depth inputs are encoded as Gaussian-distributed latent states and fused adaptively using Q-value-guided weighting. This structure enables the policy to reason over uncertain, multi-sensor inputs and improves navigation performance under partial observability.

3. Problem Formulation

Autonomous navigation in real-world environments can be formulated as a sequential decision-making problem in which a mobile robot must reach a target while avoiding obstacles based on incomplete and noisy sensory observations. In the considered setting, the robot is equipped with LiDAR and a depth camera, together with auxiliary motion-related variables provided by the onboard system. While some quantities, such as goal-related information and robot motion states, are directly available, environmental features remain only partially observable because of occlusion, limited sensing range, viewpoint dependence, and measurement noise. As a result, the robot cannot directly access the full state required for reliable navigation decisions.

This property makes a fully observable Markov Decision Process (MDP) formulation inadequate for the present task. In an MDP, the current state is assumed to be sufficient for optimal decision-making, whereas in autonomous navigation the current observation often captures only a fragment of the surrounding environment. Treating this observation as the true state may therefore lead to unstable or suboptimal behavior. Recurrent policies can partially alleviate this issue by summarizing observation history, but they do not explicitly characterize uncertainty in the latent environment state. For this reason, we formulate the navigation problem as a Partially Observable Markov Decision Process (POMDP), which provides a principled description of decision-making under incomplete and uncertain observations [30].

A POMDP extends the classical Markov Decision Process (MDP) to accommodate scenarios with partial state observability. It is formally defined as a 7-tuple $(S,A,T,O,Z,R,\gamma )$, where:

• S is the state space, representing all possible environmental factors relevant to navigation tasks.
• A denotes the action space, including discrete or continuous actions the robot can take, such as moving forward, turning, or adjusting speed.
• $T:S\times A\times S\to [0,1]$ represents the state transition probability function. $T\left(s^{\prime }\right|s,a)$ defines the probability of transitioning from state s to state $s^{\prime }$ after executing action a.
• O is the observation space, generated from the data acquired by the robot’s laser sensor and depth camera, providing partial information about the state S.
• $Z:S\times A\times O\to [0,1]$ is the observation function. $Z\left(o\right|s^{\prime },a)$ indicates the likelihood of perceiving observation o after taking action a and arriving at state $s^{\prime }$.
• $R:S\times A\to \mathbb{R}$ is the reward function. It assigns a scalar value based on the robot’s action and its effectiveness in navigating the environment.
• $\gamma$ is the discount factor, a value between 0 and 1, which determines the importance of future rewards compared to immediate ones.

In the context of this work, S includes both directly available robot-related variables and latent environmental factors that must be inferred from sensor observations. The observation space O consists of LiDAR scans, depth images, and auxiliary navigation-related variables, while the action space A corresponds to the robot’s motion commands. Under this formulation, optimal action selection depends not only on the current observation but also on the robot’s belief about the hidden environment state.

The objective in a POMDP is to learn a policy $\pi$ that selects actions based on observed inputs in order to maximize the expected sum of discounted future rewards. Because the true state is not directly observable, the agent maintains a belief state $b_t$, namely a probability distribution over possible states conditioned on the observation and action history.

For the present task, belief-state modeling offers two advantages. First, it allows complementary information from different sensing modalities to be integrated into a unified latent representation. Second, it provides a natural way to encode uncertainty caused by noisy measurements, occlusion, and limited field of view. However, exact belief inference is generally intractable in high-dimensional navigation problems. Instead of performing analytic belief updates, this work learns a compact probabilistic approximation of the belief state from multi-modal sensor data.

Our central challenge lies in constructing a neural network architecture that can effectively learn a representation of the belief state $b_t$ from the multimodal sensor data. This representation must capture sufficient information about the latent environmental state to support effective policy learning, while remaining resilient to the noise and partial information inherent in real-world observations.

4. Methodology

4.1. Overview of the Framework

Based on the POMDP formulation, the proposed method learns a compact probabilistic approximation of the belief state from multi-modal observations and uses it for navigation policy learning. The framework integrates a multi-modal belief state representation module with a Soft Actor–Critic (SAC) agent to enable autonomous navigation under partial observability. Specifically, LiDAR and depth observations are processed to generate latent belief representations that encode both environmental structure and associated uncertainty. These representations are then fused through a Q-value-guided weighting and provided to the SAC policy and critic networks for action selection and value estimation.

The overall methodology can be summarized in four stages. First, the navigation state is decomposed into directly observable robot-related variables and latent environmental features. Second, modality-specific encoders transform LiDAR and depth observations into probabilistic belief representations. Third, these belief representations are regularized through motion-prediction supervision and cross-modal consistency learning, and are fused according to their task relevance. Finally, the fused belief representation is used by the SAC agent for policy learning under partial observability.

As illustrated in Figure 1, the architecture consists of three main components: (1) separate sensor-specific encoders that convert raw sensor input and partial state information into belief states, (2) a belief fusion mechanism that combines these representations based on Q-value weights, and (3) a decision-making network trained using SAC.

In accordance with the problem formulation, the state relevant to navigation contains both directly available robot-related variables and latent environmental factors. The directly available variables include goal information, velocity, and safety-related quantities, whereas the surrounding environmental structure must be inferred from sensor observations. This distinction motivates the use of sensor-specific belief encoders, which estimate the latent environmental state while retaining the reliable variables as direct inputs to the decision process.

Fully observable state elements:

• Goal Information: Goal Distance (the distance from the robot to the navigation goal) and Goal Heading (the direction or angle to the navigation goal relative to the robot’s current orientation).
• Velocity: Linear Velocity (the speed at which the robot is moving forward or backward) and Angular Velocity (the rate at which the robot is rotating).
• Safety Information: Nearest Obstacle Distance (the distance to the nearest obstacle) and Nearest Obstacle Orientation (the direction or angle to the nearest obstacle relative to the robot’s current orientation).

Partially observable elements:

• LiDAR Observation: distance measurements to surrounding obstacles provided by the LiDAR sensor.
• Depth Camera Observation: depth information from the camera providing a local 3D view of the surroundings.

4.2. Representation Learning for Belief Networks

The belief networks estimate the partially observable aspects of the true state (environmental features) while incorporating the fully observable and reliable elements directly. To obtain a tractable probabilistic approximation of the latent state, the belief state $b_t\left(s\right)$ is modeled as a multivariate Gaussian distribution:

$$b_t\left(s\right)\sim \mathcal{N}({\mu }_t,{\sigma }_t^2),$$

(1)

where ${\mu }_t$ and ${\sigma }_t^2$ represent the estimated mean and uncertainty of the latent state, respectively. The Gaussian assumption is adopted here as a tractable probabilistic approximation rather than an exact claim about the true state distribution. Its appeal lies in the convenient mathematical structure of Gaussian densities: they are closed under linear transformations, and their products preserve a Gaussian form up to normalization. This makes them well suited for representing uncertainty and for combining information from multiple sensing modalities. Moreover, because classical analytic Bayesian updates are often developed under Gaussian assumptions, Gaussian latent beliefs provide a natural and interpretable approximation in the proposed framework.

The belief networks include separate encoders for LiDAR and depth camera data, each structured according to the characteristics of the corresponding modality, as illustrated in Figure 2. The LiDAR encoder processes 180-degree LiDAR data using a series of fully connected layers. These layers extract features related to distance and obstacle geometry, which are then concatenated with partial state information including goal location and obstacle data. The output is a Gaussian belief representation parameterized by the mean ${\mu }_{\mathrm{LiDAR}}$ and variance ${\sigma }_{\mathrm{LiDAR}}^2$:

$$\begin{array}{cc}\hfill [{\mu }_{\mathrm{LiDAR}},{\sigma }_{\mathrm{LiDAR}}^2]& =f_{\mathrm{LiDAR}}(x_t^{\mathrm{LiDAR}},s_t^{\mathrm{observable}}),\hfill \\ \hfill b_t^{\mathrm{LiDAR}}\left(s\right)& \sim \mathcal{N}({\mu }_{\mathrm{LiDAR}},{\sigma }_{\mathrm{LiDAR}}^2).\hfill \end{array}$$

(2)

where $f_{\mathrm{LiDAR}}$ represents the LiDAR encoder neural network, $x_t^{\mathrm{LiDAR}}$ denotes the LiDAR sensor data, and $s_t^{\mathrm{observable}}$ includes fully observable state elements.

In parallel, the depth camera encoder utilizes convolutional layers to process depth images, extracting textural and structural features. Similarly to the LiDAR encoder, the extracted features are concatenated with the same partial state information and output the belief state, forming a complementary belief state that captures different aspects of the environment:

$$\begin{array}{cc}\hfill [{\mu }_{\mathrm{depth}},{\sigma }_{\mathrm{depth}}^2]& =f_{\mathrm{depth}}(x_t^{\mathrm{depth}},s_t^{\mathrm{observable}}),\hfill \\ \hfill b_t^{\mathrm{depth}}\left(s\right)& \sim \mathcal{N}({\mu }_{\mathrm{depth}},{\sigma }_{\mathrm{depth}}^2).\hfill \end{array}$$

(3)

where $f_{\mathrm{depth}}$ represents the depth camera encoder neural network, $x_t^{\mathrm{depth}}$ denotes the depth camera data, and $s_t^{\mathrm{observable}}$ includes fully observable state elements.

To guide the training of the belief networks, we employ a self-supervised motion-prediction task. A motion-prediction network estimates the relative pose change ${\widehat{\Delta }}_{\mathrm{pose}}$ between successive belief states:

$${\widehat{\Delta }}_{\mathrm{pose}}=m_{\psi }(b_t\left(s\right),b_{t+1}\left(s\right)),$$

(4)

optimized via a mean squared error loss:

$${\mathcal{L}}_{\mathrm{pose}}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\parallel {\widehat{\Delta }}_{\mathrm{pose}}-{\Delta }_{\mathrm{pose}}\parallel }^2,$$

(5)

where $m_{\psi }$ represents the motion-prediction network parameterized by $\psi$, and the ground-truth pose change ${\Delta }_{\mathrm{pose}}$ is obtained from simulation or high-precision positioning systems.

To promote alignment between modalities, a coherence loss is added during training. In practice, this consistency constraint is applied to the latent belief representations, encouraging the LiDAR- and depth-derived beliefs to remain compatible under the same scene configuration. The loss is written as

$${\mathcal{L}}_{\mathrm{coherence}}={\parallel b_t^{\mathrm{LiDAR}}-b_t^{\mathrm{depth}}\parallel }^2.$$

(6)

During inference, the modality-specific belief representations are fused using Q-value-guided weights derived from the SAC critic:

$$u_i=Q_i+c,w_i={\displaystyle \frac{u_i}{{\sum }_j{u}_j}},b_t\left(s\right)=\sum _i{w}_i·b_t^i\left(s\right),$$

(7)

where $Q_i$ denotes the critic-evaluated action value associated with the i-th modality-specific belief, and c is a fixed positive offset introduced to ensure non-negative fusion utilities. In this work, c is set to 100 according to the observed range of critic outputs during training. This affine normalization preserves the relative ordering of modality-specific action values, avoids sign ambiguity when critic outputs become negative, and yields smooth fusion weights while retaining an interpretable linear relationship between Q-value and weight. In this sense, the critic outputs are used as task-dependent utility estimates for fusion rather than as direct confidence scores of the sensing modalities. During training, we apply random sensor sampling to avoid modality overfitting.

4.3. Reinforcement Learning for Decision Network

Soft Actor–Critic (SAC) [31] is a model-free reinforcement learning algorithm that combines off-policy learning with entropy regularization to promote exploration and improve robustness. SAC operates within an actor–critic framework, using two Q-networks to mitigate overestimation bias. The integration of MM-BSR and SAC enables the robot to make informed decisions based on a probabilistic representation of its environment.

For each modality-specific belief representation, the critic estimates the utility of the action proposed by the current policy. These utility estimates are not interpreted as direct sensor confidence scores; instead, they measure how useful each modality-specific belief is for decision-making under the current policy. The resulting Q-value-guided weights are then used to form the fused belief representation. In this way, the fusion process emphasizes the modality that is more informative for the current control objective, rather than relying on fixed fusion coefficients or heuristic sensor priorities.

The SAC algorithm receives the fused belief state $b_t\left(s\right)$ as the input state representation, which includes reliable components (goal information, velocity, and safety information) together with the inferred environmental features from LiDAR and depth observations.

SAC maintains separate actor (policy) and critic (Q-function) networks, as shown in Figure 3. The actor predicts the action $a_t$ based on the fused belief state:

$$a_t={\pi }_{\theta }\left(b_t\left(s\right)\right),$$

(8)

where ${\pi }_{\theta }$ represents the actor network parameterized by $\theta$. The Actor network comprises fully connected layers and is trained to maximize the expected reward, as evaluated by the Critic network.

The Critic network evaluates the potential reward of the actions taken by the Actor network based on the current belief state. The Critic network also comprises fully connected layers and estimates the value of state-action pairs, aiding in policy evaluation and improvement:

$$q_t=Q_{\varphi }(b_t\left(s\right),a_t)$$

(9)

where $Q_{\varphi }$ represents the critic network parameterized by $\varphi$.

SAC incorporates an entropy term in the reward, which encourages exploration by promoting policies that are not deterministic. The objective is to maximize both the expected reward and the entropy of the policy, encouraging exploration:

$$J\left(\pi \right)=\sum _t{\mathbb{E}}_{(b_t,a_t)\sim {\rho }_{\pi }}\left[r_t+\alpha \mathcal{H}\left({\pi }_{\theta }(·\mid b_t\left(s\right))\right)\right],$$

(10)

where $\mathcal{H}\left({\pi }_{\theta }(·|b_t\left(s\right))\right)$ is the entropy of the policy and the temperature parameter $\alpha$ represents the weight given to the entropy term relative to the reward, facilitating a balance between adhering to known rewarding behaviors and exploring new actions.

The policy ${\pi }_{\theta }(·\mid b_t\left(s\right))$ is updated to maximize the expected reward and entropy, while the critic is trained by minimizing the temporal-difference error:

$${\mathcal{L}}_{\pi }={\mathbb{E}}_{b_t\sim D}\left[{\mathbb{E}}_{a_t\sim {\pi }_{\theta }}\left[\alpha log{\pi }_{\theta }(a_t\mid b_t\left(s\right))-Q_{\varphi }(b_t\left(s\right),a_t)\right]\right],$$

(11)

$${\mathcal{L}}_Q={\mathbb{E}}_{(b_t,a_t,r_t,b_{t+1})\sim D}\left[{\displaystyle \frac{1}{2}}{\left(Q_{\varphi }(b_t\left(s\right),a_t)-y_t\right)}^2\right],$$

(12)

where the target value is

$$y_t=r_t+\gamma {\mathbb{E}}_{a_{t+1}\sim {\pi }_{\theta }}\left[\underset{i=1,2}{min}{Q}_{{\varphi }_i^{\prime }}(b_{t+1}\left(s\right),a_{t+1})-\alpha log{\pi }_{\theta }(a_{t+1}\mid b_{t+1}\left(s\right))\right].$$

(13)

The reward function is designed to guide the agent toward goal-directed behavior while penalizing collisions and inefficient movements:

$$r_t=\left\{\begin{array}{cc}+100\hfill & \mathrm{if}\mathrm{the}\mathrm{task}\mathrm{is}\mathrm{successfully}\mathrm{completed},\hfill \\ -100\hfill & \mathrm{if}\mathrm{a}\mathrm{collision}\mathrm{occurs},\hfill \\ C_r(d_{t-1}-d_t)\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(14)

where $d_t$ denotes the Euclidean distance from the robot to the goal at time t, and $C_r$ is a scaling factor that controls the magnitude of the step-wise reward. This formulation encourages the agent to make continuous progress toward the goal while providing clear feedback for task completion and failure.

To ensure belief-state reliability during joint optimization, the coherence term defined in Equation (6) is retained throughout training. Together with the motion-prediction objective, it regularizes the learned belief representations while the SAC losses optimize the decision policy. As a result, the overall training objective jointly couples representation learning and policy learning within a unified optimization process. The full model, including the belief networks, motion-prediction network, and SAC components, is optimized jointly using gradient descent with the Adam optimizer.

5. Experimental Evaluation

5.1. Experimental Platform and Training Configuration

The training and evaluation were carried out on a composite mobile robot platform, shown in Figure 4a, equipped with a SLAMTEC S1 LiDAR scanner, a Kinect depth camera, wheel odometry, and an onboard edge controller for real-time execution. The DRL framework was implemented using Python 3.7 with PyTorch 1.8.2 and trained on an NVIDIA RTX 3090 GPU under Ubuntu 20.04. ROS Noetic was used as the communication middleware for sensor acquisition, command publishing, and module integration, while Gazebo was employed to construct the simulation environments used for policy training and controlled evaluation.

Within this software stack, LiDAR scans and depth images were synchronized and forwarded to the proposed belief representation module, which produced the fused latent belief state used by the SAC policy and critic networks. The actor output was converted into ROS velocity commands and applied to the robot through the standard motion control interface. This setup allowed the same perception–decision pipeline to be used in both simulation and real-world deployment, with only the sensor source and execution environment changing between training and physical testing.

The training environment (Env1) is a $10\times 10$ m simulated space configured with static obstacles, shown in Figure 4b. The robot receives simulated odometry and pose updates for fully observable state components, while LiDAR and depth sensors provide partial observations. The control commands generated by the trained model are published as ROS velocity commands. The robot’s maximum linear and angular velocities are constrained to 0.75 m/s and 0.5 rad/s, respectively, to reflect real-world motion limits.

Unless otherwise stated, all ablation variants and comparison models were evaluated under the same simulation settings, robot motion limits, and episode termination conditions. This ensures that performance differences can be attributed primarily to the representation and fusion strategy rather than to changes in the execution platform or control interface.

Hyperparameters used for training are summarized in

Table 1. Hyperparameters for Training.

Parameter	Value
Max Episodes	2000
Max Steps per Episode	500
Discount Factor $\gamma$	0.99
Replay Buffer Size	${10}^5$
Actor Learning Rate	$3\times {10}^{-4}$
Critic Learning Rate	$3\times {10}^{-4}$
Belief Network Learning Rate	$1\times {10}^{-5}$
Batch Size	64
Max Linear Velocity	0.75 m/s
Max Angular Velocity	0.5 rad/s
Min Linear Velocity	0.05 m/s
Reward scaling factor $C_r$	0.2

. During training, the robot begins from the center of the scene, and goal points are randomly assigned within the environment. Each episode terminates upon goal completion, collision, or exceeding a step limit. In the case of success, the robot remains stationary while a new goal is assigned; otherwise, it is repositioned to the initial state.

5.2. Training Performance and Outcomes

To evaluate the contribution of each component of the belief representation framework, we compare the full system with several ablated variants:

• MM-BSR (Full): The complete system using both LiDAR and depth sensors, with Q-weighted fusion and pre-training.
• MM-BSR-L: LiDAR-only version of the belief model (no depth input).
• MM-BSR-D: Depth-only version of the belief model (no LiDAR input).
• MM-BS-NP: The full system trained without motion prediction pre-training.
• MM-BS-NC: The full system trained without coherence loss between sensor modalities.

All variants share the same network architecture, reward function, and training conditions to ensure fairness. Each configuration is trained over 2000 episodes with three independent runs for statistical validity.

The training performance of the proposed multi-modal belief state representation (MM-BSR) framework, along with its ablation variants, was evaluated using metrics including cumulative rewards, Q-values, and loss convergence. These results provide insights into the stability, sample efficiency, and representational power of each configuration. Figure 5 shows the learning curves of cumulative rewards and Q-values during training. Solid lines indicate average values across three runs, while shaded bands denote standard deviations. All curves are smoothed using a moving average over 40 episodes.

From the training results, it is evident that the depth-only model (MM-BSR-D) achieves the lowest training efficiency. This outcome reflects the greater complexity and computational overhead of processing visual inputs, particularly depth images, which require deeper convolutional architectures to extract spatial features. In contrast, the LiDAR-only model (MM-BSR-L) exhibits faster convergence and higher stability due to the structured, low-dimensional nature of LiDAR inputs. However, this unimodal approach may lack critical spatial context in ambiguous scenes. The full multi-modal model (MM-BSR Full) ultimately matches and exceeds the unimodal baselines, confirming that fusing complementary features from LiDAR and depth sensors improves the agent’s understanding of its environment.

The ablation variant without pre-training (MM-BS-NP) shows noticeably slower learning and reduced final reward, underscoring the importance of motion-supervised belief network initialization. Pre-training enables the encoders to form an approximate belief over the latent environment state before policy learning begins, accelerating convergence and stabilizing training. The coherence loss ablation (MM-BS-NC) performs the worst across all variants, with the lowest average reward and unstable Q-value trends. This highlights the role of the coherence loss in maintaining feature consistency across sensor modalities. Without this alignment, the fused belief state becomes less reliable, degrading decision quality.

These results demonstrate that the proposed MM-BSR framework, when combined with self-supervised pre-training and cross-modal alignment, provides a robust and efficient foundation for autonomous navigation in partially observable environments. The full model benefits from sensor redundancy, smoother learning, and enhanced performance compared to its ablated counterparts.

5.3. Navigation Performance Assessment

5.3.1. Comparison in Simulated Environments

To comprehensively evaluate the navigation performance and generalization capabilities of the proposed MM-BSR framework, we conducted experiments in three simulated environments of increasing complexity: Env2, Env3, and Env4 (Figure 6). Env2, being smaller with tighter corridors, required higher obstacle avoidance capabilities, while Env3, being larger and more complex, tested the pathfinding abilities of the models without frequent collisions. Env4, simulating a common library environment, differs significantly from training environment (Env1) in style and layout, with higher obstacle density increasing navigation difficulty.

We compared MM-BSR with a set of baseline models: MM-BSR-L (Uses only LiDAR input), MM-BSR-D (Uses only depth camera input), iTD3-CLN [32] (Improved TD3 with CNN-based LiDAR encoder), SAC-SP [33] (LiDAR-based SAC with auxiliary spatial planning rewards), BND*-DDQN [34] (Dual-stream DDQN using depth and its derivatives), and MoveBase (A classical global-planning-based navigation system). The models were evaluated using three metrics: success rate (percentage of trials where the goal was reached), average traveled distance (in meters), and average time to reach the goal (in seconds). Ten goals were evaluated per environment over 10 independent trials (100 runs per environment).

From the results in Env2 and Env3 shown in

Table 2. Performance in Env2 and Env3.

	Method	Success Rate	Avg. Dist.	Avg. Time (s)
Env2	MM-BSR	91%	6.87 m	13.23
	MM-BSR-L	76%	8.12 m	19.59
	MM-BSR-D	61%	8.44 m	15.97
	MoveBase	100%	5.75 m	17.40
Env3	MM-BSR	96%	5.78 m	11.74 s
	MM-BSR-L	93%	6.08 m	10.38 s
	MM-BSR-D	82%	6.15 m	22.41 s
	MoveBase	97%	5.71 m	24.29 s

, MM-BSR shows superior performance across all metrics compared to its ablated variants. In Env2, MM-BSR achieved a success rate of 91%, compared to 76% for MM-BSR-L and 61% for MM-BSR-D. This demonstrates MM-BSR’s higher efficiency in path planning and obstacle avoidance, benefiting from the integration of high-resolution close perception information provided by the depth camera, which aids in local obstacle avoidance. In Env3, the performance gap persists, with MM-BSR achieving a 96% success rate, while MM-BSR-L and MM-BSR-D achieved 93% and 82% respectively. The results highlight the deficiency of using only depth camera data (MM-BSR-D), which lacks long-distance geometric information, leading to inefficient path planning and overly conservative control commands. Moreover, MM-BSR’s consistent performance in both Env2 and Env3 underscores its generalization capabilities.

Comparison of MM-BSR and MoveBase reveals that while MoveBase achieves optimal navigation distances by relying on global planning and pre-built maps, MM-BSR attains competitive performance without map data. In Env2, MoveBase achieves a near-perfect success rate with the shortest average distance of 5.75 m, but the time taken (17.40 s) is longer compared to MM-BSR (13.23 s). In Env3, MoveBase achieves a 97% success rate with an average distance of 5.71 m and a time of 24.29 s, while MM-BSR achieves a 96% success rate with an average distance of 5.78 m and a time of 11.74 s. These results highlight MM-BSR’s higher navigation efficiency, making it a viable alternative in scenarios where building accurate maps is challenging or infeasible.

The performance assessment in both Env2 and Env3 underscores the strengths of MM-BSR in terms of success rate, navigation efficiency, and adaptability to different environmental complexities. Its superior performance compared to MM-BSR-L, and MM-BSR-D, along with competitive results against MoveBase, validates the effectiveness of the proposed framework.

In addition, we compared MM-BSR against three related DRL-based navigation algorithms in Env4. All methods were reimplemented and trained in Env1 using respective training parameters specified by their authors. In Env4, MM-BSR again led across all metrics as summarized in

Table 3. Testing Results in Env4.

Method	Success Rate	Avg. Dist.	Avg. Time
MM-BSR	91%	7.41 m	12.99 s
iTD3-CLN	76%	8.53 m	21.78 s
SAC-SP	72%	9.48 m	18.48 s
BND*-DDQN	72%	16.41 m	72.40 s

. Other DRL-based methods, while effective in their training domains, failed to generalize as well to new layouts and complex partial observability. MM-BSR achieved significantly higher success rates and shorter completion times than all baselines. Its ability to abstract latent states from multi-modal data contributed to more informed decision-making and generalization in unfamiliar layouts.

The iTD3-CLN and SAC-SP show lower task completion rates, indicating difficulties in handling partial observability. The BND*-DDQN algorithm struggles with efficient route planning due to the lack of range sensing, resulting in the longest average distance. In order to more intuitively compare the navigation performance of each algorithm, the navigation trajectories of each method in Env4 are shown in Figure 7. MM-BSR exhibits smooth and optimal paths, while the other algorithms show less efficient and more unstable trajectories.

5.3.2. Validation in Real-World Environments

To validate the effectiveness of the proposed MM-BSR framework, we deployed the trained model on a physical mobile robot in a controlled indoor environment (Figure 8a). The testing scenario consisted of four sequentially assigned target points surrounded by box-shaped obstacles with varying textures to simulate realistic navigation conditions.

To ensure safety and stability, the robot’s maximum linear and angular velocities were limited to 0.325 m/s and 0.25 rad/s, respectively. For comparison, we evaluated MM-BSR against the ROS navigation stack (MoveBase), a widely used classical approach based on global planning and known maps.

Each system was tested for five complete navigation sequences. The same metrics used in simulation were applied: average path distance, travel time, and variation across runs.

Table 4. Real-World Testing Results.

Method	Avg. Dist.	Min. Dist.	Max Dist.	Avg. Time	Min Time	Max Time
MM-BSR	11.77 m	11.30 m	12.46 m	54.68 s	49.26 s	58.12 s
MoveBase	12.28 m	11.59 m	12.77 m	65.11 s	64.02 s	67.13 s

summarizes the real-world navigation performance. MM-BSR consistently outperformed MoveBase in terms of average distance and navigation time. Although MoveBase demonstrated more consistent control, MM-BSR generated shorter, more efficient paths. However, the stability of MM-BSR, indicated by the difference between the maximum and minimum times, requires improvement. Figure 8b,c illustrate sample trajectories for MM-BSR and MoveBase, respectively. As seen in the figures, MM-BSR plans a shorter path but exhibits greater variability in the generated control commands, which slightly affects path smoothness.

Overall, the experiments demonstrate that the MM-BSR framework transfers effectively from simulation to real-world environments, providing an end-to-end approach to autonomous navigation for practical deployment.

5.4. Discussion and Limitations

The experimental results validate the effectiveness of the proposed MM-BSR framework for autonomous navigation in partially observable environments. In both simulation and real-world experiments, the proposed method achieves more efficient and reliable navigation behavior than the comparison settings, indicating that the combination of probabilistic belief modeling, multi-modal fusion, and reinforcement learning is beneficial for decision-making under incomplete observations. The results also suggest that LiDAR and depth sensing provide complementary information, and that learning a compact belief representation helps the policy make more informed navigation decisions.

Despite these strengths, several limitations remain. First, the generalization of the learned belief model to real-world dynamics is still constrained by the discrepancy between simulated and physical sensors, including differences in noise characteristics, sparsity, and calibration. Second, control stability during real-world navigation still requires improvement. Although MM-BSR produces efficient paths, it occasionally exhibits variable execution timing and slight instability during turning maneuvers. Third, broader generalization to more diverse object types, scene layouts, and sensing conditions will require richer training environments and possibly domain adaptation techniques. In addition, extending the current framework to more sensing modalities or larger-scale environments may increase computational cost and system complexity. These limitations point to future work on sim-to-real robustness, control smoothness, and scalability.

6. Conclusions

This work presented a framework for autonomous navigation in partially observable environments, centered on a Multi-Modal Belief State Representation (MM-BSR) module integrated with a Soft Actor–Critic (SAC) policy. The framework models latent environmental states as probabilistic belief representations derived from LiDAR and depth observations, and combines them through adaptive fusion for policy learning.

Experiments in simulated environments and real-world deployments indicate that the proposed framework provides a feasible way to improve navigation performance under partial observability. Compared with unimodal and existing DRL-based methods, MM-BSR achieves better success rate, trajectory efficiency, and execution time in the evaluated settings. These results support the use of probabilistic belief modeling and multi-modal fusion for navigation decision-making, while motion-prediction pretraining and coherence regularization contribute to more stable latent representations.

Several limitations remain, particularly in real-world control smoothness and generalization to unseen environments. Future work may therefore focus on improving sim-to-real robustness, enriching training environments, and exploring domain adaptation techniques to further support autonomous navigation in real-world robotic systems.