ROSNavBench: An Automated Benchmarking Framework for Systematic Planner–Controller Selection in Mobile Robotics

Abstract

Autonomous mobile robots need reliable path planning and local execution. Plenty of tools exist to evaluate either global planners or controllers. The open-source automated frameworks built for Robot Operating System 2 (ROS 2) do not measure computational cost alongside navigational reliability or report the statistics for multi-criteria decision making. The choice falls back on manual comparison, and that becomes harder as the environment changes. ROSNavBench is an open-source framework that automates this evaluation for global planners and controllers in ROS 2. It scores efficiency, reliability, and adaptability over a range of scenarios, and thus a user can pick the planner and controller that fit the job. Runs are automated and repeated, which tightens the statistics and takes the manual work out of the loop. We tested the framework across simulations and hardware. Navigation times in the two domains correlate at $r=0.97$, but a gap remains, and its width shifts from one trial to the next.

1. Introduction

Autonomous mobile robots have to navigate reliably, and the space is not always laid out to help them (e.g., warehouses, hospitals, office buildings). A robot traveling from its starting pose to a target goal has to avoid obstacles that may be dynamic, respect kinematic limits, and not waste time in the process. Designing and choosing the algorithms that manage this remains an open problem [1].

Trajectory tracking and path following are what let a robot cope with complex and sometimes hazardous surroundings [2,3]. These approaches determine how short the path is, whether the robot hits anything, how smoothly it moves, and how accurately it stops on the goal. A navigation stack is only as dependable as they are. Nav2 has become the standard navigation stack for mobile ground robots in ROS 2 [4,5]. It is assembled from plugins arranged by behavior trees, and thus global planners, controllers, and recovery behaviors can be swapped without disturbing anything else [6]. One stack therefore serves different platforms. Nav2 includes global planners and controllers, with several options on each side (Figure 1).

Figure 1. Plugin selection in a modular navigation stack. With this many global planners and controllers to draw from, the configuration space is large enough that the combination best suited to a given application is not obvious.

A global planner works over a cost map of the environment and returns an optimal or close-to-optimal path from the initial pose to the target goal [7]. Nav2 relies mainly on search-based methods, which replan faster than sampling-based ones [6]. Nav2 planners range from NavFn and Lazy Theta*-P [8] to the Smac Planner framework [9], whose Hybrid-A*, 2D-A*, and State Lattice variants suit different vehicle kinematics.

The controller turns the path into velocity commands, working around the robot’s dynamics, distance to the obstacles, and the safety margin [6]. Nav2 supports several controllers: DWB, refactored from the Dynamic Window Approach (DWA) [6,10]; Regulated Pure Pursuit (RPP) [11]; Timed Elastic Band (TEB) [12]; Graceful [13]; Model Predictive Path Integral (MPPI) [14]; and the Rotation Shim Controller (RSC). These controllers cover predictive, reactive, and geometric strategies.

As previously stated, the standard Nav2 ecosystem already provides an abundant library of plugins, which is further expanded by community-contributed controllers and global planners. The standard suite alone therefore admits more than 30 planner–controller pairings (Figure 1), and the choice among them, together with their configuration parameters, determines how well the robot behaves. Selection needs to move from qualitative recommendation to measurement so that the choice reflects the user’s own priorities and operating conditions. Moreover, each plugin includes numerous tunable parameters (e.g., critic weights), thereby significantly expanding the configuration space. As a result, planner–controller selection and tuning are often manual, time-consuming, and dependent on prior experience. For example, recent efforts to evaluate ROS 2 navigation in institutional settings required researchers to manually construct a custom simulation to assess a single planner–controller combination for a specific custom robot [15].

Existing benchmarking platforms do not provide a comprehensive evaluation solution for ROS 2-based Nav2 systems. MAVBench [16] targets aerial vehicles, Bench-mr [17] lacks ROS 2 compatibility, and RobotPerf [18] focuses primarily on computational metrics. Arena-Bench [19] and Arena-Bench 2.0 [20] emphasize controller benchmarking without systematically evaluating global planners.

To address these challenges, we introduce ROSNavBench, an open-source, fully automated, and customizable benchmarking framework for Nav2 in ROS 2. ROSNavBench evaluates planner–controller combinations using both navigational and computational performance metrics, enabling data-driven selection and objective comparison, as shown in Figure 2.

Figure 2. The ROSNavBench workflow: input specifications on the left, the automated benchmarking loop in the center, and the generated data and report on the right.

This paper presents the following contributions:

• Automated Combinatorial Benchmarking and Ranking: This is a modular ROS 2 framework that evaluates the global planner and the controller as a coupled pair rather than one at a time. The user specifies the robot, the environment, and the mission, and a multi-criteria decision layer ranks the resulting configurations on navigational metrics such as path quality together with computational metrics such as hardware usage.
• Simulated versus Physical Performance Consistency: This is empirical validation comparing Gazebo simulation [21] results with physical robot deployments to quantify navigation performance degradation due to the reality gap.
• Open-Source Release: This is a publicly available benchmarking framework that facilitates transparency, extensibility, and community-driven development. The framework is available online [22].

2. Related Works

Global planner and controller efficiency matters across applications from industrial automation to autonomous vehicles, and it depends heavily on the type of environment. For instance, feature-sparse environments have different requirements than more confined areas, such as warehouses or narrow passages in houses or offices. The optimal combination of a planner, a controller, and their configuration parameters therefore differs from one deployment to the next. ROS 2 offers a robust framework for such developments but lacks a standardized method for evaluating and comparing the efficacy of different planners and controllers within its ecosystem.

Frameworks such as MAVBench [16] and MotionBenchMaker [23] provide specialized benchmarks for aerial robots and manipulation, respectively. Neither adapts to ROS 2-based mobile ground robots.

Robowflex [24] supports MoveIt rather than Nav2, and Bench-mr [17] does not integrate with ROS at all, whereas ROSNavBench targets ROS 2 and Nav2 directly. Meanwhile, while computational benchmarking tools such as RobotPerf [18] and RTRBench [25] assess various performance metrics, including latency, they do not address navigational aspects, such as the path length. This limitation restricts the comprehensiveness of these frameworks. Pathbench [26] does measure computational cost but only the memory a planning algorithm uses while generating a static path, not the CPU and RAM the stack consumes while the robot is running.

Furthermore, tools illustrated in Table 1 have been compared to our proposed framework, ROSNavBench. Throughout our literature review, we identified a critical gap in the assessment of both global planners and controllers within the ROS 2 ecosystem. This gap is particularly evident in evaluating navigation and computational aspects while allowing for customizability to examine performance in specific deployment scenarios. Existing frameworks measure different things for different purposes, from controller performance among moving pedestrians in Arena-Bench 2.0 [20] to static environmental complexity and obstacle density in BARN [27]. Most emit raw data for plotting, which still leaves the reader to weigh several metrics against each other for a particular scenario. ROSNavBench closes that step. It evaluates global planners and controllers as coupled pairs; it records navigational metrics such as the path length, navigation time, and safety alongside computational metrics such as CPU and memory use while the mission runs; and it applies a customizable weighted performance function that ranks configurations for a given set of operational constraints. This paper introduces and validates ROSNavBench, a comprehensive benchmarking framework, to facilitate the customizable evaluation of planners and controllers in the ROS 2 ecosystem.

Table 1. Comparison between existing tools and ROSNavBench.

Paper	ROS 1	ROS 2	Controller	Global Planner	Navigation Time	CPU Usage	Memory Usage	Safety	Path Quality	Recoveries
bench-mr [17]				X
Infrastructure for motion planning algorithms [28]	X			X
Pathbench [26]	X			X
Simulation framework for local trajectory planners [29]	X		X		X				X
MRPB 1.0 [30]	X		X		X			X	X
localPlannerBench Suite [31]	X		X		X			X	X
Arena-Bench [19]	X		X		X			X	X
BARN [27]	X		X		X
Arena-Bench 2.0 [20]		X	X		X			X	X
Latency-aware LLM benchmark [32]		X	X		X				X
ROSNavBench (ours)		X	X	X	X	X	X	X	X	X

Note: An X in the table denotes support for the feature. The metrics represent the data collected during the mission. Existing global planner frameworks score abstract path generation with static metrics such as the algorithmic planning time or theoretical path length. The metrics in this table are execution data, such as the total navigation time, which those benchmarks do not record.

3. Materials and Methods

ROSNavBench employs a modular architecture composed of three main blocks: the YAML configuration block, the experiment block, and the data analysis block (Figure 3). The YAML configuration file defines all benchmarking parameters, while the experiment and data analysis blocks execute and evaluate the benchmarking process. The framework integrates with ROS 2, Nav2, and Gazebo for simulation.

Figure 3. System architecture of ROSNavBench.

3.1. YAML Configuration Block

The YAML file specifies the robot model, simulation world, Nav2 configuration, planner and controller lists, trajectories, iteration count, and performance metrics with optional weights, defining the entire benchmarking set-up. Note that the controller and the planner plugin can be tested with different parameters in the same experiment by including multiple instances of the plugin. ROSNavBench reads an existing Nav2-compatible set-up rather than replacing it, and thus filling in the YAML file means pointing at assets the user already has. We estimate this takes about 3 min.

3.2. Experiment Block

• Module 1: Experiment Manager initializes the simulation and Nav2 and management of all modules.
• Module 2: Trajectory Generator generates navigation trajectories as sequences of waypoints. A user can give a start–goal pair, a multi-waypoint route, or one of the built-in geometric templates. The circle and square templates expand into dense waypoints on their own, which covers continuous curves and sharp corners without entering points by hand. The module then checks that every waypoint lies in obstacle-free space and that the global planner can reach it. Therefore, an infeasible trajectory is caught before the run starts.
• Module 3: Experiment Executor runs each planner–controller–trajectory combination by setting the robot pose, assigning the goal, computing a global plan, executing the controller, and logging performance metrics into a CSV file for analysis and PDF generation.

3.3. Data Analysis Block

This block analyzes the collected data to rank planner–controller configurations on the metrics the user selected through multi-criteria decision making.

Each metric is denoted by $M_i$:

$$M_i\in \{M_1,M_2,\cdots ,M_n\}$$

(1)

Each metric is averaged over the iterations of a given planner–controller–trajectory combination and then rescaled onto $[0,1]$ by Min–Max normalization [33]. Metrics where lower values reflect better performance are normalized using

$$\overline{M_i}={\displaystyle \frac{M_{i_{max}}-M_i}{M_{i_{max}}-M_{i_{min}}}}$$

(2)

Safety, the minimum clearance to the nearest obstacle, is a metric where a larger raw value is better, and thus we normalize it using

$$\overline{M_i}={\displaystyle \frac{M_i-M_{i_{min}}}{M_{i_{max}}-M_{i_{min}}}}$$

(3)

Every metric carries a weight ${\alpha }_i$, subject to (adapted from [34]):

$$\sum _i{\alpha }_i=1,0<{\alpha }_i\le 1$$

(4)

Two routes set these weights. The first reads them straight off the order the user listed the metrics in Equation (1), with m being the total number of metrics given [35]:

$${\alpha }_i={\displaystyle \frac{m-i+1}{{\sum }_{j=1}^{m}j}}$$

(5)

The second asks the user for a non-negative and nonzero score $u_i$ per metric, saying how much that metric matters to them:

$${\alpha }_i={\displaystyle \frac{u_i}{{\sum }_{j=1}^m{u}_j}}$$

(6)

A weighted sum then gives the score of combination c [33,34]:

$$P_c={\displaystyle \sum _{t=1}^{n_t}}{\displaystyle \sum _{i=1}^m}{\alpha }_i\overline{M_i^t}$$

(7)

Here, $\overline{M_i^t}$ is the metric i normalized for trajectory type t, $n_t$ is the number of the evaluated trajectories, and the winning configuration is simply

$$P_c^{*}=\max \left(P_c\right)$$

(8)

The Report Generation module then plots the results and lists the configurations in rank order.

3.4. Reporting and Automated Insights

Once the experiments finish, ROSNavBench writes its output at two levels of detail, namely one for close technical inspection and one for deciding what to deploy:

• Raw Data: One time-stamped CSV holds every metric the run recorded. This is the data to work from for any post-processing of one’s own.
• Comprehensive Analytical Report: This is a PDF collecting the statistical distribution of each metric, plots comparing planner–controller performance, and the final weighted ranking from Equation (7).

• The report supports two styles of comparison. A user can compare different planner–controller combinations against each other or hold one component fixed and vary the other, such as running a single planner under several allowed planning times while keeping the controller unchanged, which shows what a particular parameter costs. The temporal metrics, particularly the navigation time, measure the system end to end. They therefore include algorithmic replanning delay, ROS 2 middleware overhead, and actuator latency together and do not separate them. ROSNavBench reports the latency the mission actually experiences; it does not profile the execution time of an individual replanning request.

4. Results

We compared the Nav2 stack in simulation against the same stack on hardware, with both on ROS 2 Jazzy Jalisco. Varying the hardware and the software together was what tested whether ROSNavBench generalized and how much the two domains agreed.

4.1. Hardware and Simulation Set-Up

Experiments were conducted in real and simulated environments. Simulation experiments were performed on a high-performance workstation (Table 2). The physical runs used a TurtleBot3 Waffle Pi [36] on a Raspberry Pi 5 with 8 GB of RAM under Ubuntu 24.04 and ROS 2 Jazzy. Between the workstation and the Pi, the same algorithms were therefore measured once with computing power to spare and once on an edge device that had none. Automation here refers to unattended batch execution across configurations; in the simulations, ROSNavBench cycled through every planner–controller–trajectory combination without intervention. On hardware, this batch execution was not automated, since resetting the robot’s pose to a precise starting position required an operator. We rebuilt the physical room in Gazebo Sim from a pre-generated map so that the two domains matched as closely as possible. We logged the execution time, path length, path deviation, safety, recoveries, and success rate, together with system-wide CPU and memory use. Those last two factors covered the whole machine and not one process.

Table 2. Simulation workstation specifications.

Feature	Specification
Processor	Intel i7-12700K (20 Cores)
RAM	32.0 GB
GPU	NVIDIA RTX 3080
OS	Ubuntu 24.04
ROS 2 Distro	Jazzy Jalisco
Sim Engine	Gazebo Sim

4.2. Experimental Design and Scenarios

Two scenarios carry the comparison between simulation and physical deployment. One extends wide across planner–controller combinations; the other stress-tests a single planner–controller combination (Figure 4 and Figure 5).

Figure 4. The corridor used for Scenario I.

Figure 5. The laboratory used for Scenario II.

4.2.1. Scenario I: Combinatorial Cross-Domain Validation

This scenario asks how far simulation fidelity holds once the planner–controller space is covered broadly. In the corridor of Figure 4, we ran 24 configurations: four global planners, GridBased/NavFn, Smac 2D, Theta*, and State Lattice, against six controllers, MPPI, Graceful, DWB, RPP, and the RSC-augmented forms of the last two.

Each configuration was run three times in a simulation and three times on the physical robot. The task was a single goal around a 90° corner. Nothing failed; the corridor was controlled enough that the success rate held at 100% across every trial in Table 3. With no failures to account for, the differences that remain between the workstation and the Raspberry Pi 5 runs are differences in kinematic performance and in computational overhead.

Table 3. ScenarioI path length and navigation time, simulation against reality, given as mean [range] over three trials.

		Navigation Time (s)		Path Length (m)
Controller	Planner	Real	Sim	Real	Sim
DWB	GridBased	55.17 [51.90, 57.81]	53.00 [52.32, 54.28]	12.18 [11.64, 12.49]	11.83 [11.81, 11.86]
DWB	Lattice	55.11 [53.76, 55.82]	49.48 [49.02, 50.03]	12.14 [11.83, 12.30]	11.63 [11.55, 11.75]
DWB	ThetaStar	55.75 [53.79, 59.64]	48.38 [48.27, 48.58]	12.03 [11.78, 12.43]	11.32 [11.29, 11.38]
DWB	smac_planner	57.82 [55.84, 59.82]	51.89 [51.29, 52.82]	12.01 [11.73, 12.21]	11.74 [11.69, 11.77]
DWB_RSC	GridBased	58.52 [57.85, 59.81]	51.75 [48.47, 53.47]	12.33 [12.27, 12.43]	11.63 [11.47, 11.71]
DWB_RSC	Lattice	55.73 [55.71, 55.74]	49.21 [48.13, 49.88]	12.32 [12.27, 12.35]	11.59 [11.49, 11.67]
DWB_RSC	ThetaStar	53.07 [51.75, 53.75]	48.27 [47.75, 48.64]	11.76 [11.52, 11.87]	11.32 [11.19, 11.40]
DWB_RSC	smac_planner	61.15 [59.76, 61.86]	51.72 [47.80, 55.56]	12.41 [12.35, 12.45]	11.45 [11.23, 11.60]
Graceful	GridBased	65.12 [63.76, 67.76]	60.21 [59.50, 60.84]	12.15 [11.97, 12.48]	11.52 [11.45, 11.61]
Graceful	Lattice	65.84 [65.82, 65.85]	62.54 [61.90, 62.91]	12.15 [12.10, 12.19]	11.77 [11.73, 11.78]
Graceful	ThetaStar	64.47 [63.80, 65.78]	60.20 [59.03, 60.91]	11.97 [11.79, 12.27]	11.54 [11.48, 11.61]
Graceful	smac_planner	64.50 [63.81, 65.84]	62.69 [61.77, 63.24]	11.89 [11.70, 12.13]	11.49 [11.41, 11.55]
MPPI	GridBased	95.08 [89.72, 99.77]	79.86 [77.52, 81.39]	12.99 [12.77, 13.17]	11.88 [11.80, 11.97]
MPPI	Lattice	98.42 [87.79, 105.69]	83.79 [76.74, 90.12]	13.72 [13.38, 14.32]	12.17 [11.66, 12.79]
MPPI	ThetaStar	85.82 [83.77, 87.81]	80.42 [77.76, 84.10]	12.33 [11.81, 12.67]	11.83 [11.64, 12.05]
MPPI	smac_planner	87.80 [79.88, 99.75]	79.65 [77.31, 81.65]	12.55 [11.85, 13.15]	11.63 [11.37, 11.80]
RPP	GridBased	53.18 [51.88, 53.85]	47.45 [46.36, 48.07]	12.08 [11.94, 12.16]	11.37 [11.21, 11.52]
RPP	Lattice	55.13 [53.86, 55.79]	48.96 [48.57, 49.51]	12.34 [12.18, 12.47]	11.71 [11.61, 11.80]
RPP	ThetaStar	52.47 [51.77, 53.80]	47.37 [46.80, 48.10]	11.80 [11.74, 11.88]	11.38 [11.28, 11.52]
RPP	smac_planner	57.80 [57.78, 57.83]	47.28 [46.64, 47.70]	12.15 [12.10, 12.18]	11.30 [11.23, 11.39]
RPP_RSC	GridBased	57.15 [55.79, 59.79]	47.67 [47.22, 47.96]	12.42 [12.25, 12.53]	11.39 [11.28, 11.45]
RPP_RSC	Lattice	53.85 [53.82, 53.89]	48.35 [47.31, 49.30]	12.32 [12.23, 12.38]	11.63 [11.38, 11.78]
RPP_RSC	ThetaStar	54.46 [53.75, 55.83]	48.98 [47.95, 49.60]	12.14 [12.02, 12.29]	11.51 [11.40, 11.59]
RPP_RSC	smac_planner	58.49 [57.80, 59.83]	48.45 [47.78, 49.34]	12.17 [12.13, 12.21]	11.36 [11.27, 11.43]

To see replanning under load, we added obstacles, keeping the Smac planner and the RPP controller. Two static obstacles went into the same corridor at (−4.3, 3.0) and (−5.5, 1.7), and the robot drove past them three times in a simulation and three times for real. Figure 6 shows every trial succeeding. The physical robot took, on average, 3 s longer to complete the trajectory than in the simulation.

Figure 6. Obstacle avoidance extension to Scenario I, Smac planner with RPP controller: the trajectories actually driven.

4.2.2. Scenario II: Stress Testing in High-Density Environments

Scenario I went for breadth. Scenario II went for the limit in a laboratory with narrow passages and a map that had degraded (Figure 5). We fixed the navigation stack to a representative pair of the Smac Global Planner and DWB controller. Two tasks were benchmarked through three iterations in both simulation and reality:

• Single-Goal: point to point through clutter.
• Multi-Waypoint: two waypoints in sequence down a path narrow enough to demand precision.

What this measures is the fidelity gap at its worst, where a poor static map and tight clearances work against the planner and the controller at the same time.

4.3. Comparative Analysis: Scenario I

Figure 7 and Figure 8 put the real and simulated trajectories side by side. The two domains behaved alike down to small details. For instance, MPPI was visibly jerky on hardware, and it was just as jerky in the simulation.

Figure 7. Paths as planned by the global planner, with simulated as a blue line and real as a red line. The environment, start, and goal are identical in both.

Figure 8. Paths as actually driven in Scenario I, with simulated as a blue line and real as a red line.

The RSC lets the robot rotate in place before setting off, which lines it up with the path immediately instead of sweeping through the opening arc a plain controller produces (Figure 9). Table 3 and Figure 10 carry the numbers. All 24 configurations reached 100% success, and thus in an environment this controlled, what the simulation said about the kinematic behavior carried over.

Figure 9. RSC effect in the simulated corridor executed path (Scenario I).

Figure 10. CPU usage, real against simulated, for Scenario I.

4.4. Comparative Analysis: Scenario II

The static map is degraded here, yet the paths still line up well between the simulation and hardware, as shown in Figure 11. Where they parted company was for the time and compute in Table 4. The computing power of the two platforms accounts for the second. The timing difference says that how closely the domains tracked each other depended on the scenario.

Figure 11. Real versus simulation executed path for each Scenario II task.

Table 4. Scenario II: simulation against reality.

	Waypoints		Single Goal
Metric	Sim	Real	Sim	Real
Time (s)	35.79	45.91	18.96	35.25
Path Length (m)	7.51	8.20	3.60	3.96
CPU Usage (%)	13.69	17.25	10.50	17.00

Note: Values were averaged over 3 iterations for each task type.

4.5. Comprehensive Reporting

ROSNavBench generates a detailed PDF report summarizing the benchmarking process through statistical tables, metric distributions, and computational profiling. Performance is ranked using a multi-criteria scoring function (Equation (7)).

Figure 12a shows part of one such report, which aggregates user-specified metrics into a single score for each planner–controller pair. This compact visualization supports selection of the planner–controller combinations that perform best on the user’s criteria. In Figure 12a, the scores prioritize time, followed by the path length, safety, CPU, and memory. Consequently, this weighting scheme resulted in the highest overall score for the combination of the RPP_RSC controller and the GridBased planner in the Scenario I simulation.

Figure 12. The same Scenario I simulation data scored four ways. Each heatmap uses a different weighting profile, and the ordering of the planner–controller configurations moves with it. (a) Priority: time, path length, safety, CPU, memory. (b) Priority: CPU and memory. (c) Priority: CPU, memory, safety, path length, time. (d) Priority: safety, path length, time, CPU.

How much of that ranking was the data, and how much was the weights? To find out, we scored the Scenario I simulation data again under other weightings. Turning the scheme around for a resource-constrained deployment, with CPU and memory above the navigation time, moved the ordering a long way. GridBased + RPP_RSC stayed on top in this environment. Below it, the field resorted itself by computational cost, and smac_planner + RPP was strong whenever the time and path length led the way, but dropped behind lighter pairings. Therefore, the ranking did follow the weights, and it followed whichever metric was pushed up, which is the behavior a weighted sum is supposed to have. Read any recommendation ROSNavBench makes next to the weights that produced it.

4.6. Comparative Analysis with Existing Frameworks

Arena-Bench 2.0 and its relatives were built for human-centric social navigation, and they concentrate on social interaction and on how the controller behaves. ROSNavBench sits beside that rather than against it, evaluating the stack as a whole and, in particular, how the global planner and the controller behave as a pair (Table 5). The Arena suite leans on generative models for complicated social scenes; we leaned on repeated automated runs measuring computational cost and navigational reliability. The other difference is the cross-domain fidelity work. It is not automated inside the software yet, but this study shows what the framework can say about the distance between a simulated result and a deployed one.

Table 5. Comparison of existing benchmarking frameworks with ROSNavBench.

Framework	Sim.	Primary Metrics	Joint Eval. 1	Fidelity Analysis 2
Arena-Bench 2.0 [20]	✓	Social and Nav.	×	×
ROSNavBench (Ours)	✓	Comp. and Nav.	✓	✓

¹ Combined global planner and controller evaluation. ² Cross-domain fidelity analysis performed within the work. Note: Nav. = navigational; Comp. = computational; Sim. = simulation; Eval. = evaluation.

Comparison of performance claims across studies and the need for a standard framework becomes concrete. A recent evaluation of Nav2 controllers driven by natural language instructions reported numbers we cannot fairly place against ours [32]. They ran a TurtleBot4, whereas we ran a TurtleBot3 Waffle Pi, and the acceleration parameters they documented differed from ours. Add whatever parameters went unreported and could still have moved the result, and the two sets of numbers stop being comparable at all. That is the gap a standardized framework closes.

5. Discussion

In a simulation, ROSNavBench did what it was built to do: running the tests, collecting the data, and writing the reports across a wide set of global planners and controllers without supervision. We then repeated those runs on the robot under matched conditions, which is what makes the cross-platform comparison below possible.

5.1. Cross-Platform Performance and the Reality Gap

The navigation times in the two domains tracked each other closely in Scenario I at $r=0.97$, yet a measurable reality gap remained. Scenario II put that gap at 16.29 s on the single-goal task and 10.12 s on the waypoint one. Over all of Scenario I, the gap averaged 6.85 s in time and 0.68 m in path length. This difference was not fixed; with the planner and controller parameters held constant, the gap still varied between repeated runs. The variation was likely driven by localization drift, wheel slippage, and sensor noise, none of which repeated identically. Therefore, the reality gap behaved as a quantity that varied rather than a constant one could subtract. Gazebo does model sensor noise, and an offset in time and path length survives it anyway. Pinning that offset down would take navigation data gathered across many platforms. CPU and memory tell a different story again, with the Raspberry Pi having far less capacity than the workstation (Figure 10). CPU utilization is expressed as a percentage of each machine’s total capacity. The two values were normalized against different denominators, and thus their absolute values did not belong on the same axis.

Two separate things drove these cross-domain differences, and reading one as the other would misattribute the result. There is the platform effect, with a 20-core workstation on one side and a Raspberry Pi 5 on the other. Then, there is the reality gap itself: wheel slippage, sensor noise, localization drift, and actuator dynamics, none of which the simulator reproduced in full. The computational metrics were almost purely the first. The geometric ones, with path length among them, were almost purely the second. Navigation time is the awkward case, because it moved with both.

5.2. Fidelity Versus Experimental Throughput

Fidelity and throughput pulled against each other. Only the real robot showed the actuator performance, floor friction, and latency as they actually were. Achieving that costs time and attention; someone has to watch the runs, swap batteries, and pick the robot up after it hits something. A simulation asks for none of that and wears nothing out. Thus, the runs can be frequent and unattended. That is what makes broad statistical validation practical overnight and across edge cases in a way a real lab never will be. Developing a new controller iteratively or stress-testing one to find where it gives out is exactly the case that benefits, and the robot survives it.

5.3. Limitations and Future Directions

The trends correlate, but a numerical gap survives because the simulator simplified some real-world effects. ROSNavBench also stopped short of tuning; it will test any planner–controller configuration given to it automatically, and it will not search for a better one. These results reflect our chosen parameters and could shift under a different setting, which highlights the importance of a customizable benchmarking framework. We plan to close both with predictive system identification to align the simulator to the hardware and automated tuning loops on top of the existing scoring. ROSNavBench currently evaluates a single robot per trial. Extending it to multi-robot navigation would require synchronized multi-agent spawning and goal assignment, together with metrics that do not exist in the present framework, such as the path conflict rate. We regard this as a future direction.

6. Conclusions

This study introduced ROSNavBench, an open-source framework for the automated, systematic evaluation of global planners and controllers. By evaluating computational and navigational metrics, the framework allows researchers to identify optimal configurations tailored to specific robotic requirements.

Our cross-platform analysis shows that the simulations correlated well with the physical experiments, though a reality gap remained. Even with identical parameters on the same robot, the size of that gap changed from trial to trial. Future work will also explore Docker-based resource isolation to more closely replicate edge device constraints and integrate LLMs for automated hyperparameter tuning.