LLM-Enhanced Control of a Mobile Robotic Platform for Smart Industry

Abstract

The emergence of highly complex generative AI and large language models represents both a significant challenge and an opportunity for multiple engineering domains. Under the Industry 4.0 paradigm, various connected automation and industrial engineering applications can leverage the inference and generative design capabilities of these models to improve control algorithms and systems. In particular, widespread deployment of mobile robotic platforms in modern industry, enhanced with LLM capabilities, can provide a substantial increase in the efficiency and cost-effectiveness of such solutions. In this study, we investigate the suitability of current-generation LLM systems for industrial mobile robot control. We present a systematic, end-to-end methodology for benchmarking four GenAI/LLMs, SmolLM2, Llama 3.2, Gemma3, and Gemma3-qat, for a typical mobile robot platform configuration. The approach is two-staged, based on both assessing the specific domain knowledge of the models in an industrial context and their integration with a robotic simulation environment based on ROS2. Reported results focus on quantitative assessment of multiple metrics (quality, coverage, speed, and reliability) and their integration in aggregated scoring mechanisms, which can help developers select and adapt the best model for a particular application, together with custom software implementation.

1. Introduction

As mentioned in Wang et al. [1], the rising expectations in robotics are closely tied to the rapid evolution of large language models (LLMs). As LLMs continue to evolve and accumulate grounded knowledge from multimodal data, they are expected to play a pivotal role in the next generation of intelligent robotic systems. By addressing the existing challenges through collaborative and interdisciplinary efforts, the full potential of LLMs in robotics can be unlocked, leading to advancements that could profoundly impact various industries and aspects of daily life Zeng et al. [2]. We build upon this vision by integrating LLMs Liu et al. [3] into an industrial mobile robot logistics application, where the models act as ROS-based expert agents that help end-users monitor and control the mobile platform through natural language.

Following the work of Okumuş et al. [4], which emphasizes that simulation-based experiments are well suited to demonstrate the theoretical applicabillity of novel control approaches, we evaluated our LLM-enhanced control solution both on the physical ASTIbot platform and in a ROS2-based simulation environment.

In their recent survey, the authors of Bernardo et al. [5] argue that AI strategies are increasingly being incorporated into robots, with a growing emphasis on semantic-knowledge-based approaches that allow robots to understand not only spatial aspects of the environment but also the meaning of each element and how humans interact with them.

Large industrial organizations, such as SIEMENS AG and Beckhoff Automation, are actively deploying many types of software agents across various areas of industrial automation Rainer Jr. et al. [6], from software development assistants to self-diagnostics and self-maintenance software agents. Human-centric automation paradigms Dong et al. [7] also lead to the integration of such software agents in complex frameworks of LLM-based solutions, including applications in prognostics and maintenance Vidyaratne et al. [8].

We consider two types of robotic systems for the purpose of our study: hardware and software robots. Hardware or physical robots are tangible systems composed of a multitude of sensors and actuators designed to manipulate or execute a predefined task in a variety of work environments while collaborating with human operators Sherwani et al. [9]. Software or virtual robots are those entities that operate entirely in digital environments and simulations and consist of algorithms, and even ML algorithms, that allow them to imitate human actions Agostinelli et al. [10].

1.1. Hardware Robots

Physical mobile robots, such as autonomous mobile robots (AMRs) and automated guided vehicles (AGVs), are now a standard element in many smart factories and warehouses Pavel et al. [11]. They move materials between storage, production lines, and shipping areas, and they can also perform routine inspection or inventory tasks. Their main advantage is that they execute repetitive transport and handling operations with high precision and consistent timing, which supports stable and predictable production flows.

Compared to manual handling, mobile robots can operate continuously, with no need for breaks or shift changes. This enables higher throughput and smoother coordination with production schedules, especially in multi-shift or 24/7 environments Raj and Kos [12]. They can also handle heavy loads and work in areas that may be uncomfortable or unsafe for human workers, such as narrow aisles, cold storage, or zones with moving machinery. This can reduce workplace accidents and ergonomic strain.

The introduction of mobile robots requires a significant initial investment in hardware, infrastructure adaptation, and integration with existing IT and production systems. However, once deployed, they can reduce the dependency on manual transport tasks and lower labor costs over time Liu et al. [13]. Because they offer repeatable performance and fewer handling errors, they can also reduce damage to materials and improve overall process quality. As fleets grow, companies can scale up robot usage more easily than recruiting and training additional operators for the same tasks.

1.2. Software Robots

Alongside physical robots, many industrial processes now rely on software robots. These include rule-based process automation scripts, orchestration services, monitoring agents, and, more recently, large language model-based assistants. Unlike physical robots, virtual robots act on digital information, reading data from databases or logs, communicating with APIs, and issuing commands to other systems, including physical robots Shidaganti et al. [14].

In the context of mobile robots, software agents are responsible for planning and coordinating tasks, interfacing with higher-level systems, and providing feedback to human operators. Traditional approaches rely on fixed rules or handcrafted decision logic: each workflow or exception must be encoded manually. This makes the system rigid and hard to adapt when production requirements change or when unexpected events occur on the shop floor.

LLM-based agents extend these software robots by adding the ability to interpret natural language, combine heterogeneous information, and generate structured plans. They can translate high-level human instructions into concrete missions for mobile robots, propose re-planning options when paths are blocked or orders are delayed, and summarize system status in a way that is easier for operators to understand. Because they are purely software, they are easy to update, copy, and scale, and they can be integrated with existing industrial IT systems without changes to the physical environment.

1.3. LLM-Enhanced Control for Mobile Platforms

This research focuses on the interaction between physical mobile robots and LLM-based software agents in smart industry. The physical robots provide the ability to act in the factory; they move materials, inspect areas, and interact with other equipment. The goal is not to replace existing low-level control and navigation algorithms, but to enhance higher-level decision-making and interactions around them.

In the proposed approach, operators and production planners can express tasks in natural language using terms from their daily work (for example, specifying which line needs material, or which area needs inspection). The LLM agent interprets these instructions, checks contextual information such as current robot locations and production priorities, and then generates structured missions that existing planning and control modules can execute. When disturbances occur, such as blocked routes or urgent orders, the LLM agent can propose alternative plans or rescheduling options and present them in an understandable way.

By combining physical mobile platforms with LLM-based software robots, the system aims to increase flexibility and reduce the engineering effort required to adapt to new products, layouts, or workflows. Instead of modifying low-level code for each change, many adjustments can be made through high-level interaction with the LLM agent. At the same time, physical robots still provide the reliable, repeatable execution needed for industrial environments. This research evaluates how this combined architecture affects efficiency, robustness, and usability in representative smart industry scenarios. Table 1 presents a key overview of hardware and software robots, along with the potential of AI components and methods for the improvement of their operation.

Table 1. Overview of physical and software (virtual) robots, their characteristics, their applications, and the impact of AI.

Robots	Characteristics	Applications	AI Impact
Physical industrial robots	High-precision, repeatable motions; can handle heavy loads; continuous operation with minimal breaks; high upfront investment but long-term cost savings.	Assembly, welding, painting, material handling in manufacturing Pavel and Stamatescu [15]; logistics and warehousing.	AI-based perception and control improve path planning, quality inspection, and adaptive behavior to variations in the environment.
Mobile service robots	Autonomous navigation in dynamic environments; interaction with humans and objects; equipped with sensors and onboard computing.	Hospital logistics, delivery robots, inspection and maintenance, hospitality and retail.	AI enables robust localization, mapping, and human–robot interaction, extending deployment to unstructured and crowded spaces.
Software process robots (RPA)	Operate in purely digital environments; mimic human interactions with user interfaces and APIs; highly scalable and easily replicable.	Back-office processes (invoice handling, order processing, report generation), integration of legacy IT systems.	ML and NLP allow bots to handle semi-structured data, classify documents, and make context-aware decisions.
Cloud orchestration and infrastructure bots	Monitor and manage distributed computational resources; automatically deploy, scale, and update services.	Fleet management platforms for robots, cloud-based perception or planning services, CI/CD pipelines for robotic software.	AI-driven policies optimize resource allocation, predict load, and enable self-healing infrastructures that support large-scale robotic deployments.
Digital twin and simulation agents	Virtual replicas of physical robots and environments Rosioru et al. [16]; run accelerated simulations and what-if scenarios.	Design and testing of robotic cells, optimization of production lines, energy and throughput analysis, operator training.	AI uses simulation data for policy learning and transfer learning to improve real-world performance.
Data-analysis and monitoring agents	Continuously analyze sensor and log data; detect anomalies and predict failures; provide decision-support dashboards.	Predictive maintenance for robots, quality control, energy optimization, safety monitoring in industrial plants.	Advanced ML models increase accuracy of failure prediction, reduce false alarms, and support proactive interventions that minimize downtime.
Human-facing conversational agents	Natural-language interfaces to robotic systems; support operators and end-users with explanations and task specification.	Voice or chat interfaces for configuring robots, training support, remote assistance and teleoperation.	LLMs enable more intuitive multimodal interaction, automatic documentation, and human–machine collaboration.

This paper is organized as follows: Section 2 describes the experimental setup, including the evaluated language models, the ROS/ROS2 benchmark suite, and the scoring methodology, and it details the mobile robot control and kinematic modeling background used throughout the study. Section 3 presents quantitative results on quality, coverage, speed, and reliability, together with aggregated rankings and representative prompt examples. Finally, the Conclusion summarizes the key findings and outlines implications for selecting and deploying LLM-based agents in ROS2-driven mobile robotics applications.

2. Materials and Methods

We evaluated six large language models (LLMs), presented in Table 2, as expert agents for Robot Operating System (ROS) and ROS2 development. Models were tested on 40+ prompts covering beginner- to expert-level topics including navigation, perception, hardware integration, and system optimization. Gemma3 achieved the highest composite score (58.4/100), demonstrating superior quality (84.61/100) and topic coverage (47.62/100). Our analysis reveals significant performance variations across different ROS domains, with implications for selecting appropriate models for robotics applications.

While general LLM benchmarks exist Zografos and Moussiades [17], domain-specific evaluation for robotics remains underexplored.

Table 2. Large language models evaluated as ROS/ROS2 expert agents.

Model	Description	Size
SmolLM2 Allal et al. [18]	Small-scale efficient model	256.35 (MB)
Llama 3.2 Zhao et al. [19]	Meta’s latest open model	1.87 (GB)
Gemma3, Gemma3-qat Team et al. [20]	Google’s efficient model	2.31 (GB)
DeepSeek R1 Distill	Reasoning-focused distilled model	4.6 (GB)
Qwen 3	Alibaba’s multilingual model	4.7 (GB)

Table 2. Large language models evaluated as ROS/ROS2 expert agents.

Model	Description	Size
SmolLM2 Allal et al. [18]	Small-scale efficient model	256.35 (MB)
Llama 3.2 Zhao et al. [19]	Meta’s latest open model	1.87 (GB)
Gemma3, Gemma3-qat Team et al. [20]	Google’s efficient model	2.31 (GB)
DeepSeek R1 Distill	Reasoning-focused distilled model	4.6 (GB)
Qwen 3	Alibaba’s multilingual model	4.7 (GB)

The smallest available GPT-Oss model exceeds 11 GB, which prevents it from fitting into the dedicated GPU memory on our hardware (maximum 4 GB). The same limitation applies to other state-of-the-art models such as DeepSeek-R1 and Qwen 3. This constraint highlights a central challenge of deploying LLMs on embedded platforms: limited on-device resources require highly optimized inference implementations.

We designed a comprehensive test suite with 40+ prompts across five difficulty levels:

• Beginner (n = 5): Basic concepts, installation, and simple publisher–subscriber topics;
• Intermediate (n = 6): Services, launch files, custom messages, and parameters;
• Advanced (n = 6): Actions, lifecycle nodes, navigation, and perception;
• Expert (n = 6): Performance optimization, hardware integration, and security;
• Troubleshooting (n = 3): Common issues and debugging.

Each prompt included three to five expected topics for automated coverage analysis.

We evaluated models on multiple dimensions:

• Quality Score (0–100): Composite measure based on response length, code inclusion, and topic coverage;
• Coverage Score (0–100%): Percentage of expected topics mentioned;
• Performance Metrics: Response time, tokens/second;
• Reliability: Success rate, error rate, and timeout rate;
• Code Generation: Frequency and quality of code examples.

A weighted composite score combining these metrics follows the formula described in Equation (1):

$$S=0.35\times Q+0.3\times C+0.2\times Sp+0.15\times R,$$

(1)

where S is the composite score, Q is the quality of the response, C is the coverage of the subject, $Sp$ is the speed of token generation, and R is the reliability of the model, with more interest in quality and coverage, as speed is not our main focus. In Equation (2), each of these metrics have been normed and scaled between 0 and 100 to fit the final formula.

$$\begin{array}{c}\hfill Q={\displaystyle \frac{\sum Q_{scores}}{N_{tests}}},C={\displaystyle \frac{\sum C_{scores}}{N_{tests}}}\\ \hfill Sp=min({\displaystyle \frac{\frac{\sum S{p}_{scores}}{N_{tests}}}{maxS{p}_{scores}}},100)\\ \hfill R={\displaystyle \frac{N_{successfultests}}{N_{tests}}}\times 100\end{array}$$

(2)

Models were provided with a consistent system prompt establishing them as ROS/ROS2 experts with experience in architecture, navigation, perception, hardware integration, and debugging.

The software robot is implemented as an LLM-based control agent running alongside the ROS2 stack (Figure 1) Koubaa et al. [21]. It exposes a web interface where operators can issue natural-language instructions and inspect execution logs. Internally, the agent maintains a catalog of available topics, services, and actions, and uses a planning layer to map high-level tasks (for example, “inspect zone B and report obstacles”) into concrete ROS2 commands and navigation goals. Execution feedback, diagnostics, and sensor summaries are continuously streamed back to the agent, which interprets them with the LLM and presents concise, human-readable status reports. This architecture allows safety-critical navigation and control loops in conversational ROS2 nodes.

Figure 1. Architecture of the LLM-enhanced software robot for ROS2-based platforms. The agent connects the user web interface with the underlying ROS2 ecosystem, combining a large language model, a command-planning layer, and ROS2 integration nodes that access topics, services, and logs. High-level natural-language requests are translated into structured ROS2 commands and missions, while feedback from the robot and infrastructure is aggregated and summarized back to the user.

Table 3 summarizes the main hardware components used in the experimental setup, together with their role in the proposed mobile robot platform.

Table 3. Hardware platforms and technical specifications.

Computer model	ASUS Vivobook Pro 15 OLED	NVIDIA Jetson Orin Nano 4 GB
CPU	AMD Ryzen 9 5900HX	6-core Arm Cortex-A78AE v8.2
GPU	NVIDIA GeForce RTX 3050	512-core NVIDIA Ampere architecture GPU with 16 tensor cores
Memory	Dedicated GPU memory 4 GB	Shared RAM memory 4 GB
OS	MS Windows 11 Pro	Ubuntu 20.04.6 LTS
Role	Running software LLM agents	Controlling hardware mobile platform

The ASTIbot platform (Figure 2) is a complex industrial-grade mobile robot equipped with a PLC-based control box for the conveyor belt with two digital sensors for material detection and a signal lamp for status indication. It uses four Mecanum wheels driven by independent 24 V DC motors to achieve omnidirectional motion and carries an integrated conveyor module with dedicated sensors for load detection and transfer. The robot also includes a self-charging docking interface for the charging station and additional onboard sensors for safe operation in warehouse and production environments, such as 360° LiDAR and a 3D depth camera Pavel and Stamatescu [22], all managed and controlled by the NVIDIA Jetson Orin Nano embedded GPU development board.

Figure 2. Mobile robot platform (ASTIbot) used for real-life validation experiments: (a) rear view with detailed components of the platform; (b) frontal view.

2.1. Mobile Robot Model

The mobile platform used in this work features four omnidirectional Mecanum wheels, which provide a high degree of maneuverability but also increase the complexity of its kinematic model. Depending on the direction and rotational speed of each wheel, the robot can move in any direction without rotating its chassis around the vertical axis Williams et al. [23]. Example motion patterns are illustrated in Figure 3.

Figure 3. Examples of omnidirectional motion patterns of the Mecanum-wheeled mobile platform.

To analyze how wheel motions relate to the commands sent by the controller, we consider both the direct and inverse kinematic models. The controller receives velocity commands of the form ${\mathbf{u}}^T={[v_x,v_y,w_z]}^T$, which are then decomposed into individual wheel speeds for each motor, as summarized in Table 4.

Table 4. Mobile platform wheel geometry and roller angles used in the kinematic model.

Wheel Index	Position	Associated Velocity	Angle of Rollers
1	Front-left	$w_1$	−45°
2	Front-right	$w_2$	45°
3	Rear-left	$w_3$	45°
4	Rear-right	$w_4$	−45°

2.2. Direct Kinematics

From Figure 3 it can be seen that the linear velocity along the $O_x$-axis is given by the sum of the wheel velocities, whereas the linear velocity along the $O_y$-axis and the angular velocity require a different treatment. For the linear velocity along $O_y$, we consider the resulting direction of motion and the fact that there are always two pairs of wheels whose rotation directions are opposite. The same principle applies when determining the platform’s angular velocity, but in this case we must also account for the distance between the system’s center of gravity and the reference wheel. All of these relationships can be expressed compactly by the Equation (3). For simplicity, we assume that the wheel radius is equal to one.

$$\mathbf{u}=\left[\begin{array}{c}{v}_x\\ v_y\\ {\omega }_z\end{array}\right]=\left[\begin{array}{cccc}1& 1& 1& 1\\ -1& 1& 1& -1\\ {\displaystyle \frac{-1}{l_x+l_y}}& {\displaystyle \frac{1}{l_x+l_y}}& {\displaystyle \frac{-1}{l_x+l_y}}& {\displaystyle \frac{1}{l_x+l_y}}\end{array}\right]\left[\begin{array}{c}{w}_1\\ w_2\\ w_3\\ w_4\end{array}\right]=\mathbf{T}\left[\begin{array}{c}{w}_1\\ w_2\\ w_3\\ w_4\end{array}\right]$$

(3)

where $l_x$ is the distance from the robot’s center of rotation to the wheel along the x-axis (half the robot’s length), and $l_y$ is the distance from the robot’s center of rotation to the wheel along the y-axis (half the robot’s width).

2.3. Inverse Kinematics

Starting from Equation (3), the inverse kinematics can be derived by inverting the transformation matrix $\mathbf{T}$. Since $\mathbf{T}$ is not square, we instead employ its pseudo-inverse ${\mathbf{T}}^{+}$, defined to satisfy the condition in Equation (4).

$$\mathbf{T}{\mathbf{T}}^{+}=\mathbf{T}{\mathbf{T}}^{\mathbf{T}}{\left(\mathbf{T}{\mathbf{T}}^{\mathbf{T}}\right)}^{-1}={\mathbf{I}}_{\mathbf{3}}$$

(4)

Using this definition, the resulting pseudo-inverse can be written as in Equation (5).

$${\mathbf{T}}^{+}={\displaystyle \frac{1}{4}}\left[\begin{array}{ccc}1& -1& -(l_x+l_y)\\ 1& 1& (l_x+l_y)\\ 1& 1& -(l_x+l_y)\\ 1& -1& (l_x+l_y)\end{array}\right]$$

(5)

Consequently, the wheel angular velocities are obtained from the relationship given in Equation (6).

$$\left[\begin{array}{c}{w}_1\\ w_2\\ w_3\\ w_4\end{array}\right]={\mathbf{T}}^{+}\mathbf{u}={\displaystyle \frac{1}{4}}\left[\begin{array}{ccc}1& -1& -(l_x+l_y)\\ 1& 1& (l_x+l_y)\\ 1& 1& -(l_x+l_y)\\ 1& -1& (l_x+l_y)\end{array}\right]\left[\begin{array}{c}{v}_x\\ v_y\\ {\omega }_z\end{array}\right]$$

(6)

2.4. System Control

For the control problem, we are interested in commanding the motion of the entire robotic system, and therefore focus on inputs of the form ${\mathbf{u}}^T={[v_x,v_y,{\omega }_z]}^T$. To control the mobile platform, we require its current position and the orientation of the robot’s front (camera) with respect to a global coordinate frame.

As shown in Figure 4, the robot state can be represented by three variables: the Cartesian coordinates $(x,y)$ in the global x–y plane and the platform orientation $\theta$ relative to the x-axis. This convention is also commonly used in the ROS development environment. Consequently, the state vector for the kinematic model is defined as ${\mathbf{x}}^T={[x,y,\theta ]}^T$, and together with the control input ${\mathbf{u}}^T={[v,\omega ]}^T$ (linear and angular velocities), the system dynamics are given by Equation (7).

$$\dot{\mathbf{x}}=\left[\begin{array}{c}\dot{x}\\ \dot{y}\\ \dot{\theta }\end{array}\right]=\left[\begin{array}{cc}cos\theta & 0\\ sin\theta & 0\\ 0& 1\end{array}\right]\left[\begin{array}{c}v\\ \omega \end{array}\right]$$

(7)

Figure 4. Global and robot-fixed coordinate frames used to define the mobile platform pose.

As shown in Figure 5, if the desired destination is represented by the state vector ${{\mathbf{x}}_{\mathbf{d}}}^T={[x_d,y_d,{\theta }_d]}^T$, the objective of the controller is to minimize the tracking error defined in Equation (8).

$$\underset{t\to \infty }{lim}\mathbf{e}=\underset{t\to \infty }{lim}\left[\begin{array}{c}{x}_e\\ y_e\\ {\theta }_e\end{array}\right]=\underset{t\to \infty }{lim}\left[\begin{array}{ccc}cos\theta & sin\theta & 0\\ -sin\theta & cos\theta & 0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}{x}_d-x\\ y_d-y\\ {\theta }_d-\theta \end{array}\right]\approx \left[\begin{array}{c}0\\ 0\\ 0\end{array}\right]$$

(8)

Figure 5. Geometric definition of the pose-tracking error between the robot state and the desired goal.

It was demonstrated by Gao et al. [24] that the tracking-error dynamics converge asymptotically to zero by employing a Lyapunov-based analysis. In their work, a suitable Lyapunov candidate function was constructed and proven to be monotonically decreasing, which in turn guaranteed the stability of the closed-loop system. The ASTIbot platform runs a complete vendor-provided ROS stack that already exposes all low-level base control, safety functions, and monitoring interfaces, including a local web server for manual teleoperation and diagnostics (Figure 6). Rather than replacing this existing infrastructure, our LLM-based control agent connects to the available topics, services, and frames, and operates purely at the command and interpretation layer. Building on the benchmarking results reported in Section 2—where Gemma3 demonstrated the best balance between command-generation quality, coverage, and reliability—this integration represents the intended “finish line” of our evaluation: an industrial AI agent Salazar and Vogel-Heuser [25] that can observe the live robot state, generate valid ROS2 commands, and provide high-level, natural-language status reports to end-users, all on top of an unmodified industrial robot stack.

Figure 6. Mobile robot local web server and ROS node stack used as the integration layer for the AI control agent: (a) Vendor-provided web interface for manual base control and real-time status monitoring on the ASTIbot platform. (b) RViz visualization of the full robot model and sensor configuration.

Table 5 summarizes the robot’s hardware platform, listing the main components (computer model, CPU, GPU, memory, operating system, and role) together with their technical specifications used in our experimental setup.

Table 5. Hardware components and specifications of the mobile robot platform.

Name	Description
NVIDIA Jetson Orin Nano	ARM Cortex-A78E processor 4 GB RAM LPDDR5 MicroSD card slot Gigabit Ethernet (10/100/1000BASE-T) 4 × USB 3.0 Type-A ports 1 × USB-C port
LsLiDAR M10 Scanner	TOF (Time-of-Flight) principle 2D scanning of the surrounding 360° environment Measuring frequency: 10 Hz Accuracy: ±3 cm Maximum range: 25 m
ASTRA Depth Camera	Distance range: 0.6–8 m RGB image resolution: 1920 × 1080 @ 30 fps Depth image resolution: 640 × 480 @ 30 fps Depth precision: ±3 mm @ 1 m
MD60 100W DC Motor	Chassis Wheel Independent Suspension Shock Absorber Damper MD60 Reduction ratio: 1:18 Motor: DC Brush, rated voltage 24 V, rated power 100 W Rated current: 6 A Rated speed: 175 rpm Rated torque: 55.7 kg·cm
Omnidirectional Wheels (Mecanum)	Material: stainless steel + rubber Weight: 700 g (per wheel) Diameter: 152.4 mm Width: 55.5 mm Roller dimensions: 20.4 mm × 41.3 mm
Power Supply	2 × VRLA Ultracell 12 V battery, 22 Ah (UCG22-12)
Total Weight	45 kg
Payload	∼30 kg
Dimensions (W × H × D)	73.5 cm × 62 cm × 100 cm

3. Results

Table 6 and Table 7 show examples of prompts used to test the agent. Our full implementation is openly available under the dedicated GitHub repository [26] for replicable benchmarking and evaluation.

Table 6. Representative ROS/ROS2 benchmark prompts by skill level and topic category.

Level	Category	Representative Prompt	Main Topics
Beginner	Basics	Difference between ROS and ROS2? When to use each?	DDS, real-time, lifecycle
Intermediate	Services	Topics vs. services + Python service example	request–response, srv, client–server
Advanced	Actions	Action server with feedback (Python 3)	goal, feedback, result, ActionServer
Expert	Security	Implement SROS2 for production system	DDS security, certificates, encryption
Expert	Migration	Best strategy: large ROS 1 → ROS 2 migration	ros1_bridge, gradual migration
Troubleshooting	Debugging	Nodes cannot see each other—troubleshoot	domain_id, discovery, firewall

Table 7. Natural-language robot queries with expected ROS topics and command examples.

Category	Representative User Query	Key Expected Topics	Typical ROS Command
Position	Where is the robot right now?	pose, position, location, odom, coordinates	ros2 topic echo /robot_pose
Battery	What is the battery level?	battery, percentage, voltage, charging	ros2 topic echo /battery_status
Status	What is the robot status?	status, state, activity, idle, navigating	ros2 topic echo /robot_status
Sensors	Are there any obstacles around the robot?	scan, lidar, obstacles, range, laser	ros2 topic echo /scan
Command Generation	Send the robot to the assembly station	navigate, goto, position, cmd_vel, service	ros2 service call /goto_position

Figure 7 shows the overall performance comparison. SmolLM2 achieved the fastest response time (5.22 s) and token generation speed (177.1 tokens/second), while Gemma3 and Gemma3-qat demonstrated the highest quality scores (84.6 and 85.6) and topic coverages (47.6% and 48.6%).

Figure 7. Overall performance comparison across LLMs. (a) Average response time, (b) generation speed (tokens/s), (c) quality score (0–100), (d) topic coverage percentage.

Following the results shown in Figure 8, we see that not all models are working with our environment; larger models such as DeepSeek-r1 and Qwen3 are giving timeouts as they cannot load in time to process our requests, but all the other models show 100% reliability.

Figure 8. Model reliability across evaluated LLMs, reporting the proportion of successful responses, errors, and timeouts in the ROS/ROS2 benchmarking environment.

Errors only appear when the model is not available or is not downloaded or when there is no connection with the containers, which in our case never occurs as we only test our machine on the local environment.

Performance varied significantly across ROS domains (Figure 9). Models generally performed well on basic tasks but showed varied expertise in specialized areas like Launch Files, Lifecycles, and Security. Llama 3.2 excelled in Basic Programming and Launch Files, and Gemma3-qat was the only one to cover 100% in Debugging, Real-time Systems, and TF.

Figure 9. Heatmap showing quality scores across different ROS/ROS2 categories. Darker colors indicate better performance.

Figure 10 summarizes the relative strengths of each model across all evaluation dimensions. Gemma3 and Gemma3-qat show the most balanced performance, combining high quality and coverage with competitive speed and reliability. SmolLM2 stands out for its superior speed but lags in coverage, while Llama 3.2 provides strong quality but slightly lower robustness in some specialized ROS domains.

Figure 10. Radar chart comparing the evaluated language models across multiple dimensions (quality, topic coverage, speed, reliability, and completeness).

Based on our weighted composite score, Gemma3-qat ranks first (62.7/100), followed by Gemma3 (62.1/100) (Figure 11). The breakdown reveals that the slight difference in scores comes from the quality of the response and the coverage of the topics, which is not a surprise, as the Quaternization version of the model offers comparable performance and quality but on a reduced memory footprint.

Figure 11. Model rankings based on weighted composite scores. (a) Overall ranking; (b) score breakdown by component.

Our results (Table 8) reveal three key patterns:

Table 8. Overall model ranking based on the weighted composite benchmark score.

Rank	Model	Quality	Coverage	Speed	Composite
1	Gemma3-qat	85.57	48.58	15.99	62.72
2	Gemma3	84.61	47.62	16.03	62.11
3	SmolLM2	62.5	14.48	86.04	58.42
4	Llama 3.2	80.76	34.74	18.35	57.36

• Domain-specific performance varies: All models showed strengths in certain areas, indicating specialization opportunities.
• Quality–speed trade-offs exist: Practitioners can choose models based on whether they prioritize response quality or generation speed.
• Model size does not guarantee adequacy: Although Llama 3.2 achieves a higher quality score (80.76) than SmolLM2 (62.5), their overall composite scores are comparable—57.36 for Llama 3.2 versus 58.42 for SmolLM2—because SmolLM2 performs better on the other weighted evaluation metrics. If our priority is speed and coverage of basic topics at the level of an operator, then the best option will be to use a small-sized model. Gemma3 and Gemma3-qat models are only 20% larger than Llama 3.2, but the difference in completeness response is almost double and the time of response is almost the same.

Our evaluation has several limitations:

• Automated quality scoring may not capture all aspects of expertise;
• Expected topics are subjective and may miss valid alternative approaches;
• Testing was conducted in English only;
• Evaluation focused on text responses, not executable code verification;
• Results may vary with different system prompts.

Based on these results, we moved to stage 2 for evaluation, where we built an agent able to answer general questions but also to generate ROS2 commands when needed to interact with the mobile robot, checking the ROS topics and services and interpreting the result. For this, we prepared a special system prompt to enable the agent access to the available ROS topics and services opened on our robot and specified an explicit format on which it could respond to work with our API. This application illustrated the practical integration of LLMs within a ROS2-based workflow, using the ASTIbot mobile robotic platform as a representative case study.

Figure 12 presents the command-level performance of the evaluated LLMs when interacting with the ROS2-based mobile robot. Gemma3 and Gemma3-qat achieve the highest command generation and interpretation scores, producing a larger proportion of valid commands and correct interpretations of ROS feedback. SmolLM2 remains competitive in terms of valid-command rate but shows lower overall quality, while Llama 3.2 offers strong but slightly less consistent behavior across the four metrics. These results confirm that models with better general ROS expertise also translate into more reliable closed-loop command execution.

Figure 12. Overall performance comparison across LLMs. (a) Command generation quality (scores, 0–100), (b) interpretation quality (scores, 0–100), (c) valid-command success rate (%), (d) valid interpretation success rate (%).

Figure 13 shows that command-generation performance is consistently higher than the interpretation performance across most task categories, particularly for navigation and diagnostics. Gemma3 and Gemma3-qat maintain the most stable command-quality scores, producing valid ROS2 commands even in more complex scenarios such as multi-step navigation goals or combined diagnostic checks. Interpretation scores, in contrast, vary more strongly between models and categories, reflecting differences in how reliably each model summarizes status messages, error codes, and log outputs. Interestingly, from a practical robotics perspective, this imbalance is acceptable, and even preferable: generating correct commands is safety-critical and difficult to post-correct automatically, while interpretation logic (e.g., mapping raw feedback into human-readable summaries or dashboard labels) can be iteratively refined in downstream components over time. In other words, a model with slightly lower interpretation scores but consistently high command-generation quality is more suitable for closed-loop control, since command templates are harder to remediate externally, whereas interpretation layers can be adjusted and specialized as the system evolves.

Figure 13. Category-wise command-generation performance of the evaluated LLMs. Each subplot reports per-category scores for (a) command generation quality (0–100) and (b) interpretation quality (0–100). Higher command-generation scores indicate a larger proportion of syntactically and semantically valid ROS2 commands, while interpretation scores reflect how accurately models summarize and explain ROS feedback across navigation, perception, diagnostics, and system-configuration tasks (%).

The radar comparison in Figure 14 highlights the relative strengths and weaknesses of the evaluated models at the command level. Gemma3 and Gemma3-qat form the largest and most regular profiles, confirming their balanced performance across all four dimensions: they combine high command-generation quality with consistently strong interpretation of ROS2 feedback. SmolLM2 exhibits a pronounced peak on the speed-related and valid-command axes but a noticeably smaller radius on quality-oriented metrics, indicating that it is well suited for fast, routine interactions but less reliable for complex, multi-step tasks. Llama 3.2 lies between these extremes, delivering high-quality commands and good interpretation performance, albeit with slightly lower robustness in some diagnostic and configuration scenarios. Overall, the radar plot reinforces the conclusion that models optimized for domain expertise (Gemma3/Gemma3-qat) provide the best trade-off for closed-loop control, whereas smaller models such as SmolLM2 are attractive when resource constraints and response times are dominant concerns.

Figure 14. Radar plot summarizing command-level performance of the evaluated LLMs. Each axis corresponds to one evaluation dimension (command-generation mean score, interpretation mean score, both valid scores, speed score, and consistency of the model). Larger, more regular polygons indicate models that achieve a more balanced trade-off between ROS2 command generation and robust interpretation of feedback across all metrics.

We describe the final scores in Equation (9) as a composite of the following formula:

$$S=0.25\times C+0.25\times I+0.3\times BV+0.2\times Sp,$$

(9)

where S is the composite score, C is the correct command generation, I is the correct interpretation of the results received after calling the ROS command, $BV$ is the validity of both the generated command and the interpreted results, and T is the time needed to complete the task. All these metrics are scaled between 0 and 100 in Equation (10):

$$\begin{array}{c}\hfill C={\displaystyle \frac{\sum C_{scores}}{N_{tests}}},I={\displaystyle \frac{\sum I_{scores}}{N_{tests}}}\\ \hfill BV={\displaystyle \frac{\sum C_{valid}+\sum I_{valid}}{N_{tests}}}\times 100\\ \hfill Sp=max(0,100-{\displaystyle \frac{\frac{\sum Totaltime}{N_{tests}}}{max\left(Totaltime\right)}}\times 100)\end{array}$$

(10)

Using the composite metric, we obtain the final command-level rankings summarized in Figure 15. Models that score highly on the “Both Valid” component dominate the top positions, confirming that simultaneous correctness of commands and interpretations is more decisive than raw speed or isolated quality metrics. Gemma3 and Llama 3.2 achieve strong scores across all four dimensions, while smaller models like SmolLM2 benefit from the speed weighting but are penalized when either commands or interpretations fail. This ranking underlines that, for closed-loop control of mobile robots, robustness of the entire interaction cycle is more critical than maximizing any single metric in isolation.

Figure 15. Final command-level rankings of the evaluated LLMs based on the composite score that combines command-generation quality (25%), interpretation quality (25%), the rate of interactions where both command and interpretation are valid (30%), and speed (20%). Models with higher composite scores achieve a more favorable trade-off between generating correct ROS2 commands, reliably interpreting feedback, and responding within practical latency bounds.

Figure 16 illustrates the web application that mediates interaction between human operators, AI control agents, and the simulated TurtleBot3 platform, developed by the Open Source Robotics Foundation, Mountain View, California USA. The home dashboard (Figure 16a) provides an overview of active ROS2 topics and services, allowing the operator to quickly verify connectivity and system status. The log view (Figure 16b) records all prompts, generated ROS2 commands, and feedback messages, which we use to analyze the agent’s behavior and identify typical failure modes. The chat interface (Figure 16c) is the main entry point for natural-language task specification, from which the agent derives structured commands for navigation and diagnostics. Finally, the location management page (Figure 16d) stores commonly used waypoints in the simulated environment, enabling repeatable experiments and rapid comparison of different models under identical task sequences.

Figure 16. Web application used to interact with the AI control agents and to monitor the simulated TurtleBot3 mobile robot: (a) Home dashboard summarizing the status of connected ROS2 topics, services, and agents. (b) Log view displaying recent interactions, generated commands, and execution feedback used to test the agent’s abilities. (c) Chat interface where operators issue natural-language instructions that are translated into ROS2 commands. (d) Location management page for defining and storing frequently used waypoints in the simulated environment.

In addition to the web-based control interface, we evaluated the agent in a full ROS2 navigation setup using the standard TurtleBot3 simulation stack. Figure 17 illustrates the simulated environment and the corresponding navigation view, where the LLM-based control agent issues high-level commands that are translated into concrete goals for the ROS2 navigation stack.

Figure 17. ROS2 TurtleBot3 navigation simulation environment: (a) 3D simulated warehouse-like world in Gazebo with the TurtleBot3 platform. (b) RViz2 view showing the full ROS2 navigation stack, including the robot model, map, local and global paths, and active navigation goals.

Deploying the agent in the simulation demonstrates that it can successfully execute basic navigation and monitoring tasks in a ROS2 environment (Figure 18 and Figure 19). The agent reports the robot status using simple natural-language descriptions, stores and reuses named locations on the map, and interacts with the ROS2 navigation stack to send goal positions and monitor progress. It also interprets sensor information via the prepared ROS topics and analysis tools, allowing it to answer operator queries about the surrounding environment with structured, quantitative statements (for example, reporting the number, type, and relative position of nearby obstacles).

Figure 18. ROS2 computation graph of the mobile robot stack: navigation, perception, and low-level control nodes and their communication links.

Figure 19. TF transform tree of coordinate frames used for localization, odometry, and motion control on the mobile platform.

Performing this on the real system is illustrated below.

Our LLM-based agent connects to this pre-existing stack, issuing ROS commands and interpreting feedback via natural language, without modifying the underlying control software.

Our experiments confirm that the proposed agent can both monitor and control the real platform. Figure 20 depicts the mapped laboratory environment used for validation, where the robot charging station and the CPS storage station are highlighted in green and red, respectively. The results show that the agent integrates successfully with the existing ROS stack and enables operators to supervise and command the platform through natural language via the web interface. By translating user requests into consistent, human-readable commands, the system increases the level of abstraction available to end-users and improves human–machine interaction.

Figure 20. RViz view of the mobile robot in our laboratory.

4. Conclusions

This paper introduced an industrial-level application of LLMs for the end-user to monitor and control a mobile robotic platform in natural language using advanced technologies, such as ROS and Docker Model Runner. We found that this study achieved its goals by demonstrating an end-to-end methodology for LLM-based interaction and control of industrial mobile robot platforms.

This work highlights the growing role of AI assistants in robotics development, where large language models can support design, debugging, and integration of complex systems. To be truly useful in this context, models must be evaluated in realistic, domain-specific settings rather than only on general-purpose benchmarks. For mobile robots and smart industry applications, ROS/ROS2 remains the core middleware Carreira et al. [27], so understanding how well different LLMs handle ROS/ROS2 concepts and workflows is essential.

In this study, we focused on four main questions: (i) which LLMs best understand ROS/ROS2 concepts and typical development tasks; (ii) how model performance changes across different difficulty levels, from simple API usage to multi-node integration; (iii) what quality–speed trade-offs arise when using different models in an interactive development loop; and (iv) which models are reliable enough to be considered for production use. Based on our experiments, we show that Gemma3 is a good fit if we talk about reliability, quality, and coverage of ROS-specific topics, and we discuss how these findings can guide both practitioners choosing AI assistants for ROS/ROS2 projects and researchers designing future LLMs tailored to robotics.

Future work will be focused on embedded deployment and benchmarking of the LLM systems on the mobile robotic platform and the evaluation of increasingly complex tasks and scenarios.