Reinforcement Learning for Real-Time Control Using Quanser Platforms: A Structured Narrative Review

Abstract

Reinforcement learning (RL) is increasingly used for real-time control of complex dynamical systems, but its practical performance must be evaluated under hardware constraints that are often simplified in simulation. This paper presents a comprehensive review of published RL-based control studies using the Quanser Aero, Aero 2, 3-DOF Helicopter, and Autonomous Vehicles Research Studio (AVRS) platforms. The reviewed studies are compared according to the RL algorithm, control objective, hardware configuration, implementation environment, and reported experimental performance. The synthesis shows that Aero and Aero 2 are used primarily for stabilisation, trajectory tracking, and energy-aware control, whereas the 3-DOF Helicopter provides a more demanding benchmark for adaptive and Actor–Critic methods under nonlinear and coupled dynamics. AVRS offers significant potential for vision-based and multi-agent RL; however, the available experimental literature remains limited. Across the reviewed comparisons, policy-gradient and Actor–Critic methods, including PPO and SAC, generally demonstrate greater adaptability and smoother continuous-control behaviour, while conventional controllers frequently retain advantages in steady-state accuracy, computational predictability, and safety verification. Nevertheless, no RL algorithm can be identified as universally superior because the published studies employ heterogeneous reward functions, reference trajectories, sampling rates, performance measures, and hardware configurations. Recurring limitations include sample inefficiency, simulation-to-hardware discrepancies, computational latency, safety constraints, and incomplete reporting of experimental protocols. The review therefore identifies standardised evaluation procedures, reproducible reporting, safety-aware RL, interoperable software interfaces, and higher-fidelity digital twins as priorities for future research. No new experimental data are generated; the contribution is a comparative synthesis of experimental evidence reported in the literature.

1. Introduction

In the swiftly advancing fields of control systems and reinforcement learning (RL), experimental testbeds are essential instruments for connecting theoretical design with practical applications. Hardware-in-the-loop (HIL) systems, which directly connect algorithms with emulations of physical plants, offer a dependable method for validating control strategies amidst practical uncertainties such as noise, delay, and nonlinear dynamics [1,2]. Such frameworks are essential in robotics, aerospace, and autonomous systems, where simulation-based methods frequently inadequately represent the intricacies of real-world situations [3]. Throughout the years, numerous benchmark configurations, such as inverted pendulums, rotary servos, and unmanned aerial vehicle test rigs, have been employed to evaluate reinforcement learning algorithms within realistic restrictions [4].

Among the commercially available laboratory control platforms, Quanser systems are widely used in engineering education and experimental control research because they combine modular hardware, sensing, real-time interfaces, and integrated software toolchains [5]. Quanser reports that its systems have been adopted by more than 2500 academic institutions worldwide (https://www.quanser.com/community/our-customers/, accesssed on 8 September 2026). This manufacturer-reported adoption figure demonstrates broad institutional reach, but it should not be interpreted as bibliometric evidence that Quanser constitutes a universal or de facto research standard. Accordingly, this review evaluates Quanser as a widely deployed family of academic testbeds rather than claiming that it dominates all hardware-based RL research. The Quanser Aero, Aero 2, Quanser 3-DOF Helicopter, and Autonomous Vehicle Research Studio (AVRS) are examples of systems that have found widespread use in the fields of education and advanced research [6,7], as shown in Figure 1. Quanser platforms are integrated with MATLAB/SimulinkR2025b through the QUARC real-time control environment, which supports real-time execution, code generation, parameter tuning, and deployment on supported targets [8]. Furthermore, MathWorks provides a documented reinforcement learning toolbox workflow for training and deploying an RL controller on the Quanser QUBE-Servo 2 pendulum through a Raspberry Pi interface [9]. Polzounov et al. introduced Blue River Controls, an OpenAI-Gym-compatible Python framework supporting common simulation and hardware interfaces for RL experiments on the QUBE-Servo 2 [10]. These sources directly substantiate the MATLAB/Simulink, Python 3.14.7, RL-toolbox, and Gym-style interface capabilities discussed here. Despite the growing body of research on reinforcement learning, the published experimental evidence concerning the use of Quanser platforms for RL-based control has not yet been comprehensively reviewed. Existing surveys generally address broad RL benchmarks, simulation environments, or instructional control systems, but provide limited synthesis of how Quanser testbeds have been used to evaluate RL controllers under physical hardware constraints. This article addresses this gap through a structured review of previously published studies. It does not report original experiments; instead, it critically synthesises the experimental configurations, control objectives, algorithms, findings, and implementation challenges documented in the reviewed literature. Accordingly, this review aims to:

• Synthesise the experimental findings reported in the selected literature according to the RL algorithm, control objective, hardware configuration, and reported performance.
• Critically assess recurring limitations related to hardware–software integration, computational latency, sample efficiency, safety, reproducibility, and simulation-to-hardware transfer.
• Identify research opportunities for improving Quanser-based RL evaluation through enhanced sensing, modular multi-agent configurations, standardised machine learning interfaces, safety-aware control mechanisms, and higher-fidelity digital twins.

For clarity, the term “experimental validation” in this review refers exclusively to experiments conducted and reported in the cited primary studies. The present article does not introduce a new RL controller, perform new hardware experiments, or generate an original experimental dataset. Its contribution is the comparative analysis and critical synthesis of the experimental evidence available in the published literature. Through this synthesis of published experimental studies, the paper evaluates the suitability of Quanser systems as adaptable benchmark platforms for RL-based real-time control and identifies the conditions under which findings obtained in simulation may be transferred to physical implementations. The principal contributions of this review are as follows:

• A comprehensive overview and structured classification of published research employing Quanser systems for reinforcement learning (RL)-based real-time control.
• A comparative synthesis of the RL algorithms, control objectives, hardware platforms, and experimentally observed outcomes reported in the reviewed primary studies.
• A critical assessment of recurring challenges related to hardware–software integration, computational latency, sample efficiency, safety constraints, reproducibility, and simulation-to-hardware transfer.
• A platform-level comparison of the sensing, actuation, software-interface, and multi-agent capabilities of the Quanser Aero, Aero 2, 3-DOF Helicopter, and Autonomous Vehicles Research Studio (AVRS) systems.
• A research roadmap for improving the suitability of Quanser platforms as reproducible RL benchmarks through advanced sensing, modular architectures, standardised software interfaces, safety-aware RL, and higher-fidelity digital twins.

Figure 1. Quanser testbeds: (a) Quanser Aero, (b) Aero 2, (c) Quanser 3-DOF Helicopter, (d) AVRS System.

Figure 1. Quanser testbeds: (a) Quanser Aero, (b) Aero 2, (c) Quanser 3-DOF Helicopter, (d) AVRS System.

This work was conducted as a structured narrative review. Relevant publications were identified through systematic searches of Scopus, Web of Science, and IEEE Xplore using combinations of the keywords “reinforcement learning,” “real-time control,” “Quanser,” “Quanser Aero,” “Aero 2,” “3-DOF Helicopter,” “AVRS,” “QDrone,” and “QBot.” The search encompassed publications available up to 2026. Studies were considered eligible if they investigated an RL-based control method using a Quanser platform in simulation, hardware-in-the-loop operation, or physical experiments and provided sufficient information regarding the learning algorithm, control objective, experimental platform, or reported performance. Publications unrelated to RL-based control or lacking an explicit connection to a Quanser system were excluded. Through this screening and selection process, 129 relevant research articles were identified, shortlisted, and considered in the preparation of this manuscript. The selected studies were subsequently compared according to the Quanser platform employed, RL algorithm, control task, implementation environment, and principal experimental findings. Owing to the substantial heterogeneity among the reviewed studies in terms of control tasks, performance metrics, and experimental configurations, the evidence was synthesised qualitatively rather than through a formal meta-analysis.

The remainder of this article is organised as follows: Section 2 describes the principal hardware, sensing, and software characteristics of the Quanser Aero, Aero 2, 3-DOF Helicopter, and AVRS platforms. Section 3 provides background on RL for control and discusses the distinction between simulation-based and hardware-based evaluation. Section 4 reviews and compares published RL studies using Quanser platforms. Section 5 summarises the principal capabilities of these platforms, whereas Section 6 examines their limitations and practical challenges. Section 7 presents future research recommendations. Section 8 concludes the review and summarises its principal findings.

• Note: All Figures containing product images were obtained from the official Quanser website and are reproduced solely for scholarly identification and discussion of the reviewed experimental platforms. In accordance with the Quanser Terms and Conditions (https://www.quanser.com/terms-conditions/, accesssed on 8 September 2026), which permit website content to be used as a resource when appropriate copyright notices, references, and credits are provided, the original source is acknowledged in each applicable figure caption. All product names, images, trademarks, and logos remain the property of Quanser Inc., and their inclusion in this review does not imply sponsorship or endorsement.

2. Overview of Quanser Platforms

This section evaluates the primary Quanser platforms (Quanser Inc., Markham, ON, Canada) utilised in reinforcement learning (RL) and control system validation [11,12,13], examining their advantages and drawbacks, and concludes with a comparative analysis [14,15,16].

2.1. Quanser Aero

Quanser Aero, a small rotor-based testbed depicted in Figure 2, interfaces seamlessly with MATLAB/Simulink [17,18,19] and simulates quadrotor dynamics with either one (pitch-only) or two (pitch plus yaw) degrees of freedom. The system is MIMO and nonlinear, enabling experimentation with sophisticated control strategies such as LQR, sliding-mode control, and versions of reinforcement learning [20,21,22].

Furthermore,

Table 1. Technical specifications of the Quanser AERO (https://www.quanser.com/wp-content/uploads/2017/01/Quanser_AERO_Product_Info_Sheet_v1.1.pdf, accesssed on 8 September 2026).

Parameter	Value
Device mass	3.6 kg
Device height (ground to top of base)	45 cm
Helicopter body mass	1.39 kg
Helicopter body length	48 cm
Base dimensions (W × L)	17.5 cm × 17.5 cm
Encoder resolution (in quadrature)	512 counts/rev
Pitch angle range	$75$° (±$37.5$°)
Yaw angle range	$360$°
Motor/propeller force–thrust constant	0.119 N/V
Motor/propeller torque–thrust constant	0.0036 Nm/V
Propeller diameter	12.7 cm
Propeller pitch	15.2 cm
Motor armature resistance	0.83 $\mathsf{\Omega }$
Motor current–torque constant	57.7 mN.m/A

summarises the physical and electromechanical specifications of the Quanser helicopter-based platform. The relatively low device mass (3.46 kg) and compact base dimensions make it suitable for laboratory-scale experiments. The high-resolution encoder (8192 counts/rev) enables precise measurement of pitch and yaw, critical for reinforcement learning validation [23,24,25]. The wide yaw range ($360$°) allows for full rotational studies, while the pitch range supports nonlinear maneuver testing. Motor and propeller constants provide a realistic force–torque relationship, ensuring experimental fidelity, though the modest torque–thrust constant limits scalability to larger UAV dynamics. Overall, these specifications balance safety, realism, and repeatability, making the platform well suited for both control education and reinforcement learning research [26].

The Quanser Aero platform is concise and cost-effective, simulates authentic aerial dynamics, and is supported by a robust ecosystem of software, curriculum, and community resources [25]. Nonetheless, its highly interdependent and nonlinear dynamics render modelling and control difficult, and the restricted onboard sensing limits studies necessitating high-speed manoeuvres or vision-based reinforcement learning.

2.2. Quanser Aero 2

The Quanser Aero 2 platform depicted in Figure 3 enhances the Aero system with improved modularity, superior abstraction layers, and multi-input functionality [24,25,26,27,28,29]. The integration of Simulink models with Python-based reinforcement learning libraries was demonstrated by Schäfer et al., who trained a PPO agent using Stable-Baselines3 and transferred the resulting policy to a physical Quanser Aero 2 system [30]. Moreover, multi-objective reinforcement learning can concurrently optimise energy efficiency and tracking performance [9,27]. The Quanser Aero 2 platform has numerous benefits, such as enhanced abstraction layers, adaptable interaction with Python and Simulink, and robust interoperability with contemporary reinforcement learning frameworks. Nonetheless, its effectiveness is still limited by hardware constraints regarding control precision and velocity, while the sample efficiency of physical reinforcement learning training frequently remains suboptimal.

Table 2. Technical specifications of the Quanser Aero 2 (https://www.quanser.com/products/aero-2/, accesssed on 8 September 2026).

Parameter	Value
Device Dimensions (D × W × H)	18 cm × 52 cm × 40 cm
Operating Space (D × W × H)	52 cm × 52 cm × 62 cm
Mass	4.7 kg
Pitch Angle Range	$90$° (±$45$° from horizontal)
Yaw Angle Range	$360$° continuous
Pitch Encoder Resolution	2880 counts/revolution
Yaw Encoder Resolution	4096 counts/revolution
Prop Thrust Constant	$5\times {10}^{-4}$ N·s/rad
Inertial Thrust Constant	0.042 Nm/A
Inertial Measurement Unit (IMU)	IIM-42652 compact six-axis MEMS device
Tri-axis Gyroscope Range	±500 dps
Tri-axis Accelerometer Range	±2 g

[28] delineates the characteristics of the Quanser platform, emphasising its mechanical dimensions, operational envelope, sensor capabilities, and motor parameters. The gadget, weighing 4.7 kg and measuring $18\mathrm{cm}\times 52\mathrm{cm}\times 40\mathrm{cm}$, is appropriate for laboratory settings, providing adequate operational space of $52\mathrm{cm}\times 52\mathrm{cm}\times 62\mathrm{cm}$. The pitch and yaw angle ranges ($90$° and $360$° continuous, respectively) provide comprehensive attitude control studies. High-resolution encoders (2880 counts/rev for pitch and 4096 counts/rev for yaw) provide precise state feedback, essential for reinforcement learning and control validation. The thrust constants ensure realistic actuation dynamics, while the onboard IIM-42652 six-axis MEMS IMU (TDK InvenSense, San Jose, CA, USA) offers dependable motion sensing. The tri-axis gyroscope (±500 dps) and accelerometer (±2 g) enhance the platform’s sensing accuracy, facilitating both traditional control and data-driven reinforcement learning applications.

2.3. Quanser 3-DOF Helicopter Model

The Quanser 3-DOF Helicopter model shown in Figure 4 is a tri-axial rotor system that facilitates roll, pitch, and yaw, affixed to a gimbaled pivot for accurate attitude control [29]. It has been widely utilised in observer design, controller validation, and fault-tolerant control research [31,32,33,34,35,36,37,38]. The main advantages consist of realistic attitude dynamics, high-fidelity encoders, and appropriateness for verifying sophisticated control schemes. Nonetheless, the platform exhibits specific constraints: it does not incorporate thruster-based translational motion characteristic of complete UAVs, its modelling may overlook aerodynamic influences and friction [12,38] and its relevance to multi-agent reinforcement learning contexts is restricted.

According to

Table 3. Technical specifications of the Quanser 3-DOF Helicopter (https://www.quanser.com/products/3-dof-helicopter/, accesssed on 8 September 2026).

Parameter	Value
Device Mass	6.2 kg
Device Height (ground to top of base)	45 cm
Device Length (counterweight to front of propellers)	127 cm
Base Dimensions (W × L)	17.5 cm × 17.5 cm
Pitch Encoder Resolution (quadrature mode)	4096 counts/rev
Travel Encoder Resolution (quadrature mode)	8192 counts/rev
Pitch Angle Range	±$32.0$°
Elevation Angle Range	$63.5$°
Travel Angle Range	$360$°

, the Quanser 3-DOF Helicopter system, a twin-rotor airborne testbed, is intended for laboratory-scale validation of sophisticated control and reinforcement learning algorithms. The device has a weight of 6.2 kg, a height of 45 cm, and an overall length of 127 cm from counterweight to rotors, while featuring a compact base footprint of $17.5\mathrm{cm}\times 17.5\mathrm{cm}$, rendering it appropriate for restricted laboratory settings. It features high-precision encoders, providing 4096 counts/rev for pitch measurement and 8192 counts/rev for journey measurement, facilitating meticulous tracking of angular positions. The technology facilitates a pitch angle range of ±$32$°, an elevation range of $63.5$°, and a complete $360$° journey range, thereby ensuring authentic aerial dynamics. The parameters together indicate that the platform is lightweight, small, and precise, offering extensive angular freedom, hence providing an exceptional experimental setting for secure and reproducible testing of control and reinforcement learning systems.

2.4. Quanser Autonomous Vehicle Research Studio (AVRS)

The Autonomous Vehicles Research Studio (AVRS), depicted in Figure 5, incorporates airborne and terrestrial vehicles, specifically the QDrone/QDrone 2 and QBot/QBot 2e, alongside a ground control station, localisation cameras, and networking equipment [13,25]. The QDrone features onboard computing capabilities such the Jetson Xavier NX or Intel Aero and vision sensors, and facilitates networked swarm operations [34,39,40]. AVRS, as a comprehensive platform, offers significant benefits: it enables multi-agent and vision-based reinforcement learning, is reasonably easy to deploy, and is ideally suited for educational and research applications. Nonetheless, the platform exhibits constraints, such as elevated costs, increased spatial demands, and the intricacies of sensor fusion and safety considerations, which may hinder scalability. Moreover, the reproducibility of investigations may be influenced by dependencies on wireless connectivity and localisation systems [16].

2.5. Comparison of Hardware, Sensors, and APIs

A comparative investigation of the Quanser Aero, Aero 2, 3-DOF Helicopter, and Autonomous Vehicles Research Studio (AVRS) identifies unique strengths and weaknesses in hardware, sensing capabilities, and software integration. The Aero and Aero 2 are compact and lightweight, ideal for desktop laboratory research, while the 3-DOF Helicopter is larger and heavier, providing greater motion ranges. Conversely, AVRS constitutes a comprehensive platform encompassing both airborne and terrestrial vehicles, necessitating greater specialised areas and resulting in elevated expenses. Aero offers fundamental high-resolution encoders for pitch and yaw measurements, but Aero 2 improves this configuration with an integrated six-axis MEMS IMU, facilitating more precise attitude estimation. The 3-DOF Helicopter features dual high-fidelity encoders for accurate pitch and travel angle measurements, but it does not possess contemporary onboard inertial sensor capabilities. AVRS provides the most comprehensive sensor suite, incorporating cameras, IMUs, depth sensors, and localisation systems, thus facilitating multi-agent vision-based reinforcement learning experiments.

Table 4. Comparison of Quanser testbeds (https://www.quanser.com/resource-type/technical-resources/, accesssed on 8 September 2026): hardware, sensors, and software interfaces.

Platform	Hardware Features	Sensors	API/Software Support	ROS/Multi-Agent Support
Quanser Aero	Lightweight (3.46 kg), compact base (17.5 cm × 17.5 cm), 1–2 DOF (pitch, yaw), dual-rotor testbed	High-resolution encoder (8192 counts/rev) for pitch and yaw	Seamless with MATLAB/Simulink, QUARC	Not natively ROS; primarily single-agent, suitable for introductory RL
Quanser Aero 2	Moderate size (4.7 kg), enhanced modular design, 2 DOF (pitch, yaw), improved mechanical structure	Encoders: pitch (2880 counts/rev) and yaw (4096 counts/rev); onboard six-axis MEMS IMU (IIM-42652), gyro (±500 dps), accel (±2 g)	MATLAB/Simulink, QUARC, Python APIs	Python-friendly; partial ROS integration possible; single-agent RL with extended workflows
Quanser 3-DOF Helicopter	Heavier (6.2 kg), larger footprint (127 cm length), 3 DOF (pitch, elevation, travel), gimbaled pivot for attitude control	Encoders: pitch (4096 counts/rev), travel (8192 counts/rev), and wide angular ranges; lacks modern onboard IMU	MATLAB/Simulink, QUARC	Not designed for ROS; focus on single-vehicle control validation; limited RL scalability
AVRS (QDrone/QBot)	Full-stack lab setup with aerial (QDrone/QDrone 2) and ground (QBot/QBot 2e) vehicles; requires larger workspace; onboard compute (Jetson Xavier NX/Intel Aero)	Rich sensor suite: vision cameras, IMUs, depth sensors, localisation cameras, and Wi-Fi-based swarm networking	MATLAB/Simulink, QUARC, Python, C++, ROS integration	ROS-native, multi-agent ready; supports swarm robotics, vision-based RL, and scalable experiments

shows all platforms integrating effortlessly with MATLAB/Simulink and Quanser’s QUARC real-time control software regarding APIs and ecosystem support. Aero and the 3-DOF Helicopter predominantly depend on this integration, whereas Aero 2 enhances capability through Python APIs, facilitating reinforcement learning workflows beyond MATLAB. AVRS offers exceptional flexibility, ensuring interoperability with MATLAB/Simulink, QUARC, Python, ROS, and C++ libraries, rendering it optimal for multi-agent robotics and swarm research. Aero and Aero 2 are optimal for introductory reinforcement learning and control experiments due to their cost-effectiveness and accessibility. The 3-DOF Helicopter offers highly realistic single-vehicle dynamics for controller validation, whereas AVRS connects academic testbeds with real-world multi-robot systems, facilitating state-of-the-art reinforcement learning validation in swarm and vision-based applications.

3. Reinforcement Learning for Control: Background

Reinforcement learning (RL) has developed into a robust paradigm for control, wherein agents acquire decision-making skills through interaction with an environment to optimise cumulative reward, generally articulated within the Markov Decision Process formalism [29,41]. This has facilitated the development of learning policies that manage complicated, unpredictable, or nonlinear dynamics without the need for explicit modelling in control systems. Initial applications concentrated on conventional control methodologies; however, recent advancements in deep reinforcement learning have empowered agents to function directly from high-dimensional sensory inputs across various domains, including robotics and industrial process control [42,43].

3.1. Analytical Taxonomy of RL-Based Control Studies

RL-based control studies vary considerably in their objectives, algorithms, and implementation environments. To support consistent comparison, this review classifies the literature along three complementary dimensions: (i) control task, (ii) RL algorithm paradigm, and (iii) training and deployment mode. This framework is used to organise the Quanser studies reviewed in Section 4.

The first dimension defines the control task. Attitude stabilisation regulates variables such as pitch, yaw, elevation, or travel and is evaluated using steady-state error, overshoot, rise time, settling time, and disturbance rejection. Trajectory tracking considers time-varying references or spatial paths and is assessed through tracking error, transient response, control effort, and constraint satisfaction. Energy-aware control additionally considers actuator usage or power consumption. Fault-tolerant control addresses disturbances, modelling uncertainty, actuator faults, saturation, and state constraints. Multi-agent coordination covers formation control, cooperative navigation, collision avoidance, swarm behaviour, and coordinated aerial–ground operation. The second dimension concerns the RL paradigm. Value-based methods, such as Q-learning and DQN, estimate state–action values and are mainly suited to discrete actions. Policy-gradient methods directly optimise stochastic or continuous policies. Actor–Critic methods combine a policy-generating actor with a value-estimating critic and include deterministic off-policy methods (DDPG and TD3), stochastic off-policy methods (SAC), and on-policy methods (PPO). Offline RL learns from fixed datasets, while multi-agent RL extends value-based or Actor–Critic learning to cooperative or competitive systems. The third dimension describes the training and deployment mode. Pure-simulation studies offer high throughput and low risk but do not establish physical performance. Sim2Real studies train policies in simulation before deploying them on hardware, either directly or with limited adaptation. Pure hardware online learning captures real dynamics but raises concerns regarding safety, wear, reset time, and sample efficiency. Hybrid training combines simulation pretraining with controlled hardware refinement, balancing safety and adaptation. These dimensions are related but distinct. The same task may use different algorithms and training modes; for example, stabilisation may employ simulation-trained PPO with hardware transfer or an adaptive Actor–Critic method trained directly on the testbed. Likewise, multi-agent coordination may use centralised training with decentralised execution in simulation or on AVRS hardware. Performance comparisons must therefore consider the control task, algorithm, and training mode together rather than attributing outcomes to the RL algorithm alone.

Table 5. Compact taxonomy of RL-based control studies.

Dimension	Category	Typical Methods	Objective	Key Evaluation Criteria
Control task	Stabilisation	PPO, SAC, DDPG, TD3	Regulate attitude or position	Error, overshoot, settling time
	Tracking	PPO, DDPG, TD3, SAC	Follow references or paths	Tracking error, effort, constraints
	Energy-aware	Multi-objective RL, PPO, SAC	Balance accuracy and energy	Energy–accuracy trade-off
	Fault-tolerant	Robust/constrained RL	Operate under faults or disturbances	Robustness, safety, saturation
	Multi-agent	MADDPG, MAPPO, QMIX	Coordinate vehicles	Scalability, communication, safety
Algorithm	Value-based	Q-learning, DQN	Discrete action control	Discretisation, scalability, bias
	Policy gradient	REINFORCE, PPO	Direct policy optimisation	Stability, interaction cost
	Deterministic Actor–Critic	DDPG, TD3	Continuous control	Exploration, critic bias, reuse
	Stochastic Actor–Critic	SAC	Entropy-based continuous control	Exploration, efficiency, tuning
	Offline RL	CQL, IQL, BCQ	Learn from fixed datasets	Coverage, unseen actions
	Multi-agent RL	MADDPG, MAPPO, QMIX	Cooperative or competitive control	Non-stationarity, credit assignment
Training	Simulation	Any RL method	Train without hardware	Model fidelity, hardware validation
	Sim2Real	PPO, SAC, DDPG, TD3	Transfer policy to hardware	Transfer loss, model mismatch
	Online hardware	Adaptive/integral RL	Learn directly on the testbed	Safety, wear, time, repeatability
	Hybrid	Pretraining + refinement	Simulate, transfer, and fine-tune	Robustness versus hardware cost

summarises the analytical framework. In Section 4, each reviewed study is classified according to these dimensions wherever the original publication provides sufficient information. Algorithms included as part of the broader RL control taxonomy are distinguished from algorithms with verified experimental evidence on the reviewed Quanser platforms. Therefore, inclusion of DQN, DDPG, TD3, SAC, offline RL, or multi-agent RL in this taxonomy does not imply that each method has already been physically validated on Aero, Aero 2, the 3-DOF Helicopter, or AVRS.

3.2. Evolution of RL Algorithms for Control

The technical evolution of reinforcement learning for control has progressed from tabular value-learning methods to deep value approximation, continuous-action actor–critic algorithms, and more recent safety-aware, offline, and multi-agent formulations. Early Q-learning methods were primarily applicable to problems with relatively small discrete state and action spaces and therefore scaled poorly to high-dimensional continuous-control systems. Deep Q-networks (DQN) addressed part of this limitation by combining Q-learning with deep neural function approximation, experience replay, and a target network [44]. Nevertheless, DQN still requires a discrete action set. For real-time control applications, discretising continuous quantities such as motor voltages, torques, or thrust commands may reduce control resolution, while using a fine discretisation can produce a rapidly expanding action space. Deep deterministic policy gradient (DDPG) extended deep RL to continuous-action problems by combining a deterministic actor with an off-policy critic and a replay buffer [45]. Its ability to reuse previously collected transitions makes it relevant to continuous control, where physical interaction may be expensive. However, DDPG can be sensitive to exploration noise, critic approximation error, hyperparameter selection, and overestimation of action values. Twin delayed deep deterministic policy gradient (TD3) was subsequently introduced to mitigate these limitations through clipped double-Q learning, delayed actor updates, and target-policy smoothing [46]. These mechanisms reduce critic overestimation and improve the stability of deterministic continuous-control learning.

Soft Actor–Critic (SAC) introduced an alternative off-policy approach based on a stochastic maximum-entropy objective [47]. By optimising both expected return and policy entropy, SAC encourages exploration while retaining the sample-reuse advantages of off-policy learning. Proximal policy optimisation (PPO) followed a different approach as an on-policy policy-gradient method that constrains policy changes using a clipped optimisation objective [48]. Its relative implementation simplicity and stable policy updates have contributed to its adoption in robotic and real-time control research. However, because PPO is on-policy and cannot reuse older interactions as extensively as DDPG, TD3, or SAC, it may require a larger number of newly collected transitions. The development of RL control has also expanded beyond single-agent unconstrained optimisation. Multi-agent Actor–Critic methods enable cooperative or competitive decision-making among interacting agents by incorporating centralised information during training and decentralised policies during execution [49]. Such approaches are relevant to cooperative navigation, vehicle formations, and the aerial–ground coordination tasks supported by multi-vehicle environments such as AVRS. Offline RL provides another important direction by learning policies from fixed datasets rather than requiring unrestricted online exploration. Conservative Q-learning, for example, regularises value estimation to reduce the selection of actions that are insufficiently represented in the training dataset [50].

This paradigm is particularly relevant to physical control systems, for which unsafe exploration, hardware wear, and experimental reset time can restrict online learning. More recent RL control research has placed greater emphasis on stability, safety, and integration with established control architectures. For example, Zhang et al. developed a stability-constrained RL-based coordinated-control strategy for the integrated active front-steering and direct-yaw-control problem of a vehicle chassis [51]. This study illustrates the broader evolution from unconstrained reward optimisation to RL formulations that explicitly incorporate closed-loop stability and coordinated actuator behaviour. Although this application does not employ a Quanser platform, it provides relevant context for the development of safety- and stability-aware RL in real-time control. Within the Quanser literature examined in this review, PPO and adaptive Actor–Critic formulations currently have clearer physical-platform evidence than DQN, DDPG, TD3, SAC, multi-agent Actor–Critic methods, and offline RL. The latter algorithms are nevertheless included because they represent mainstream developments in RL-based control and provide important reference points for identifying gaps in Quanser-based experimental validation. Their inclusion as algorithmic context should not be interpreted as evidence that each method has already been experimentally validated on the reviewed Quanser platforms. This taxonomy is used in Section 4 to distinguish algorithms that are broadly established in RL control from algorithms for which physical Quanser-based evidence is identified. Inclusion of an algorithm in the taxonomy does not imply that it has already been experimentally validated on every reviewed Quanser platform.

Recent advances in adjacent learning-based perception tasks also illustrate the increasing importance of multimodal and generative models for autonomous systems. For example, WaterCycleDiffusion integrates visual–textual information with a diffusion model to improve underwater images, demonstrating how enhanced visual perception can support downstream scene understanding and decision-making in visually degraded environments [52]. Although this method is not based on reinforcement learning, such perception-oriented developments are relevant to vision-enabled autonomous platforms, where the reliability of sensory observations can influence the state information supplied to an RL controller.

3.3. Simulation-Based, Sim2real and Hardware-Based Training

The validation modes considered in this review are distinguished as follows: Simulation-only evaluation executes both the learned controller and plant model in a virtual environment without a real-time requirement. Software-in-the-loop (SIL) evaluation connects the implemented policy software to a simulated plant and is primarily used to verify interfaces and algorithm execution. Hardware-in-the-loop (HIL) evaluation executes part of the control system on real-time computing or interface hardware while the controlled plant remains simulated. Hardware inference deploys a previously trained policy on the physical Quanser plant without updating its parameters. Hardware fine-tuning begins with a simulation-trained policy and subsequently updates it through limited physical interaction, whereas direct online hardware learning trains or adapts the policy on the physical plant from the outset. Hybrid validation combines two or more of these stages. These modes expose controllers to progressively different sources of uncertainty and should therefore not be treated as equivalent evidence of real-world performance. Research in reinforcement learning for control is mostly divided between simulation-based investigations and hardware-focused applications. Simulation-centric reinforcement learning facilitates swift experimentation and algorithmic refinement, leveraging low-risk, high-throughput settings. Benchmark suites, including the continuous control tasks in MuJoCo and OpenAI Gym, along with frameworks like Safe-Control-Gym that combine safety-certified control with reinforcement learning, have facilitated methodological advancements while guaranteeing reproducibility. Nonetheless, simulation frequently inadequately represents real-world intricacies, prompting an increasing focus on hardware-in-the-loop and real-robot reinforcement learning research. Benchmark studies conducted on physical robots reveal issues related to hyperparameter sensitivity and reproducibility inherent to actual configurations [53]; analogous initiatives encompass real-robot offline reinforcement learning datasets that assist in identifying discrepancies between simulation and reality [54].

Benchmarking and reproducibility are crucial for achieving significant advancements in control-oriented reinforcement learning. The community has reacted with open-source platforms and standardised protocols, including RL Unplugged for offline reinforcement learning, SURREAL for manipulation benchmarking, and Open RL Benchmark for comparative experiment tracking. These initiatives facilitate the transparent assessment of RL methodologies and expedite technology validation. Recent research underscores the evaluation of reinforcement learning (RL) algorithms within real-time restrictions, acknowledging that computational latencies might impair control efficacy in time-sensitive applications, and advocates for the comparison of RL with model-based control methodologies utilising standardised robotic platforms. In process control, reinforcement learning (RL) has progressed, with benchmark studies assessing various RL algorithms (e.g., fourteen versions across six benchmark settings) to determine their appropriateness for particular control scenarios [55]. Moreover, frameworks in operations research, such as Fabricatio-RL, seek to connect scheduling management issues with clearly specified reinforcement learning interfaces and reproducibility standards [56]. Ultimately, although simulation provides scalability and safety, hardware implementations deliver realism and reveal essential restrictions like delays, noise, and reproducibility. Dependable benchmarks and standardised frameworks, encompassing simulation, hybrid, and physical domains, are essential for the objective progression of RL control research.

Figure 6 presents five reinforcement learning (RL) training and deployment pathways for simulation and Quanser hardware. Pathway A represents a simulation-only approach, in which the RL algorithm is trained, optimised, and evaluated entirely within a simulated environment [44,45,46,47,48,57]. Pathway B illustrates the Sim2Real workflow, where a policy trained in simulation is transferred to the physical platform for real-time inference. Pathway C extends this process through hardware fine-tuning, allowing the transferred policy to adapt using data obtained from the physical system. Pathway D depicts direct online learning, beginning with a safety-oriented initialisation and proceeding through physical interaction and continuous policy updates. Finally, Pathway E presents a hybrid approach that combines simulation, software-in-the-loop (SIL), and hardware-in-the-loop (HIL) pretraining with hardware inference and controlled adaptation. These pathways enable RL algorithms to be evaluated and refined across simulated and physical environments using Quanser testbeds, including the Aero, Aero 2, 3-DOF Helicopter, and Autonomous Vehicles Research Studio (AVRS), thereby connecting theoretical development with safe and practical implementation.

4. Review of RL-Based Research Contributions Using Quanser Products

The hardware platforms offered by Quanser have greatly aided in the use of reinforcement learning (RL) in control research. This section examines RL-based contributions across four notable Quanser testbeds—Aero, Aero 2, 3-DOF Helicopter, and AVRS—alongside comparative analyses of algorithmic performance and a synthesis of experimental findings. The Quanser Aero platform has been a foundational bed for RL research in aerial control. DRL algorithms such as PPO and SAC have been tested for pitch–yaw stabilisation and trajectory tracking in both simulation and hardware implementations. Comparative evaluations show that PPO offers faster rise time and adaptability, while classical controllers like LQR achieve superior steady-state accuracy on Aero 2 in 1-DOF settings [58]. Although Aero lacks rich onboard sensing, it remains valuable for rapid RL prototyping and education. Examining the reinforcement learning-based research on Aero 2 focusses on multi-objective reinforcement learning, especially regarding energy-efficient control architectures. A multi-objective reinforcement learning framework was established to optimise pitch tracking and energy consumption, utilising Aero 2 in 1-DOF mode, demonstrating considerable trade-offs affected by reward-shaping factors [59].

4.1. Study-Level Evidence Base

Moreover, research on problem formulation has shown that meticulous design of state-action spaces and incentive structures can significantly improve real-world reinforcement learning performance and sample efficiency when implementing policies on Aero 2 hardware [60,61,62]. Moreover, the 3-DOF Helicopter presents a more intricate and tightly interrelated dynamic environment. Researchers have utilised Actor–Critic reinforcement learning approaches, derived from heuristic dynamic programming, to attain real-time adaptive control, showcasing effective attitude stabilisation and disturbance rejection on a physical platform [62,63,64]. These model-free methodologies provide online optimisation among uncertainty, demonstrating the helicopter’s value as a demanding reinforcement learning benchmark. Lastly, AVRS provides hardware and software capabilities for investigating vision-based control, cooperative navigation, decentralised decision-making, and multi-agent RL using aerial and ground vehicles. However, the existence of these platform capabilities should not be interpreted as evidence that complete multi-agent RL pipelines have already been extensively validated on AVRS hardware. The reviewed literature predominantly addresses enabling components or conventional coordination methods, while reproducible experimental evidence concerning multi-agent RL performance remains limited. However, comparatively few peer-reviewed studies currently report complete RL training and hardware evaluation using the integrated AVRS environment. Accordingly, the available evidence is not yet sufficient for a robust comparison of RL algorithms on this platform. The following subsection examines the technical and practical factors that may have contributed to this limited publication record.

Table 6. Study-level summary of RL applications on Quanser platforms.

Ref.	Year	Platform	Task	RL Method	Training Mode	Reported Outcome
[65]	2018	Aero, 2-DOF	Pitch–yaw stabilisation	RL controller	Simulation	Feasible coupled control; quantitative hardware result NR.
[57]	2022	2-DOF Helicopter	Constrained tracking	Robust Actor–Critic RL	Simulation + hardware	Improved tracking while satisfying state constraints.
[39]	2023	QDrone with load	Robust trajectory tracking	Fuzzy RL ensemble	Sim2Real	The 3D RMSE decreased to 0.0343 m and 0.0524 m under wind.
[40]	2023	QDrone with load	Load-position control	DDPG with homogeneity transformation	Sim2Real	Reported 96% success and 0.0253 m 3D RMSE.
[28]	2024	Aero 2, 1-DOF	Pitch tracking	PPO vs. MPC/LQR	Simulation + hardware fine-tuning	Mean error: $3.80$° simulation, $3.75$° hardware, and $3.2$° after fine-tuning.
[30]	2024	Aero 2, 1-DOF	Reference tracking	PPO	Sim2Real	Best simulation mean deviation was $4.6$°, demonstrated by hardware transfer.
[66]	2025	2-DOF Helicopter	Quantised constrained tracking	Integral Actor–Critic RL	Simulation + hardware	Tracking achieved under input quantisation and state constraints.

Note: NR indicates that a directly comparable numerical result was not reported. Differences in tasks, metrics, and experimental settings prevent direct algorithm ranking.

lists the individual studies that satisfied the review’s eligibility criteria and for which a Quanser-based RL control application could be identified. Unlike the analytical taxonomy in Table 5, which classifies the wider RL control field, this table distinguishes the physical platform, control task, algorithm, training mode, hardware-validation status, and principal reported outcome of each eligible study. The evidence is concentrated on Aero/Aero 2 and 2-DOF Helicopter systems, while peer-reviewed hardware results for the 3-DOF Helicopter and AVRS remain scarce. PPO, DDPG, and adaptive or integral Actor–Critic formulations have received some physical validation, whereas comparable Quanser experiments involving DQN, TD3, SAC, offline RL, and multi-agent RL were not identified in the reviewed corpus. Because the studies employ different tasks, trajectories, reward functions, training budgets, and performance measures, their numerical results should be interpreted as study-specific evidence rather than as a controlled ranking of algorithms.

Real-time performance was also examined as a comparison dimension; however, the reviewed Quanser RL studies report timing information inconsistently. Although several studies demonstrate successful closed-loop hardware execution, most do not provide sufficient information on control-loop frequency, mean or worst-case policy-inference time, communication delay, timing jitter, processor specifications, or missed deadlines. Hardware implementation is therefore interpreted here as evidence of practical closed-loop feasibility rather than formal confirmation of deadline-compliant real-time operation. This distinction is important because a controller operating at 10 Hz and one consistently satisfying a 1 kHz deadline represent fundamentally different computational capabilities. Future studies should report the nominal sampling period, processor and software configuration, end-to-end sensing-to-actuation latency, mean and maximum inference time, jitter distribution, and missed-deadline rate over repeated experiments. The limited availability of these quantities prevents a reliable quantitative timing comparison and represents a significant reporting and benchmarking gap in current Quanser-based RL research.

The reviewed studies differ considerably in their experimental specifications. In [65], the 2-DOF Aero stabilisation problem used pitch–yaw states and motor-voltage actions, with LQR as the baseline, although hardware trials, sampling frequency, safety provisions, and code availability were not reported. The constrained Actor–Critic method in [57] used helicopter states and tracking errors to generate two motor inputs, with barrier-Lyapunov constraints providing an explicit safety mechanism during physical validation. For the QDrone suspended-load experiments, ref. [39] employed an ensemble of RL policies and reported three-dimensional RMSE values of 0.0343 m and 0.0524 m under nominal and wind-disturbance conditions, respectively, whereas the DDPG formulation in [40] reported a 96% success rate and 0.0253 m RMSE. In the Aero 2 study [28], PPO generated continuous motor commands from measurable pitch information; comparison with MPC and LQR produced mean deviations of $3.80$° in simulation, $3.75$° after initial hardware transfer, and $3.2$° following hardware fine-tuning. Study [30] reported five PPO training runs of 500,000 steps and a best simulated mean deviation of $4.6$°, followed by physical deployment. Finally, ref. [66] applied integral Actor–Critic RL to constrained tracking with quantised motor inputs and reported successful hardware validation. Several publications did not disclose reward coefficients, trial counts, control frequencies, complete safety procedures, or public code and datasets; these omissions are identified as “not reported” rather than inferred and represent an important reproducibility limitation of the existing evidence.

4.2. Factors Limiting Published RL Experiments on AVRS

Although AVRS provides a technically capable environment for vision-based, multi-agent, and heterogeneous aerial–ground RL, relatively few peer-reviewed studies report complete RL experiments using the platform. Several interrelated factors may explain this scarcity. First, the entry barrier is higher than for compact, single-device testbeds such as Aero or Aero 2. A representative AVRS configuration requires multiple aerial or ground vehicles, an indoor localisation system, wireless communication infrastructure, a ground-control workstation, safety equipment, software integration, and a dedicated experimental area. Consequently, fewer laboratories can reproduce an identical physical configuration. Second, an AVRS experiment requires the simultaneous integration of several technical layers, including vehicle dynamics, onboard sensing, localisation, wireless communication, perception, inter-agent coordination, and real-time control. Failure or uncertainty in any of these layers can affect the RL policy, making system debugging and the attribution of performance considerably more difficult than in a single-agent, low-degree-of-freedom testbed. Multi-agent RL further increases the dimensionality of the state and action spaces, introduces non-stationarity as multiple agents learn or operate simultaneously, and requires the management of communication delays and synchronisation errors. Third, hardware-based RL is sample-intensive, whereas aerial experiments involve limited battery duration, repeated charging, manual resets, safety supervision, and the risk of vehicle damage during exploratory actions. These factors reduce experimental throughput and encourage researchers to conduct most policy training in simulation, followed by only limited hardware deployment. Fourth, reproducibility is complicated by differences in the number and type of vehicles, sensor calibration, localisation layout, communication network, software version, safety constraints, and physical workspace used by individual laboratories. Results may therefore be difficult to reproduce without detailed reporting and access to a comparable studio configuration. Finally, the AVRS ecosystem has continued to evolve in terms of its vehicles, onboard computing capabilities, digital twins, application programming interfaces (APIs), and product nomenclature. The time required to establish a laboratory, develop stable RL environments, conduct safe multi-agent experiments, and complete the peer-review process may create a publication lag relative to the platform’s technological development.

The available AVRS-related literature does not yet provide a mature body of directly comparable multi-agent RL experiments. Published studies more commonly demonstrate individual enabling components—such as aerial–ground coordination, leader–follower formation, vision-based navigation, localisation, or communication—than complete multi-agent RL pipelines involving centralised training, decentralised execution, hardware policy deployment, and repeated comparative evaluation. Consequently, the principal finding of this review is not that a particular multi-agent RL algorithm has been validated successfully on AVRS, but that the platform possesses relevant experimental capabilities while substantive and reproducible hardware evidence for multi-agent RL remains limited. Product functionality should therefore not be presented as equivalent to demonstrated algorithmic performance. Three technical bottlenecks are particularly important. First, wireless latency, jitter, packet loss, and unequal communication delays can cause agents to act on stale observations or neighbouring-agent states, degrading formation accuracy and violating the timing assumptions used during centralised training. Second, AVRS localisation depends on coordinated sensing and reference-frame calibration; occlusion, measurement noise, update delay, and temporary pose loss can propagate through the policies of several agents and increase collision or coordination risk. Third, collaborative scheduling becomes more difficult as the number and heterogeneity of vehicles increase because task allocation, shared-resource access, action synchronisation, episode reset, battery availability, and conflict resolution must be coordinated simultaneously. These factors interact: delayed communication can corrupt scheduling decisions, while localisation uncertainty can invalidate assigned trajectories. Future AVRS studies should therefore report end-to-end latency and jitter, packet-loss rate, localisation error and dropout frequency, synchronisation error, task-completion rate, collision or safety-intervention frequency, and performance as the number of agents increases. These factors should be interpreted as plausible and mutually reinforcing explanations derived from the platform requirements and the broader hardware-RL literature, rather than as independently verified causes. The current scarcity of publications therefore appears to reflect barriers related to accessibility, system integration, experimental throughput, and reproducibility rather than a lack of research potential.

4.3. Cross-Comparison of RL Algorithms

The reviewed studies investigate different RL families under substantially different experimental conditions. Value-based, policy-gradient, and Actor–Critic methods differ in their treatment of discrete or continuous actions, sample reuse, exploration, and policy updating; however, these methodological differences do not establish that one algorithm is inherently better suited to a particular Quanser platform. The available studies employ different control objectives, state and action definitions, reward functions, reference trajectories, training budgets, sampling rates, hardware configurations, baselines, and performance measures [67,68,69,70]. Moreover, direct controlled comparisons among DQN, PPO, SAC, DDPG, and TD3 on an identical Quanser system were not identified in the reviewed evidence. Consequently, the reported results should be interpreted as study-specific demonstrations rather than evidence of a general algorithm hierarchy. A reliable ranking would require evaluating the algorithms on the same hardware, task, training budget, safety constraints, random seeds, and evaluation protocol [71,72]. The current literature therefore supports identifying algorithm-specific opportunities and experimental gaps, but does not permit a platform-independent conclusion regarding the superiority of DQN, PPO, SAC, or other RL families [73].

5. Capabilities of Quanser Platforms for RL

Quanser platforms provide significant benefits for reinforcement learning (RL) research through the integration of high-fidelity sensors and real-time hardware-in-the-loop (HIL) capabilities. Their devices are meticulously designed for repeatability and deterministic behaviour, facilitating realistic physical testing of algorithms inside a hardware-in-the-loop research environment, as detailed in a lengthy whitepaper [74]. Furthermore, the open-architecture design enables modular reconfiguration of hardware configurations for various experiments, while high-resolution encoders, IMUs, vision sensors, and localisation systems on platforms such as QBot, AVRS, and Aero 2 significantly improve sensing accuracy. Quanser’s educational platforms, such as QBot and the Controls and Dynamics Lab, are designed for accessibility and adaptability, catering to both classroom instruction and advanced research. These technologies are augmented by comprehensive course-ware, scaffolding learning modules, and virtual twins, facilitating swift implementation and hybrid learning experiences [75,76,77,78]. Entire laboratories can be established or replicated in a virtual environment within hours, providing scalable infrastructure appropriate for both educational and research purposes.

Quanser systems exhibit exceptional adaptability with many software ecosystems. They provide support for MATLAB/Simulink and QUARC for real-time control, in addition to supporting Python APIs and ROS interoperability for control tasks and algorithm implementation [79,80,81,82]. The QBot platform offers explicit multi-language support (MATLAB, Simulink, Python, C++, ROS), facilitating easy connection with RL toolkits and expeditious control prototyping [83,84,85]. Complementary digital twins enhance accessibility by enabling experimentation with virtual representations that accurately replicate actual platforms [86,87,88,89]. Quanser reports that its products have been used by more than 2500 academic institutions worldwide [90], indicating substantial adoption in engineering education and laboratory research. However, institutional adoption does not establish the relative publication share of Quanser compared with other experimental platforms. Within the narrower corpus reviewed in this article, Quanser systems provide reusable hardware configurations and common software interfaces that can support cross-institutional replication. Their actual contribution to reproducibility nevertheless depends on complete reporting of hardware versions, model parameters, sampling rates, software configurations, reward functions, trained policies, and evaluation protocols. Figure 7 provides a comparative overview of the reported reinforcement learning (RL) capabilities across selected Quanser platforms, considering simulation, hardware inference, hardware fine-tuning or online learning, and multi-agent RL. The Aero and Aero 2 platforms exhibit evidence of both simulation-based training and hardware inference, while hardware-level adaptation has been reported only to a limited extent and no clear multi-agent RL implementation has been identified. The 2-DOF Helicopter is comparatively well established for simulation, hardware inference, and adaptive or integral RL [91,92], whereas multi-agent learning is not applicable to this single-system platform. Although simulation studies are reported for the 3-DOF Helicopter, evidence of RL deployment on physical hardware remains limited, and hardware fine-tuning has not been established. For QDrone and AVRS, simulation-based RL is reported, but evidence concerning QDrone hardware inference, online adaptation, and AVRS-based multi-agent RL remains scarce. Overall, the matrix distinguishes mature areas of RL application from capabilities that require further experimental validation, thereby identifying opportunities for future research on real-time adaptation, hardware-based learning, and cooperative multi-agent control.

These capabilities make Quanser platforms useful for repeatable proof-of-concept evaluation; however, their laboratory scale, constrained dynamics, standardised environments, and safety-protected operation limit the extent to which successful results can be generalised directly to full-scale or industrial systems. Their principal role should therefore be understood as intermediate validation between simulation and application-specific field testing.

6. Limitations and Challenges

Although Quanser systems have shown to be valuable in reinforcement learning (RL) validation, they encounter some intrinsic limits and obstacles that need to be resolved for wider applicability.

6.1. Simulator-to-Reality Gap: Evidence from Quanser Studies

The simulation-to-reality gap arises when an RL policy trained using a mathematical model or simulated plant encounters physical dynamics, measurements, and constraints that were omitted or inaccurately represented during training [30]. On Quanser platforms, this discrepancy is not limited to generic noise and delays; it can be attributed to specific electromechanical and aerodynamic characteristics, including motor and propeller dead zones, encoder quantisation, cross-axis aerodynamic coupling, friction, actuator saturation, mechanical clearance or backlash, unmeasured states, and communication latency [1,93]. These effects modify the state transitions and responses to control actions experienced by the physical system and may therefore invalidate the relationships learned by the policy in simulation. Motor dead zones and nonlinear thrust characteristics are particularly important near an equilibrium point. Simulated motor models commonly assume that thrust varies continuously or linearly with the commanded voltage. In the physical system, however, small commands may produce little or no rotor response because they must overcome motor friction and the effective thrust threshold [1,93]. Consequently, an RL policy trained using an idealised command-to-thrust relationship may generate frequent low-amplitude actions that influence the simulated plant but fail to move the physical system. Following transfer, this discrepancy may result in steady-state error, oscillatory corrective actions, or delayed responses.

Encoder quantisation introduces a different form of discrepancy. Simulated environments generally provide continuous angle and angular-velocity states, whereas a physical controller receives discrete encoder counts and may derive angular velocity through numerical differentiation or state estimation. The resulting staircase-like observations and derivative noise may cause rapid switching between neighbouring actions, particularly when a policy is sensitive to small changes in the observed state. For example, the encoder specifications of the reviewed platforms correspond to finite angular increments rather than continuous measurements [26]. Nevertheless, the practical effect of quantisation also depends on the sampling frequency, filtering method, observation normalisation, and policy architecture; therefore, it cannot be determined from encoder resolution alone. Aerodynamic coupling is especially significant in the 2-DOF Aero/Aero 2 and 3-DOF Helicopter configurations [1,93]. Variations in the speed of one rotor can influence multiple rotational axes through reaction torque, asymmetric airflow, and cross-axis forces. A policy trained using decoupled or linearised pitch and yaw dynamics may therefore produce unintended motion along an unmodelled axis when deployed on the physical hardware. This challenge is further amplified in the 3-DOF Helicopter because its pitch, elevation, and travel dynamics are strongly coupled. Mechanical clearance, joint flexibility, and backlash may similarly create a direction-dependent region in which a change in the control command does not immediately produce a corresponding measurable motion. Because these mechanical characteristics can vary with assembly condition, component wear, payload, and operating direction, they may not be represented consistently by a nominal simulation model. These platform-specific factors operate through different mechanisms but may occur simultaneously. Motor dead zones and mechanical clearance primarily distort the relationship between control actions and physical motion; encoder quantisation alters the observations supplied to the policy; and aerodynamic coupling introduces unmodelled interactions among state variables. Their combined influence explains why a controller may remain stable following transfer while still exhibiting changes in tracking error, response time, oscillatory behaviour, or control effort [28].

For Quanser systems, this gap is associated with imperfect representations of friction, rotor thrust, aerodynamic coupling, sensor resolution, state-estimation errors, actuator nonlinearities, communication delays, and safety limits. Because the significance of these effects varies across platforms, the simulation-to-reality gap should not be treated solely as a general limitation of RL. Studies involving the Quanser Aero demonstrate how plant-model fidelity can influence subsequent controller transfer. Dyvik et al. [1] showed that accurate modelling of the Aero requires explicit consideration of friction and centripetal forces, whereas Segerstrom et al. [6] employed parameter optimisation and model validation to improve the agreement between simulated and physical Aero responses. Although these studies did not directly evaluate RL policies, they identified modelling discrepancies that were particularly relevant to RL because a policy may exploit inaccuracies in the simulated dynamics during training. Consequently, a policy trained using an idealised thrust relationship or an incomplete friction model may generate actions that are effective in simulation but produce substantially different transient or steady-state behaviour on the physical testbed.

A direct RL example was reported by Schäfer et al. [28], who trained a proximal policy optimisation (PPO) controller in simulation and subsequently evaluated it on a physical Quanser Aero 2 operating in a 1-DOF configuration. Their implementation revealed an important difference between the two environments: the simulation provided direct access to the pitch angle and angular velocity, whereas the physical platform measured the pitch angle but did not directly measure the angular velocity. To improve transferability, the RL state was reformulated using measurable pitch information and pitch variation instead of relying on an unavailable velocity measurement. The simulation-trained agent achieved an average pitch deviation of approximately $3.80$° in simulation and $3.75$° during its initial evaluation on the physical system. Subsequent training on the physical platform reduced the reported average deviation to approximately $3.2$°; however, this improvement required an additional 500,000 interaction steps and more than 15 h of hardware operation. This example demonstrates that successful initial transfer does not eliminate the simulation-to-reality problem, as observation design, sensor availability, hardware interaction time, and post-transfer adaptation remain important practical considerations. The results reported by Schäfer et al. [28] permit an aggregate, although not factor-by-factor, assessment of policy transfer. The average pitch deviation changed from approximately $3.80$° in simulation to $3.75$° during the initial hardware evaluation. For this specific metric, the absolute difference between the simulation and hardware results was $0.05$°, corresponding to approximately $1.3\%$ of the simulation value. Thus, the initial transfer did not result in a measurable degradation in average tracking performance. Subsequent training on the physical system reduced the average deviation from $3.75$° to approximately $3.2$°, representing an improvement of about $0.55$°, or $14.7\%$, relative to the initial hardware result. However, this improvement required an additional 500,000 hardware-interaction steps and more than 15 h of physical operation. These findings quantify the aggregate transfer outcome for a specific Aero 2 configuration; they do not isolate the individual contributions of motor dead zones, encoder quantisation, aerodynamic coupling, or mechanical clearance. The reviewed literature does not report controlled RL ablation experiments in which these physical factors are introduced individually while the same trained policy is evaluated under otherwise identical conditions. Consequently, separate quantitative effects or percentages cannot currently be assigned to their respective influence on transfer performance. Encoder resolution provides a physical measure of observation quantisation, but it does not independently determine the resulting RL tracking error. Similarly, the cited modelling studies identify friction, nonlinear thrust characteristics, and coupled dynamics without quantifying their individual effects on RL-policy performance. The absence of factor-isolated experimental evidence represents an important research gap and motivates the development of standardised simulation-to-hardware evaluation protocols in which each source of discrepancy is varied independently. The manifestation of the simulation-to-reality gap differs across the reviewed platforms, as shown in

Table 7. Quanser-specific simulation-to-reality gaps and mitigation strategies.

Gap Source	Platform	Transfer Effect	Mitigation	Evidence
Motor dead zone and nonlinear thrust	Aero, Aero 2, 3-DOF Helicopter	Ineffective small actions, tracking error, and oscillations	Identify dead zones, model nonlinear thrust, and randomise parameters	Addressed in control studies; isolated RL evidence is unavailable.
Encoder quantisation and missing states	All platforms; Aero 2 transfer study	Noisy observations, poor velocity estimates, and action switching	Use filtering, observers, measurement history, and realistic sensor models	State redesign is shown in [28]; related methods appear in [51,52,66].
Aerodynamic coupling	Aero/Aero 2 (2-DOF) and 3-DOF Helicopter	Unintended cross-axis motion and altered response	Use coupled nonlinear models, domain randomisation, and residual adaptation	Modelling is reported in [1,66]; RL ablation evidence is unavailable.
Clearance, backlash, and friction	Aero and 3-DOF mechanical joints	Delayed motion, hysteresis, and limit cycles	Identify mechanical effects, randomise parameters, and adapt online	Friction is modelled in [1]; RL-specific evidence is limited.
Sensor and communication latency	AVRS and embedded systems	Delayed actions, instability, and poor coordination	Randomise latency and use timestamped observations or delay-aware policies	Matched AVRS simulation–hardware results remain scarce.
Different simulated and measured states	Aero 2 and sensor-limited systems	Policy requires states unavailable on hardware	Align observation spaces and simulate realistic sensors	Directly demonstrated in [28].
Residual transfer mismatch	Aero 2 and other platforms	Remaining hardware tracking error	Apply limited hardware fine-tuning or residual learning	Fine-tuning improved performance in [28], but required extensive hardware use.

.

First, experimental system identification and model validation can be used to estimate friction, thrust coefficients, inertial parameters, and cross-axis coupling before policy training [1,6]. Second, the simulated observation space should reproduce the signals that are genuinely available on hardware. Schäfer et al. [28], for example, reformulated the PPO state to avoid the dependence on an angular-velocity measurement that was unavailable on the physical Aero 2. Third, dead-zone- and quantisation-aware control studies [51,52,66,84] indicate that actuator thresholds and discrete inputs or measurements should be represented explicitly or addressed through adaptive compensation. Although not all of these studies employ RL, they identify physical effects and compensation mechanisms that can be incorporated into RL environments. Fourth, domain randomisation can vary friction, thrust gain, dead-zone width, sensor offset, quantisation, time delay, payload, and coupling coefficients during training so that the learned policy is not optimised for a single nominal model. Fifth, limited physical-system fine-tuning can compensate for residual mismatch, as illustrated in [28], although its interaction time and hardware-wear costs should be reported. Finally, physics-informed or experimentally calibrated digital twins can combine nominal plant equations with parameters obtained from measured Quanser responses. At present, however, the reviewed literature provides stronger evidence for system identification, observation alignment, and post-transfer adaptation than for controlled comparisons of domain randomisation or residual learning on Quanser hardware. Future studies should therefore compare these strategies using identical policies, test trajectories, random seeds, and pre-defined transfer metrics.

The manifestation of the simulation-to-reality gap differs across the reviewed platforms. For Aero and Aero 2, the principal sources include thrust approximation, friction, aerodynamic coupling, sensor resolution, and unmeasured states. For the 3-DOF Helicopter, the gap is intensified by strongly coupled nonlinear motion, input constraints, and interactions among the pitch, elevation, and travel dynamics. In AVRS, the transfer problem additionally encompasses vision and localisation errors, wireless latency, synchronisation uncertainty, variations among individual vehicles, and changes in the physical environment. These platform-specific differences explain why a policy-transfer procedure demonstrated using a single-axis Aero configuration cannot be assumed to generalise directly to a 3-DOF system or a multi-agent AVRS experiment.

Taken together, the available evidence indicates that Quanser Sim2Real evaluation should report at least: (i) simulation and hardware tracking error before adaptation; (ii) relative transfer degradation; (iii) success and safety-violation rates; (iv) changes in rise time, settling time, overshoot, and control effort; and (v) the number and duration of physical interactions required for adaptation. Studies should also document the dead-zone model, encoder resolution, observation filtering, actuator saturation, sampling rate, delay distribution, coupling parameters, and mechanical configuration. Reporting these quantities would permit future reviews to distinguish zero-shot transfer effectiveness from performance obtained after hardware-specific fine-tuning and would support quantitative attribution of transfer error to individual platform characteristics.

6.2. Limited Scalability for Multi-Agent Learning

The Autonomous Vehicles Research Studio (AVRS) endorses swarm robotics; nonetheless, its scalability is limited by communication bottlenecks, spatial requirements, and the intricacy of localisation systems. Experiments in multi-agent reinforcement learning using extensive teams of robots are challenging to replicate reliably due to networking delays and synchronisation problems that diminish robustness [94]. Consequently, AVRS can authenticate small-scale cooperative techniques but it encounters difficulties in facilitating large-scale swarm trials. These scalability constraints are particularly relevant to AVRS. In this platform, the difficulty of RL experimentation arises not only from the number of agents but also from the need to coordinate localisation, wireless communication, onboard perception, safety supervision, and real-time control across heterogeneous vehicles. The resulting setup and replication burden provides one explanation for why the number of complete AVRS-based RL experiments remains smaller than the literature associated with compact single-agent Quanser testbeds [65,86,87,88,89,91].

6.3. Hardware and Safety Constraints

Reinforcement learning requires exploration, which can damage actuators, sensors, and structures. Long-term training on real hardware increases wear and tear, limiting experimental cycles and maintenance costs. Safety limitations must also be applied to prevent accidents and overheating, which might limit learning space and algorithm performance [95,96,97].

6.4. Academic Scope and Engineering Transferability

Quanser platforms provide an intermediate validation stage between simulation and full-scale deployment. They enable safe, repeatable, and observable experiments, but their results should not be interpreted as direct evidence of industrial readiness. Aero and Aero 2 are fixed-base systems with one or two dominant rotational degrees of freedom. They do not fully represent free-flight translation, position–attitude coupling, payload variation, battery depletion, structural flexibility, environmental disturbances, wide operating ranges, or realistic failure consequences. The 3-DOF Helicopter captures nonlinear and cross-axis dynamics but remains mechanically constrained within bounded pitch, elevation, and travel ranges. These platforms therefore demonstrate RL control feasibility under controlled physical conditions rather than robustness in unrestricted flight. AVRS adds mobile aerial and ground vehicles, onboard perception, wireless communication, and multi-agent operation. However, experiments are generally performed indoors with structured environments, controlled lighting, assisted localisation, few vehicles, and short missions. This differs from real deployments involving weather, uncertain terrain, large communication networks, component degradation, regulatory constraints, and uncontrolled agents.

Standardised testbeds improve reproducibility but may cause policies to overfit to specific hardware, sensor calibrations, workspaces, trajectories, or disturbance ranges. Moreover, laboratory metrics such as tracking error, rise time, and cumulative reward do not fully capture reliability, certification, maintainability, energy endurance, fault coverage, or long-term operation. Quanser experiments should therefore be regarded as proof of concept or intermediate validation. Stronger transfer claims require varied parameters and disturbances, unseen payloads and trajectories, trials across multiple hardware units, long-duration operation, realistic outdoor or industrial testing, and comparison with established safety-certified controllers.

6.5. Computational Challenges in Real-Time Deployment

Real-time RL deployment faces computational bottlenecks due to the high processing requirements of deep neural policies [98,99], particularly when running on embedded processors within Quanser hardware. Latency in policy inference, especially under complex tasks or multi-agent settings, can degrade control performance and hinder real-world applicability [100,101,102,103].

6.6. Constructive Solutions for the Limitations

Therefore, to mitigate these challenges, several constructive solutions can be proposed:

• Bridging the Simulator-to-Reality Gap: Methods including domain randomisation, system identification, and physics-informed digital twins can mitigate the discrepancies between simulated and physical settings.
• Improving Multi-Agent Scalability: Implementing lightweight communication protocols, modular testbed expansions, and edge-computing technologies can improve the scalability of AVRS for extensive multi-agent reinforcement learning studies.
• Reducing Hardware Stress: Hybrid training methodologies, wherein policies are initially established in simulation and subsequently refined on hardware, might mitigate wear and tear. Safety-conscious reinforcement learning and confined exploration frameworks can enhance hardware protection during the learning process.
• Addressing Computational Demands: Utilising optimised neural architectures, hardware accelerators (such as GPUs and TPUs), and real-time inference frameworks helps mitigate latency challenges in embedded reinforcement learning deployment.

In general, despite the fact that these problems provide obstacles to scaling RL validation on Quanser platforms, ongoing advancements in simulation quality, scalable system design, and algorithmic efficiency have the potential to considerably improve their long-term sustainability as RL benchmarking and validation platforms.

6.7. Recommended Experimental Protocol for Reproducible and Safe RL Validation

A five-stage protocol is recommended to improve the safety and reproducibility of Quanser-based RL experiments. Rather than prescribing a universal algorithm, it defines the minimum procedures and reporting requirements for training, deployment, and evaluation.

• Hardware characterisation and baseline control: Calibrate the platform and verify encoder offsets, actuator directions, command limits, sampling rate, communication delay, and emergency-stop operation. Use safe, low-amplitude excitation to estimate friction, dead zones, thrust gains, coupling, and delays. Implement PID, LQR, or MPC as both a performance baseline and, where appropriate, a backup controller.
• Environment and reward specification: Report the observation vector, action limits, reward terms, termination conditions, reference signals, control frequency, filtering, normalisation, and safety constraints. Simulated observations should reproduce the noise, resolution, delay, filtering, and state availability of the physical system. Individual reward components should be provided to allow reconstruction.
• Simulation training and tuning: Tune hyperparameters in simulation using a documented search procedure and multiple random seeds. Report interactions, episodes, wall-clock time, convergence criteria, and variability. For Sim2Real transfer, vary model parameters, sensor noise, latency, friction, actuator gains, and payload during training.
• Safety-gated hardware deployment: Before deployment, test the policy under uncertainty, disturbances, noise, delay, saturation, and unseen references. Begin hardware trials with reduced state and action limits. Action clipping, rate limits, software end stops, watchdog timers, automatic termination, and a backup controller should operate independently of the RL policy.
• Evaluation and reproducibility reporting: Evaluate using predefined, unseen trajectories and disturbances. Compare against at least one conventional controller under identical conditions. Report repeated-trial statistics, tracking error, overshoot, settling time, control effort, constraint violations, safety interventions, inference time, hardware interaction time, and failures. Where possible, release code, configurations, model parameters, random seeds, trained policies, and hardware/software versions.

Table 8. Practical starting points for simulation-based tuning before Quanser deployment.

Method	Key Parameters	Simulation Starting Point	Main Hardware Risk
PPO	Learning rate, rollout, batch size, clipping, entropy, GAE	$\alpha =3\times {10}^{-4}$; $\gamma =0.99$; $\lambda =0.95$; clip $=0.2$; batch = 64–256	Large updates or high entropy may cause abrupt actions.
SAC	Learning rates, replay buffer, batch size, target update, entropy	$\alpha =3\times {10}^{-4}$; $\gamma =0.99$; $\tau =0.005$; batch $=256$	High entropy may increase unsafe exploration.
DDPG	Actor/critic rates, exploration noise, replay buffer, target update	Actor $={10}^{-4}$; critic $={10}^{-3}$; $\tau =0.005$; batch $=128$–256	Critic errors or noise may produce saturated commands.
TD3	Learning rates, target noise, policy delay, replay buffer	$\alpha =3\times {10}^{-4}$; delay $=2$; $\tau =0.005$; batch $=128$–256	Noise must remain within physical action limits.
DQN	Action discretisation, exploration decay, replay buffer, target update	$\alpha ={10}^{-4}$; buffer $={10}^5$–${10}^6$; gradual exploration decay	Coarse actions reduce resolution; fine actions increase complexity.

provides practical simulation-based starting points. These values are not universal settings and should be tuned for the selected platform, task, control frequency, and action scale.

The values in Table 8 are initial simulation settings commonly used in the RL literature [44,45,46,47,48] and should not be interpreted as platform-independent optimal values. Final settings must be selected according to the sampling rate, normalised observation and action ranges, control objective, and safety envelope of the specific Quanser system. Hyperparameter selection should be completed primarily in simulation, and the search space, selection criterion, random seeds, and final configuration should be reported. In addition to algorithm-specific tuning, safe and reproducible hardware deployment requires systematic management of common experimental failures. Encoder miscalibration, noisy state estimates, actuator dead zones and saturation, communication loss, control-loop overruns, localisation dropout, battery variation, and unintended policy oscillations should be detected through calibration checks, signal-validity monitoring, watchdog timers, and automatic termination criteria. Safety constraints should operate independently of the learned policy through conservative action and slew-rate limits, software end stops, state-safety envelopes, and a verified backup controller that assumes control when timing, communication, tracking, or physical limits are violated. All safety interventions, failed trials, hardware resets, and recovery actions should be recorded and reported, because excluding them can overstate policy reliability and weaken experimental reproducibility.

7. Future Recommendations and Directions

The future advancement of Quanser testbeds may concentrate on augmenting their function as dependable validation platforms for reinforcement learning (RL).

7.1. Platform-Specific Digital Twins and Benchmarking

A key research direction is the development of sophisticated digital-twin models that accurately reproduce real-world system dynamics. The integration of domain randomisation into these models could further reduce the simulation-to-reality gap by exposing RL agents to variations in system parameters, measurement noise, disturbances, and sensor latency during training. The evidence reviewed in Section 6.1 further indicates that simulation-to-reality mitigation strategies should be tailored to the specific characteristics of each Quanser platform. For Aero and Aero 2, digital twins should incorporate experimentally identified friction, thrust nonlinearities, sensor resolution, and actuator dynamics. For the 3-DOF Helicopter, simulations should accurately represent cross-axis coupling, state constraints, and input saturation. For AVRS, domain randomisation should additionally account for variations in camera and lighting conditions, localisation uncertainty, wireless communication latency, inter-vehicle variability, and environmental changes. Evaluation protocols should report performance both before and after hardware adaptation, thereby distinguishing zero-shot transfer performance from improvements achieved through subsequent fine-tuning on the physical system. Collectively, these platform-specific measures could enhance policy robustness and support safer, more reliable, and more reproducible transfer from simulation to physical hardware.

Improvement priorities should reflect the limitations of each Quanser platform. For Aero and Aero 2, standardised calibration should identify motor dead zones, thrust nonlinearities, friction, coupling, encoder offsets, and delay, enabling unit-specific digital twins and common stabilisation and tracking benchmarks. For the 3-DOF Helicopter, benchmark models should represent cross-axis coupling, saturation, quantisation, clearance, and state constraints, supported by an independent safety controller. For AVRS, timestamped communication and configurable injection of latency, packet loss, localisation noise, and pose dropout should support reproducible multi-agent tasks involving formation, waypoint allocation, collision avoidance, and aerial–ground coordination. Across all platforms, calibrated models, versioned RL environments, baseline policies, safety configurations, and common evaluation metrics should be released to enable meaningful multi-laboratory replication. These platform-specific pathways should produce measurable research outputs. Aero and Aero 2 studies should report calibrated simulation–hardware prediction error and policy-transfer degradation; 3-DOF Helicopter studies should report simultaneous three-axis tracking accuracy, constraint violations, and safety interventions; and AVRS studies should quantify latency, packet loss, localisation error, task completion, collision frequency, and scalability with agent count. Releasing the corresponding identification data, environment configurations, baseline controllers, trained policies, random seeds, and raw experimental logs would enable transparent comparison and multi-laboratory replication.

Building on the system-identification and model-validation evidence reported for Quanser platforms [1,6], the digital-twin roadmap could be extended through a predict–explain–corroborate workflow. First, an RL policy would generate control actions from synchronised hardware observations while recording the associated states, rewards, timing information, and constraint activity. Second, the policy behaviour would be interpreted by identifying the observations and learned relationships that most strongly influence its actions. Third, these interpretations would be independently examined using experimentally calibrated plant models, digital-twin predictions, known physical relationships, operating limits, and targeted hardware tests. This corroboration stage is particularly important because successful simulation-to-hardware transfer [28,30] does not establish that the policy has learned physically meaningful or causal relationships. Agreement between policy interpretations and physical-model evidence would strengthen confidence in the controller, whereas disagreement could indicate spurious correlations, inadequate model fidelity, or unsafe policy dependence. Consistent with the broader role of digital twins in aerial-control validation [64], future evaluations should report control performance together with model–hardware residuals, constraint violations, interpretation consistency, and the computational overhead of the corroboration process. This workflow is proposed as a future validation architecture rather than an experimentally established capability of the reviewed Quanser literature.

7.2. Sensing, Communication, and AVRS Multi-Agent Reproducibility

Another crucial focus is enhancing sensor accuracy and transmission bandwidth. Enhanced-resolution inertial measurement units, low-latency motion capture systems, and sophisticated vision sensors could substantially improve the precision of state estimation, facilitating more complex reinforcement learning challenges. Enhancing wireless communication infrastructure within platforms such as the Autonomous Vehicles Research Studio (AVRS) would mitigate latency and synchronisation challenges, hence facilitating more dependable multi-agent reinforcement learning studies. Modular configurations tailored for swarm and cooperative robotics would facilitate scalability, allowing researchers to expand single-agent platforms into investigations of collective intelligence with consistent setups. To reduce the barriers currently limiting AVRS-based RL publications, future work should provide standardised RL environments, reference reward functions, calibrated digital-twin models, common evaluation scenarios, and detailed reporting templates for vehicle, sensor, localisation, communication, and software configurations. Public release of training code, trained policies, random seeds, safety constraints, and hardware-deployment protocols would further improve reproducibility and enable meaningful comparison across laboratories.

7.3. Open Software, Safety, and Computational Infrastructure

In addition to hardware issues, it is essential to reconcile open-source APIs with software toolchains orientated towards machine learning. Although Quanser presently accommodates MATLAB/Simulink, Python, and ROS, the incorporation of standardised open-source APIs in conjunction with machine learning libraries like TensorFlow, PyTorch, and RLlib will facilitate wider use within research communities. Open interfaces would guarantee reproducibility and reduce obstacles for researchers familiar with data-driven machine learning workflows. The integration of safety-aware reinforcement learning should be prioritised. Integrating runtime assurance frameworks and hardware fail-safes will enable reinforcement learning agents to explore efficiently while mitigating hazardous activities that could compromise hardware integrity. Methods such as shielded exploration, constrained policy optimisation, and automated emergency shutdown systems will not only protect testbeds but also facilitate the investigation of safety-critical reinforcement learning, a progressively significant research domain.

The future development of Quanser platforms can be enhanced through further integration with edge AI and Industry 4.0 testbeds. Integrating lightweight AI accelerators like NVIDIA Jetson or Google Coral into Quanser hardware will facilitate low-latency decision-making, consistent with Industry 4.0 frameworks of cyber–physical systems and intelligent manufacturing. Furthermore, cloud-based reinforcement learning facilitates the delegation of computationally demanding training tasks, enabling extensive policy optimisation and transfer learning that can subsequently be effectively implemented on local Quanser systems. Ultimately, the expansion of the Autonomous Vehicles Research Studio (AVRS) to encompass multi-agent reinforcement learning would facilitate swarm coordination, distributed autonomy, and collaborative robotics, thereby offering a realistic setting to validate novel multi-robot RL algorithms in educational environments and provide preliminary evidence for subsequent application-specific industrial testing. Collectively, these enhancements would establish Quanser systems as innovative and robust platforms for the progression of RL research in educational and academic settings.

8. Conclusions

This article reviewed published research on RL-based real-time control using the Quanser Aero, Aero 2, 3-DOF Helicopter, and Autonomous Vehicles Research Studio platforms. The reviewed literature indicates that these systems provide useful environments for investigating RL under physical constraints that are commonly absent or simplified in simulation, including sensor noise, actuator limits, computational delay, nonlinear dynamics, communication constraints, and safety requirements. Aero and Aero 2 have primarily supported stabilisation, trajectory-tracking, and energy-aware control studies; the 3-DOF Helicopter has enabled evaluation under strongly coupled nonlinear dynamics; and AVRS provides an extensible environment for aerial–ground, vision-based, and multi-agent research. The synthesis also identifies limitations that prevent straightforward comparison and reproduction of published results. These include heterogeneous reward functions and evaluation measures, incomplete reporting of hardware and software configurations, limited availability of implementation code and trained policies, sample inefficiency, safety constraints, and discrepancies between simulation and physical deployment. Consequently, the available evidence should be interpreted as a collection of platform- and task-specific findings rather than proof that a particular RL algorithm is universally superior. Future research should prioritise standardised experimental protocols, complete reporting of training and hardware configurations, open and interoperable software interfaces, safety-aware RL mechanisms, higher-fidelity digital twins, and common benchmarks that permit comparisons across algorithms and platforms. The reviewed evidence should therefore not be interpreted as demonstrating industrial readiness. Quanser experiments primarily establish algorithmic feasibility under controlled laboratory conditions, and further validation on full-scale systems, under broader disturbances and operational constraints, remains necessary before engineering deployment. This review does not report original experiments; its contribution is a structured synthesis and critical assessment of experimental evidence reported in the existing literature.