Efficient Parameter Search for Chaotic Dynamical Systems Using Lyapunov-Based Reinforcement Learning

Abstract

This study applies reinforcement learning to search parameter regimes that yield chaotic dynamics across six systems: the Logistic map, the Hénon map, the Lorenz system, Chua’s circuit, the Lorenz–Haken model, and a custom 5D hyperchaotic design. The largest Lyapunov exponent (LLE) is used as a scalar reward to guide exploration toward regions with high sensitivity to initial conditions. Under matched evaluation budgets, the approach reduces redundant simulations relative to grid scans and accelerates discovery of parameter sets with large positive LLE. Experiments report learning curves, parameter heatmaps, and representative phase portraits that are consistent with Lyapunov-based assessments. Q-learning typically reaches high-reward regions earlier, whereas SARSA shows smoother improvements over iterations. Several evaluated systems possess equation-level symmetry—most notably sign-reversal invariance in the Lorenz system and Chua’s circuit models and a coordinate-wise sign pattern in the Lorenz–Haken equations—which manifests as mirror attractors and paired high-reward regions; one representative is reported for each symmetric pair. Overall, Lyapunov-guided reinforcement learning serves as a practical complement to grid and random search for chaos identification in both discrete maps and continuous flows, and transfers with minimal changes to higher-dimensional settings. The framework provides an efficient method for identifying high-complexity parameters for applications in chaos-based cryptography and for assessing stability boundaries in engineering design.

1. Introduction

Chaotic systems exhibit strong sensitivity to initial conditions and parameter variations, resulting in highly complex and unpredictable dynamical behavior. Owing to their inherent nonlinearity and structural richness, such systems have been extensively investigated in physics, engineering [1], and cryptography [2]. The nonlinear and high-dimensional nature of these systems further complicates comprehensive exploration using classical analytical approaches.

The identification of parameter regimes that induce chaos is not merely an academic exercise; it is a critical component for several practical applications. In chaos-based cryptography, the security and robustness of encryption algorithms, particularly for tasks like image encryption [3], depend directly on the complexity and unpredictability of the underlying chaotic system [4]. Discovering parameters that generating high-quality pseudo-random sequences and ensuring resilience against attacks [5]. Similarly, in physics and engineering, the ability to efficiently map chaotic regions is vital for both system design and complexity analysis. This capability is necessary for tasks such as inducing chaos to enhance mixing in fluid dynamics [6], or conversely, avoiding chaotic regimes to ensure the stability of mechanical structures or electrical systems. The primary challenge in these applied tasks is to meet realistic performance demands. This often requires locating sparse, narrow, or intricately structured chaotic regions within vast, high-dimensional parameter spaces [7]. For such tasks, traditional discovery methods, such as comprehensive grid-based parameter sweeps or plotting multi-parameter bifurcation diagrams, suffer from the curse of dimensionality. These brute-force approaches scale poorly, quickly becoming computationally prohibitive and inefficient as the number of system parameters increases, rendering the exhaustive exploration of the complete parameter space infeasible [8].

Several of the studied models exhibit equation-level symmetry that organizes both the attractor geometry and the reward landscape. In the Lorenz equations, sign-reversal invariance produces mirror attractors with identical Lyapunov estimates. Chua’s circuit, with an odd piecewise-linear nonlinearity, shows the same property around the origin. The Lorenz–Haken model preserves a coordinate-wise sign pattern, leading to paired structures in phase-space projections.

To address these limitations, reinforcement learning (RL) has emerged as a promising alternative for navigating parameter spaces in dynamical systems [9]. By formulating the search for chaotic regimes as a sequential decision-making task, reinforcement learning agents can adaptively select parameter combinations that maximize a formal measure of chaos. The largest Lyapunov exponent (LLE), a widely accepted indicator of chaotic divergence, is utilized as a continuous scalar reward signal [10]. Compared to brute-force methods, reinforcement learning offers adaptive sampling and enhanced convergence toward chaos-inducing parameter sets, thereby improving both efficiency and scalability [11].

This comparison reveals a clear research gap: while reinforcement learning has been successfully applied to control chaotic systems, its use as an efficient, autonomous discovery tool for mapping vast, high-dimensional parameter spaces remains underexplored. This gap motivates the following research question (RQ): Can an RL agent that uses the largest Lyapunov exponent (LLE) as its reward efficiently and autonomously discover sparse chaotic regimes in high-dimensional nonlinear dynamical systems, and does it offer a quantifiable advantage over traditional brute-force methods? An LLE-based RL framework is adopted because it aligns the agent’s learning objective (reward maximization) with a fundamental indicator of chaos, thereby enabling adaptive, sample-efficient exploration of parameter landscapes.

Model-free reinforcement learning algorithms [12], such as Q-learning and SARSA (state–action–reward–state–action) [13], are employed to autonomously discover chaotic behavior in a broad spectrum of classical and hyperchaotic systems. Q-learning, an off-policy method, estimates the optimal action value function independently of the current policy, promoting the fast exploration of high-reward regions [14]. In contrast, SARSA updates estimates based on the agent’s actual behavior, favoring more stable and conservative learning. Comparatively integrating these two strategies allows for a systematic evaluation of the exploration–stability trade-off in chaotic parameter identification. The framework identifies parameter settings that yield chaotic behavior, supported by numerical simulations and attractor plots.

The principal contributions of this study are summarized as follows:

(i)

A unified reinforcement learning-based framework for discovering chaotic parameters in multiple nonlinear dynamical systems;

(ii)

The integration of Lyapunov exponents as continuous reward signals for training reinforcement learning agents;

(iii)

An empirical comparison between Q-learning and SARSA in terms of convergence, reward profiles, and attractor richness across various systems.

This paper is organized as follows: Section 2 presents the reinforcement learning framework, including the SARSA and Q-learning algorithms. Section 3 describes the proposed methodology and experimental setup. Section 4 provides the results and discussion. Section 5 discusses these results in the context of prior work and practical applications. Section 6 concludes the paper and suggests future research directions.

2. Reinforcement Learning for Chaotic Parameter Exploration

This section presents the RL framework designed to identify chaos-inducing parameter regions in nonlinear dynamical systems. System parameters are encoded within the state–action space, and the LLE is employed as a scalar reward signal [15]. Two representative RL algorithms, Q-learning and SARSA, are introduced, followed by the formulation of the reward function. A comparison with traditional numerical techniques highlights the advantages of the proposed learning-based strategy.

2.1. Formal Problem Definition

The parameter search as a discrete-time Markov Decision Process (MDP). The agent’s goal is to find an optimal policy ${\pi }^{*}$ that maximizes the expected discounted sum of future rewards (the return), $G_t$, defined as

$$G\left(t\right)=\sum _{i=t}^{\infty }{\gamma }^{i-t}r(i+1).$$

(1)

where $r_t$ is the reward at time t, and $\gamma \in [0,1]$ is the discount factor.

The action-value function $Q^{\pi }(s,a)$ is the expected return for taking action a in state s and thereafter following policy $\pi$. The optimal action-value function $Q^{*}(s,a)$ must satisfy the Bellman optimality equation [16]:

$$Q^{*}(s,a)=\mathbb{E}\left[r_{t+1}+\gamma \underset{a^{\prime }}{max}{Q}^{*}(s_{t+1},a^{\prime })\mid s_t=s,a_t=a\right].$$

(2)

This equation forms the theoretical basis for the Q-learning algorithm. As the problem is defined by discrete, episodic parameter evaluations (one LLE calculation per action), this discrete-time Bellman equation is the appropriate foundation, rather than the continuous-time Hamilton–Jacobi–Bellman (HJB) equation [17].

2.2. Previous RL Applications in Chaotic Systems

An adaptive RL approach has been proposed for chaotic systems governed by nonlinear differential equations [18]. The method assumed bounded but unknown nonlinearities, with the bounds estimated online. Neural networks were used to approximate the reinforcement signal, and control synthesis was achieved through Lyapunov stability theory and Young’s inequality. The primary objective was controller stabilization rather than an exploratory parameter search.

Reinforcement learning has also been applied to chaos control. For example, Q-learning has been employed to construct both control and anti-control strategies for systems such as the Lorenz attractor and Hénon map [19]. The LLE served as a reward to tune system parameters and modulate chaoticity [20]. Although this study demonstrated reinforcement learning viability in chaotic environments, the focus remained on regulating system trajectories.

More recently, research has explored reinforcement learning in partially observable chaotic systems, applying the framework to the Kuramoto–Sivashinsky equation [21]. Recurrent and transformer-based policy networks were compared for memory encoding, with attention mechanisms outperforming in highly chaotic regimes.

Beyond control and stabilization, recent work has explored RL for parameter estimation and adaptive prediction in chaotic environments. For instance, some studies combine machine learning techniques with synchronization methods to estimate unknown parameters from chaotic time series data [22], a crucial step for subsequent control or analysis. Other research leverages RL for adaptive time-stepping in chaotic simulations, where the agent learns to dynamically adjust integration parameters to balance accuracy and computational cost [23].

In contrast, the present study focuses on autonomous parameter space exploration for chaos discovery rather than stabilization or control [24]. The LLE is used as a direct reward measure of chaotic behavior under full state observability [25].

These distinctions establish the proposed framework as a general-purpose methodology for chaos identification, with applications in system discovery, complexity analysis, and secure system design [26].

2.3. SARSA Algorithm

SARSA is an on-policy temporal difference method that updates the action value function $Q(s,a)$ based on actions executed under the current policy [27]. At each time step, the agent transitions from state $s_t$ to $s_{t+1}$ by executing action $a_t$ and selects the subsequent action $a_{t+1}$ using the same policy [28].

The update rule is defined as

$$Q(s_t,a_t)\leftarrow Q(s_t,a_t)+\alpha \left[r_{t+1}+\gamma Q(s_{t+1},a_{t+1})-Q(s_t,a_t)\right],$$

(3)

where $\alpha$ denotes the learning rate and $\gamma$ is the discount factor. Policy-consistent updates contribute to stable convergence, making SARSA suitable in scenarios where real-time policy adherence is desirable [29]. The convergence of tabular SARSA to the optimal policy is guaranteed as long as all state-action pairs continue to be visited, a condition satisfied by an $ϵ$-greedy policy [30].

SARSA is included in this study primarily as an on-policy baseline. We evaluate it to determine if its more conservative, policy-dependent updates offer greater stability when navigating the potentially sharp and narrow reward landscapes found in chaotic parameter spaces.

2.4. Q-Learning Algorithm

Q-learning is an off-policy temporal difference method that estimates the optimal action value function $Q^{*}(s,a)$ independently of the policy used during training [31]. The update rule is as follows [32]:

$$Q(s_t,a_t)\leftarrow Q(s_t,a_t)+\alpha \left[r_{t+1}+\gamma \underset{a}{max}Q(s_{t+1},a)-Q(s_t,a_t)\right].$$

(4)

This greedy update mechanism promotes faster convergence to high-reward regions, particularly in sparse or high-dimensional spaces [33]. Tabular Q-learning is proven to converge to the optimal action-value function $Q^{*}$ with probability 1, provided all state-action pairs are visited infinitely and a proper learning rate schedule is used [34].

Q-learning is the primary algorithm of interest for this discovery task. Its off-policy nature is a significant advantage, as it allows the agent to learn the optimal value function $Q^{*}$ independently of the exploration policy. This is ideal for a parameter search, where we desire efficient, and at times aggressive, exploration (e.g., $ϵ$-greedy) without corrupting the learned value of the optimal “chaos-inducing” policy.

2.5. Reward Function Design

The reward function provides the primary learning signal for reinforcement learning agents [35]. In this study, the LLE $\lambda$ is used to quantify the degree of chaos, serving as a scalar reward. Positive values of $\lambda$ reflect the exponential divergence of nearby trajectories [36], which is a fundamental property of chaotic systems.

To enhance reward signal resolution and ensure numerical stability, the exponentiated form of the Lyapunov exponent is used in higher-dimensional systems:

$$r_{t+1}=exp\left(\lambda \left(\theta \right)\right),$$

(5)

where $\theta$ denotes the selected parameter set. This transformation ensures all rewards remain positive and amplifies small differences in $\lambda$ near the chaotic threshold, facilitating more efficient exploration in large parameter spaces.

Using the LLE as the reward signal is the core methodological choice that connects the RL agent’s objective to the physical definition of chaos. The transformation in Equation (5) is a critical design feature to handle reward sparsity, ensuring that the agent can differentiate between weakly chaotic (small $\lambda >0$) and strongly chaotic (large $\lambda >0$) regimes, rather than treating all chaos as a simple binary reward.

For lower-dimensional systems, where chaotic regimes are well-defined and more easily separable, the untransformed Lyapunov exponent is sufficient:

$$r_{t+1}=\lambda \left(\theta \right).$$

(6)

This adaptive reward function promotes the discovery of parameter configurations that exhibit sustained chaos while implicitly penalizing periodic or fixed-point dynamics. Regularization terms such as $\lambda +ϵ$ could be introduced near the edge of chaos to avoid vanishing gradients and improve learning stability.

2.6. Advantages over Traditional Numerical Analysis

Conventional approaches to chaotic parameter discovery, such as grid-based parameter sweeps, bifurcation diagrams, and Lyapunov surface visualizations, often require expert intervention and extensive computational cost.

In contrast, the reinforcement learning-based framework provides several key advantages, which are experimentally quantified in Section 4.1. The primary advantages are as follows:

• An adaptive exploration of parameter regions with higher chaotic potential based on experience.
• The reuse of evaluated configurations to inform future decisions and reduce redundancy.
• Scalability to high-dimensional systems.
• Autonomous operation without reliance on heuristic or manual tuning strategies.

By combining Lyapunov-guided rewards with sample-efficient policy updates, the framework enables self-directed learning in complex dynamical environments, offering a scalable and generalizable solution for chaos identification. A summary contrasting the algorithms regarding their relevance to parameter discovery is provided in

Table 1. Condensed comparison of methods for chaotic parameter discovery under matched evaluation budgets.

Method	Policy Type	Pros for Search	Limitations/Notes
Q-learning	Off-policy	Fast progress in sparse high-LLE landscapes; simple tabular update; good at jumping to promising peaks.	Can over-exploit noisy estimates near the edge of chaos; needs sufficient $(s,a)$ coverage and tuned step sizes.
SARSA	On-policy	Behavior-consistent updates; smoother learning when chaotic windows are narrow.	May converge more slowly in very sparse peaks; performance depends on exploration schedule (GLIE/annealing).
Grid/Random	N/A	Simple baseline; embarrassingly parallel; stable reference for fair budgets.	Curse of dimensionality; no adaptation to outcomes; many redundant evaluations in non-chaotic regions.

.

3. Methodology

This section outlines the chaotic systems investigated, the Lyapunov-based reward function, and the implementation details of the reinforcement learning process. A unified framework is applied to all systems to ensure consistency and scalability.

3.1. Chaotic System Models

The six following dynamical systems are included, chosen to represent a range of dimension and behaviors from classic 1D maps to hyperchaotic system flows:

• Logistic Map (1D) [37]: A discrete-time model that serves as a canonical example of how complex, chaotic behavior can arise from a simple nonlinear iterative equation. It is defined by

  $$x_{t+1}=r{x}_t(1-x_t).$$

  (7)

  where $x_t\in [0,1]$ and r is the bifurcation parameter.
• Hénon Map (2D) [38]: A classic 2D discrete-time dynamical system that is well-known for exhibiting a strange attractor. Its equation is given by

  $$\left\{\begin{array}{cc}\hfill x_{t+1}& =1-a{x}_t^2+y_t\hfill \\ \hfill y_{t+1}& =b{x}_t\hfill \end{array}\right.$$

  (8)

  with control parameters a and b.
• Lorenz System (3D) [39]: A simplified mathematical model for atmospheric convection, notable for its “butterfly” strange attractor, which was one of the first systems to demonstrate deterministic chaos. It is defined by

  $$\left\{\begin{array}{cc}\hfill {\displaystyle \frac{dx}{dt}}& =\sigma (y-x),\hfill \\ \hfill {\displaystyle \frac{dy}{dt}}& =x(\rho -z)-y,\hfill \\ \hfill {\displaystyle \frac{dz}{dt}}& =xy-\beta z,\hfill \end{array}\right.$$

  (9)

  where $\sigma$, $\rho$, and $\beta$ are adjustable parameters.
• Chua’s Circuit (3D) [40]: A simple electronic circuit that exhibits a wide range of nonlinear dynamics, including bifurcations and chaos, making it a standard benchmark. Its dynamics are governed by

  $$\left\{\begin{array}{cc}\hfill {\displaystyle \frac{dx}{dt}}& =\alpha (y-x-f(x\left)\right),\hfill \\ \hfill {\displaystyle \frac{dy}{dt}}& =x-y+z,\hfill \\ \hfill {\displaystyle \frac{dz}{dt}}& =-\beta y,\hfill \end{array}\right.$$

  (10)

  with a piecewise linear function $f\left(x\right)=m_{1}x+0.5(m_0-m_1)\left(\right|x+1|-|x-1\left|\right)$ modeling the nonlinear resistor.
• Lorenz–Haken System (4D) [41]: A model that describes the dynamics of a single-mode laser, often used to study instabilities and chaos in optical systems. Its equations are given by

  $$\left\{\begin{array}{cc}\hfill {\dot{x}}_1& =\sigma (x_2-x_1),\hfill \\ \hfill {\dot{x}}_2& =-x_2-\delta x_3+(r-x_4)x_1,\hfill \\ \hfill {\dot{x}}_3& =\delta x_2-x_3,\hfill \\ \hfill {\dot{x}}_4& =-b{x}_4+x_1{x}_2,\hfill \end{array}\right.$$

  (11)

  where $\sigma$, $\delta$, r, and b are system parameters.
• Custom 5D Hyperchaotic System: A high-dimensional system designed for this study to test scalability and performance in a hyperchaotic regime. It is defined by

  $$\left\{\begin{array}{cc}\hfill {\displaystyle \frac{dx}{dt}}& =a(y-x)+sin\left(k_{1}v\right)\hfill \\ \hfill {\displaystyle \frac{dy}{dt}}& =bx-y-xz+cos\left(k_{2}u\right)\hfill \\ \hfill {\displaystyle \frac{dz}{dt}}& =xy-cz+sin\left(k_{3}u\right)\hfill \\ \hfill {\displaystyle \frac{du}{dt}}& =-du+sin\left(k_{4}x\right)\hfill \\ \hfill {\displaystyle \frac{dv}{dt}}& =ez-yu+cos\left(k_{5}y\right)\hfill \end{array}\right.$$

  (12)

  which is constructed by embedding a Lorenz-like core into a 5D structure with nonlinear interactions and auxiliary feedback states. This model yields multiple positive Lyapunov exponents and high-dimensional attractors.

3.2. Numerical Integration

All continuous systems are integrated using the fourth-order Runge–Kutta (RK4) method with a fixed time step of $\Delta t=0.01$ [42]. The RK4 algorithm is defined as

$$\left\{\begin{array}{cc}\hfill k_1& =f\left({\mathbf{x}}_t\right),\hfill \\ \hfill k_2& =f({\mathbf{x}}_t+\frac{\Delta t}{2}{k}_1),\hfill \\ \hfill k_3& =f({\mathbf{x}}_t+\frac{\Delta t}{2}{k}_2),\hfill \\ \hfill k_4& =f({\mathbf{x}}_t+\Delta t·k_3),\hfill \\ \hfill {\mathbf{x}}_{t+\Delta t}& ={\mathbf{x}}_t+\frac{\Delta t}{6}(k_1+2k_2+2k_3+k_4).\hfill \end{array}\right.$$

(13)

This method was chosen because RK4 offers a standard balance between accuracy and predictable computational cost for simulating chaotic systems [43]. Its higher order accuracy (global error $O\left(h^4\right)$ vs. Euler’s $O\left(h\right)$) is crucial for chaos, where errors amplify quickly. Furthermore, the fixed step size ensures a consistent cost per LLE evaluation, which is better suited for the tabular RL framework than adaptive step-size methods.

3.3. Search Space and Training Settings

Parameter ranges for each chaotic system are discretized, as shown in

Table 2. Parameter space discretization used in the experiments.

System	Parameters (Range)	Discretization
Logistic map	$r\in [3.5,4.0]$	100 steps
Hénon map	$a\in [1.2,1.4],b\in [0.2,0.4]$	$50\times 50$ grid
Lorenz system	$\sigma ,\rho ,\beta$	${20}^3$ grid
Chua’s circuit	$\alpha ,\beta ,m_0$	${15}^3$ grid
Lorenz–Haken	$\sigma \in [1,10],\delta \in [0.1,2],r\in [10,30],b\in [0.1,2]$	${10}^4$ grid
Custom 5D	bounded box over model parameters	${10}^4$ sampled points

. Each episode selects a parameter vector $\theta$ from the discrete set, runs a short simulation, and estimates the LLE ${\widehat{\lambda }}_{max}\left(\theta \right)$ as the scalar reward. Q-values over the discrete candidates are updated using either SARSA or Q-learning under $ϵ$-greedy behavior. Protocol details and hyperparameters are summarized in

Table 3. Training protocol and hyperparameters.

Component	Setting
Environment	Single-step evaluation; state $s=\theta$; action selects $\theta$; reward $r={\widehat{\lambda }}_{max}\left(\theta \right)$; simulator integrates dynamics under $\theta$.
Behavior policy	$ϵ$-greedy over the discrete candidate set.
Update rule	SARSA or Q-learning on tabular $Q\left(\theta \right)$.
Learning rate	$\alpha =0.5$.
Discount factor	$\gamma =0.9$ (single-step return).
Exploration rate	$ϵ\in [0.1,0.3]$.
Episodes	1000–4000 (system-dependent).
Outputs	Reward trends, parameter heatmaps/attractor plots, final $Q\left(\theta \right)$.

.

All experiments reported in this study were conducted as software simulations. The computing hardware used was an Intel Core i9-13900K processor with 64 GB of RAM, running Windows 11 Pro. The primary focus of the analysis remains on the sample efficiency (number of evaluations) rather than specific execution times or detailed resource utilization, as these can vary significantly based on implementation details.

4. Experimental Results and Analysis

This section reports an evaluation of the proposed reinforcement learning framework on a collection of six chaotic systems. The analysis covers learning performance, attractor reconstruction, and algorithm comparison. The results are organized into subsections detailing the evaluated systems, the observed training performance, and the final attractor reconstructions.

4.1. Quantitative Baseline Comparison

A primary claim of this study is that RL-based exploration reduces redundant simulations compared to a brute-force grid scan. To quantitatively verify this, define computational cost as the number of LLE evaluations required. For a grid search, this is the total number of points in the discretized space (from Table 1). For the RL agents, this is the number of episodes (evaluations) required to consistently find parameter regions with a high positive LLE (estimated from Figure 1, Figure 2, Figure 3, Figure 4, Figure 5 and Figure 6).

Table 4. Comparison of computational cost (number of LLE evaluations).

System	Grid Search Baseline (Total Evaluations)	RL Agent (Approx. Episodes to Find High LLE)	Cost Reduction
Logistic Map	100	≈100–200	≈0%
Hénon Map	2500	≈200–300	≈88–92%
Lorenz System	8000	≈100–200	≈97.5–98.8%
Chua’s Circuit	3375	≈150–200	≈94.1–95.6%
Lorenz–Haken	10,000	≈25–50	≈99.5–99.75%
Custom 5D System	10,000	≈100–150	≈98.5–99.0%

provides a direct experimental comparison. As shown in the table, the 1D Logistic map, with only 100 candidate points, serves as a low-dimensional benchmark where the search space is too small to offer a significant advantage for RL over a grid scan; the experimental run (200 episodes) simply explored this small space thoroughly, resulting in no cost reduction.

However, for all systems 2D and higher, the efficiency gain is substantial. For instance, in the 5D hyperchaotic system, the agents found high-LLE parameters within approximately 100–150 episodes, a >98% reduction in cost compared to the 10,000-point baseline. This demonstrates that the agent’s adaptive exploration policy effectively avoids redundant evaluations of non-chaotic regions and scales efficiently to high-dimensional systems.

4.2. Overview of Evaluated Systems

The experimental systems span a wide spectrum of nonlinear behaviors and dimensionalities, providing a broad benchmark for parameter search in chaotic dynamics. The two discrete-time maps—the logistic map and the Hénon map—serve as classical low-dimensional cases defined by iterative equations and are assessed via Jacobian-based LLE estimates.

The continuous-time set includes the Lorenz system, Chua’s circuit, the Lorenz–Haken laser model, and a designed 5D hyperchaotic system. These systems are simulated using the fourth-order Runge–Kutta method, with chaos identified through positive LLE estimates and corroborating attractor reconstructions.

All systems are trained independently under tabular Q-learning and SARSA using a unified pipeline, a common parameter discretization strategy, and the same reward definition. This design supports comparability across models and consistency in algorithm behavior. System-specific settings, including parameter ranges and integration details, are provided in the methodology section.

4.3. Training Performance and Reward Convergence

Figure 1, Figure 2, Figure 3, Figure 4, Figure 5 and Figure 6 show combined reward curves for each system under Q-learning and SARSA. The reward is a monotone transform of the largest Lyapunov exponent (LLE) used to reflect how often the methods reach parameter regions associated with chaotic behavior.

In the logistic and Hénon maps, both methods converge within ≈200 episodes. SARSA exhibits slightly higher variance, while Q-learning stabilizes earlier. Fast convergence is expected in these low-dimensional, discrete-time settings.

For the Lorenz system, Chua’s circuit system, and Lorenz–Haken system, both methods gradually reach regions linked to a positive LLE. SARSA is more responsive to early signals and can overshoot transiently; Q-learning shows steadier increases. In Chua’s circuit, SARSA displays early peaks that are not consistently sustained, while Q-learning tends to remain near boundary regions, consistent with the difficulty posed by narrow chaotic windows.

For the 5D multi-scroll hyperchaotic model, the parameter space is larger and reward fluctuations persist longer. Both methods eventually reach high-reward regions, with SARSA benefiting from on-policy updates and Q-learning improving more slowly under the fixed budget.

4.4. Attractor Reconstruction and Chaotic Behavior

To verify the emergence of chaotic dynamics under the learned parameters, phase space trajectories were visualized for each system. These attractor plots serve as a qualitative confirmation of chaos, complementing the Lyapunov-based reward signal. For discrete-time systems, iteration results are directly plotted. For continuous systems, the final trajectories after integration using the fourth-order Runge–Kutta method are displayed.

In the logistic map (Figure 7), both Q-learning and SARSA produced state sequences that densely occupied the available phase space.

In the Hénon map (Figure 8), attractors resemble the canonical horseshoe structure, demonstrating sensitivity to initial conditions and non-periodicity.

For Chua’s circuit in Figure 9, the reconstructed 3D trajectoriesunder SARSA spiraled toward a fixed point rather than forming a multi-scroll attractor. However, Q-learning successfully located parameters that generate the canonical double-scroll attractor. Although reward signals were obtained during training, the dynamics indicate convergence to parameters near the edge of chaos, likely in periodic or marginally stable regimes. In contrast, the trajectory under SARSA converged the results in a stable fixed point.

The Lorenz system, as shown in Figure 10, yielded the classical double-scroll attractor under both algorithms.

Figure 11 and Figure 12 show pairwise projections $(x_i,x_j)$ of trajectories under learned parameters for the Lorenz–Haken system. Q-learning (Figure 11) covers broader regions with visible folding and stretching, whereas SARSA (Figure 12) forms tighter loops with smoother occupancy around the main sheets. These patterns are consistent with positive largest Lyapunov exponent estimates and non-repeating local geometry. Figure 13 presents three-dimensional projections exhibiting layered sheets and lobe-like structures characteristic of chaotic motion in this model.

The 5D hyperchaotic system, constructed with sinusoidal nonlinearities, demonstrates intricate chaotic attractors under both SARSA and Q-learning. As shown in Figure 14 and Figure 15, the 2D phase projections reveal continuously folding trajectories with irregular returns, characteristic of chaotic motion. The corresponding 3D attractors in Figure 16 display volumetric expansion and intertwined spirals, indicative of multi-scroll dynamics and inter-variable feedback.

5. Discussion

This section analyzes the experimental findings from Section 4, compares them to related work, and discusses their practical implications and inherent limitations.

5.1. Summary of Experimental Findings

This subsection consolidates the main experimental outcomes observed across the six benchmark systems.

Table 5. Consolidated summary of experimental findings.

System	Episodes to High LLE	Attractor Quality	Q-Learning vs. SARSA Notes
Logistic Map (1D)	≈100–200	Correct	Both converge; Q slightly earlier.
Hénon Map (2D)	≈200–300	Correct	Similar; SARSA higher variance.
Lorenz System (3D)	≈100–200	Correct	Q faster; SARSA less late variance.
Chua’s Circuit (3D)	≈150–200	Correct (Q-Learning), Failed (SARSA)	Q-learning found attractor; SARSA converged to fixed point.
Lorenz–Haken (4D)	≈25–50	Correct	Both succeed; Q broader, SARSA tighter loops.
Custom 5D System	≈100–150	Correct	Both succeed; SARSA steadier curve.

provides a comparative overview, summarizing the number of episodes required for the RL agents to consistently identify high-LLE regions, the qualitative assessment of the resulting attractor’s structure and the key performance differences noted between the Q-learning and SARSA algorithms.

The approximate training times are provided in

Table 6. Approximate RL training times.

System	Q-Learning Time (s)	SARSA Time (s)
Logistic Map (1D)	≈2.5	≈2.5
Hénon Map (2D)	≈39.0	≈39.6
Lorenz System (3D)	≈226.5	≈226.7
Chua’s Circuit (3D)	≈1418.6	≈1438.6
Lorenz–Haken (4D)	≈145.3	≈145.6
Custom 5D System	≈340	≈340

, illustrating the practical duration for each simulation run.

5.2. Comparison with Related Work and Practical Implications

The framework applies RL to the problem of parameter discovery, which distinguishes it from prior studies that focused on RL for chaos stabilization or trajectory control. Because these studies did not aim to comprehensively map chaotic parameter regimes, a direct comparison of search results is less relevant than comparing the findings to the traditional numerical methods the framework seeks to complement: grid-based parameter sweeps and bifurcation analysis [44].

This study’s results demonstrate a clear efficiency gain over such brute-force methods. A traditional grid scan must evaluate every single candidate point, resulting in massive computational cost that suffers from the curse of dimensionality [45]. This is particularly true for multi-parameter bifurcation analysis, which becomes computationally prohibitive and exceedingly difficult as the number of system parameters increases. For instance, the discretized Lorenz parameter space contained ${20}^3$ (8000) candidate, while the Lorenz-Haken and 5D systems each had ${10}^4$ (10,000) candidates. In contrast, the RL agents located high-LLE regions for the Lorenz system within approximately 200 episodes and for the Hénon map within 300 episodes. This represents a small fraction of the total search space, confirming that the Lyapunov-guided RL approach successfully reduces redundant simulations through adaptive exploration. This superior sample efficiency [46] is precisely what enables scalability to the 4D and 5D systems, for which an exhaustive 10,000-point grid scan would be computationally infeasible under the same evaluation. While finding high-LLE regions can be framed as an optimization task potentially addressable by metaheuristics, the contribution of RL lies in learning an adaptive sequential sampling policy from experience, offering a different paradigm compared to many population-based optimizers and demonstrating significant efficiency gains over brute-force methods as shown here.

5.3. Analysis of the Chua’s Circuit Case: Algorithmic Limitations and Landscape Topology

A divergent performance was observed in the Chua’s circuit experiment. The Q-learning agent identified parameters that generated a double-scroll chaotic attractor, as shown in Figure 9a. In contrast, the SARSA agent, operating under the same evaluation budget and ${15}^3$ parameter grid, converged to parameters that resulted in a stable fixed point as shown in Figure 9b.

This failure of SARSA highlights a limitation of tabular methods for this system’s reward landscape, which is believed to contain narrow chaotic windows. This landscape topology interacts critically with our ${15}^3$ discretization grid and the specific update rules. Q-learning’s optimistic update rule updates its value based on the maximum expected future reward (${max}_{a}Q(s_{t+1},a)$). This allows it to identify and propagate the value of a high-reward (chaotic) state, even if the $ϵ$-greedy exploration policy often steps out of that narrow window.

Conversely, SARSA’s policy-dependent update uses the value of the next action chosen by its exploration policy ($Q(s_{t+1},a_{t+1})$). When exploring a narrow reward peak, an exploratory for the next action will frequently land in an adjacent low-reward (stable) region, which ’averages down’ the value estimate for the chaotic state. This comparison suggests SARSA’s more conservative update rule struggled to stabilize on the narrow reward peak, converging instead to the broader, suboptimal fixed-point region. Q-learning’s optimistic update proved more effective for this discovery task.

This case identifies a clear limit of the tabular approach, especially when grid resolution is coarse. A neural network for employing deep reinforcement learning approaches, such as Deep Q-Networks (DQN), it could learn a value function over the continuous parameter space. This would remove the dependency on a prior discretization and thus potentially resolve these narrow chaotic regimes more efficiently.

6. Conclusions

This study introduced and validated a reinforcement learning framework for autonomous chaotic parameter discovery. The primary contributions of this work were experimentally validated across six dynamical systems. The experimental results confirmed the framework’s effectiveness: it successfully identified complex chaotic attractors while providing a quantitative computational cost reduction of over 97% compared to traditional grid scans in high-dimensional spaces as shown in Table 4. By leveraging the LLE as a reward signal, both Q-learning and SARSA demonstrated the ability to adaptively locate desirable parameter configurations, validating this approach as a practical and efficient alternative to brute-force exploration.

Under matched sampling budgets, the approach reconstructed representative attractors across the Logistic and Hénon maps, the Lorenz and Chua systems, the Lorenz–Haken (4D) model, and a 5D hyperchaotic design. Comparative observations indicate that Q-learning tended to cover the parameter space more widely, whereas SARSA yielded steadier updates in several settings with narrow chaotic bands. For Chua’s circuit, SARSA did not locate a parameter set that sustained chaotic behavior within the reported budget; trajectories settled into a fixed point or a low-period limit cycle.

While scalability up to five dimensions was demonstrated, the tabular methods required parameter space discretization and showed sensitivity to hyperparameters, particularly when dealing with sharp transitions or narrow chaotic windows. These limitations highlight clear avenues for future research. Employing deep reinforcement learning approaches, such as Deep Q-Networks, could remove the need for discretization and scale the framework to significantly higher-dimensional parameter spaces. Further improvements to efficiency and robustness could be achieved by exploring hybrid methods that merge RL’s global search with local optimization techniques, or by implementing advanced exploration strategies beyond epsilon-greedy. Overall, Lyapunov-guided reinforcement learning serves as a practical complement to traditional search methods, offering clear paths for advancement.