Reinforcement Learning Compensatory-Based Fully Actuated Control Method for Risley Prisms

Abstract

Beam pointing control based on Risley prisms is of great significance in wide-angle, high-precision application scenarios, such as laser communication, but its inherent nonlinear system characteristics seriously restrict the performance of beam pointing control, such as accuracy. For this reason, this paper combines the theory of fully actuated control with reinforcement learning methods and designs a fully actuated control method based on reinforcement learning compensation: suppressing the influence of system nonlinearity through fully actuated control, using reinforcement learning to estimate system perturbations and nonlinearities, and then outputting a compensated control quantity using the low-dimensional output of fully actuated control as the reference input of reinforcement learning reduces the complexity of learning and realises the end-to-end uncertainty estimation. Finally, the stability of the method is theoretically analyzed, and the effectiveness of the method is verified by experimental analysis, which can further improve the beam pointing accuracy of the Risley prism system.

1. Introduction

Beam control technology as the core link to realize high-bandwidth and low-latency space information transmission plays a crucial role in the field of laser communication. In order to establish and maintain a stable and reliable optical communication connection in dynamically changing interstellar or space–ground links, high-precision and wide-angle pointing control of laser beams is an indispensable key technology [1,2]. There are currently many beam control methods, such as Liquid Crystal Phase Arrays, Fast Steering Mirror Compensation, and Risley prisms. Liquid Crystal Phase Arrays are a programmable, non-mechanical beam deflection technology that uses electrical signals to control the phase of each unit in a liquid crystal spatial light modulator array, enabling inertia-free beam scanning [3]. They offer advantages such as compact size, lightweight design, and low power consumption, but their deflection range is limited, and they are susceptible to temperature fluctuations. Fast Steering Mirror Compensation is based on high-bandwidth closed-loop feedback control, using position detectors to monitor spot deviation in real time and driving fast steering mirrors for high-precision, high-speed compensation and correction [4,5,6,7]. While it offers high precision and a fast response, this method uses a large number of components, resulting in a bulky system and high power consumption. The Risley prism method, which is the focus of this paper, utilizes the optical principle of prism refraction to change the direction of light beams by controlling the rotation of two prisms with a motor. This technology plays a crucial role in applications requiring both a wide range and high precision [8]. However, this system has multiple error sources and exhibits significant nonlinearity.

The complex nonlinear relationship between the beam pointing and the prism rotation angle in Risley prism systems can lead to challenges such as the complex mapping relationship between beam deflection rate and prism rotation speed. This nonlinear mapping causes slight fluctuations in the prism rotation speed to be significantly amplified into macroscopic deviations or jitter in the beam pointing. Therefore, in high-precision applications such as laser communication and target tracking, precise and stable control of the prism rotation speed is key to overcoming system nonlinear effects and ensuring beam pointing accuracy. To solve the nonlinearity problem in the Risley prism and improve control accuracy, Ball Aerospace and Technologies Corp. introduced a third prism to enable smooth movement of the beam within the pointing range. However, this method increased the weight and complexity of the system structure, making it more difficult to calculate and control and limiting its feasibility in practical applications [9]. Sun J used a lookup table method to replace complex solutions and combined genetic algorithms with PID control algorithms to improve the control accuracy and stability of the rotating Risley prism [10]. Li A studied a cam-mechanism-based drive method to reduce the error sources caused by nonlinear motor control through physical structure, optimize system performance, and improve tracking accuracy [11]. Shen Y used a particle swarm algorithm to improve the pointing accuracy when the prism apex angle increased [12]. Li Y, Li J, Zhou Y, et al. analyzed the nonlinearity and singularity of the rotating Risley prism, explained the internal mechanism of the double prism scan in principle, and provided a theoretical basis for the nonlinear control of the double prism [13,14,15,16,17,18,19,20,21,22,23]. Ma R et al. designed a robust control algorithm based on interference observers to improve control performance [24]. Yuan L et al. improved pointing accuracy through system identification and error correction [25].

With the development of artificial intelligence technology, Yuan L et al. used a double BP neural network to alleviate the contradiction between speed and accuracy in traditional inverse solutions [26]. Torales et al. combined neural networks with PID and applied them to a Risley prism control system [27]. Yao Y conducted partial research on Risley prism control based on deep reinforcement learning [28]. However, there are still difficulties in directly applying neural-network-based artificial intelligence methods to control systems, such as high training costs and convergence difficulties [29].

The inherent nonlinear characteristics of the Risley prism system also depend on factors such as system structure [30], prism material, parameters, and environment, making it difficult to design a relatively general high-precision robust control method. At the same time, there has been little research on high-precision control algorithms for the Risley prism itself. Therefore, research on the nonlinearity and high-precision control of the Risley prism system remains important.

Fully actuated control is a control system framework that has been proposed in recent years [31,32,33]. This method has great advantages in handling complex nonlinear systems, but its control capabilities are still limited for highly uncertain or time-varying disturbance nonlinear systems. Reinforcement learning has robust nonlinear learning and adaptive capabilities and does not rely on system models, but its learning and training complexity is relatively high [34,35,36]. In response to the above issues, this paper combines reinforcement learning methods with fully actuated control. A fully actuated control method based on reinforcement learning for Risley prisms is designed to further solve the nonlinearity issues in Risley prism control and improve control accuracy. The main methods and innovations are as follows.

(1) Design of the main controller: A robust, fully actuated control model is designed for the motor control of the Risley prism. This controller overcomes the nonlinearity in some systems through full state feedback and achieves good control results.

(2) Design of reinforcement learning compensators: By using system control errors, error change rates, main controller outputs, and other factors such as reinforcement learning observations, Actor-Critic networks and reward functions are designed to further learn and estimate modeling uncertainties, such as disturbances and nonlinearities in the system, and output control compensation amounts to improve system control accuracy.

Part of the fully actuated control information is used as a reinforcement learning reference to estimate system disturbances and uncertainties end-to-end with low-dimensional information and learn the output control compensation amount. This improves the control performance of the system and reduces the complexity of reinforcement learning. At the same time, this paper also theoretically proves the stability and superiority of the method and finally verifies the effectiveness of the method through experimental analysis.

2. System Module

2.1. Risley Prism Beam Pointing Model

Figure 1 shows a schematic diagram of the Risley prism beam pointing. Prism 1 and Prism 2 are placed coaxially and can rotate independently around the Z-axis driven by a motor. The right angles of the two prisms are parallel and perpendicular to the Z-axis. The refractive indices and apex angles of the two prisms are $n_1$, $n_2$, ${\alpha }_1$ and ${\alpha }_2$.

The rotation angles of the prisms are ${\theta }_{M1}$ and ${\theta }_{M2}$, the azimuth angle of the beam is $\theta$, defined within the range $[0,2\pi ]$, and the deflection angle is $\mathsf{\Phi }$. The distance between Prism 2 and the right-side observation screen is P.

When the incident light is directed along the Z-axis in the opposite direction, the cosine of the direction of the emerging light can be obtained using the classical forward non-paraxial ray-tracing method:

$$\left\{\begin{array}{c}K=a_{1}cos{\theta }_{M1}+a_{3}sin{\alpha }_{2}cos{\theta }_{M2}\hfill \\ L=a_{1}sin{\theta }_{M1}+a_{3}sin{\alpha }_{2}sin{\theta }_{M2}\hfill \\ M=a_2-a_{3}cos{\alpha }_2\hfill \end{array}\right.$$

(1)

where

$$a_1={\displaystyle \frac{n_2}{n_1}}sin{\alpha }_1\left(cos{\alpha }_1-\sqrt{n_1^2-{sin}^2{\alpha }_1}\right)$$

(2)

$$a_2=-{\displaystyle \frac{n_2}{n_1}}\left(\sqrt{n_1^2-{sin}^2{\alpha }_1}cos{\alpha }_1+{sin}^2{\alpha }_1\right)$$

(3)

$$\begin{array}{cc}\hfill {\mathit{a}}_3& =-({\mathit{a}}_{1}sin{\mathit{\alpha }}_{2}cos\Delta \theta -{\mathit{a}}_{2}cos{\alpha }_2)+\sqrt{1-{\mathit{n}}_2^2+{({\mathit{a}}_{1}sin{\alpha }_{2}cos\Delta \theta -{\mathit{a}}_{2}cos{\alpha }_2)}^2}\hfill \end{array}$$

(4)

$$\Delta \theta ={\theta }_{M2}-{\theta }_{M1}$$

(5)

The deflection angle $\mathsf{\Phi }$ and azimuth angle $\theta$ of the emitted beam are, respectively:

$$\mathsf{\Phi }=arccos(-M)$$

(6)

$$\theta =\left\{\begin{array}{cc}\text{arctan}\left({\displaystyle \frac{L}{K}}\right);\mathrm{when}K⩾0\mathrm{and}L⩾0\hfill & \\ \text{arctan}\left({\displaystyle \frac{L}{K}}\right)+2\pi ;\mathrm{when}K⩾0\text{and}L<0\hfill & \\ \text{arctan}\left({\displaystyle \frac{L}{K}}\right)+\pi ;\mathrm{when}K<0\hfill \end{array}\right.$$

(7)

From Equation (1) ∼ Equation (6), it can be seen that the deflection angle $\mathsf{\Phi }$ depends only on the orientation angle between the two prisms $|\Delta \theta |$. Given the deflection angle $\mathsf{\Phi }$ of the beam target position, the following can be obtained from Equation (1) ∼ Equation (6):

$$|\Delta \theta |=arccos\left({\displaystyle \frac{1}{a_{1}tan{\alpha }_2}}\left(a_2+{\displaystyle \frac{1}{2(a_2+cos\mathsf{\Phi })}}\times \left(1-n_2^2-{\left({\displaystyle \frac{a_2+cos\mathsf{\Phi }}{cos{\alpha }_2}}\right)}^2\right)\right)\right)$$

Subsequently, the classic two-step method can be used to obtain two sets of solutions for the prism rotation angle.

First solution:

$${\theta }_0^{\prime }=\left\{\begin{array}{cc}\text{arctan}{\left({\displaystyle \frac{L}{K}}\right)}_{{\theta }_{M1}=0,{\theta }_{M2}=|\Delta \theta |},\mathit{K}>0,\mathit{L}⩾0\hfill & \\ & \\ \text{arctan}{\left({\displaystyle \frac{L}{K}}\right)}_{{\theta }_{M1}=0,{\theta }_{M2}=|\Delta \theta |}+2\pi ,\mathit{K}⩾0,\mathit{L}⩾0\hfill & \\ & \\ \text{arctan}{\left({\displaystyle \frac{L}{K}}\right)}_{{\theta }_{M1}=0,{\theta }_{M2}=|\Delta \theta |}+\pi ,\mathit{K}<0\hfill \end{array}\right.$$

Then, synchronously rotate the two prisms to make the beam reach the specified azimuth angle $\theta$. The synchronous rotation angle is

$${\theta }_{M1}^{\prime }=\theta -{\theta }_0^{\prime }$$

The rotation angles of prism 1 and prism 2 are, respectively,

$${\theta }_{1n}^{\prime }={\theta }_{M1}^{\prime },{\theta }_{2n}^{\prime }={\theta }_{M1}^{\prime }+|\Delta \theta |$$

Second solution:

$${\theta }_0^{″}=\left\{\begin{array}{cc}\text{arctan}{\left({\displaystyle \frac{L}{K}}\right)}_{{\theta }_{M1}=|\Delta \theta |,{\theta }_{M2}=0},K⩾0,L⩾0\hfill & \\ & \\ \text{arctan}{\left({\displaystyle \frac{L}{K}}\right)}_{{\theta }_{M1}=|\Delta \theta |,{\theta }_{M2}=0}+2\pi ,K⩾0,L<0\hfill \\ \text{arctan}{\left({\displaystyle \frac{L}{K}}\right)}_{{\theta }_{M1}=|\Delta \theta |,{\theta }_{M2}=0}+\pi ,K<0\hfill \end{array}\right.$$

Then, synchronously rotate the two prisms to make the beam reach the specified azimuth angle $\theta$. The synchronous rotation angle is

$${\theta }_{M1}^{″}=\theta -{\theta }_0^{″}$$

The rotation angles of prism 1 and prism 2 are, respectively,

$${\theta ″}_{1n}={\theta ″}_{M1}+|\Delta \theta |,{\theta ″}_{2n}={\theta ″}_{M1}$$

2.2. Motor Control Models

In a rotating double prism system, the optical wedge can be rotated by a DC torque motor and bearings. The motor is fixed to the housing, and the double prism and the motor rotor are fixedly connected. Therefore, the motor and the load are rigidly connected. This method makes the system structure more compact and ensures the accuracy and stability of the control system.

In the system, the coupling effects between armature inductance, rotational inertia, and damping are fully considered. The transfer function of the load angular velocity and armature control voltage is modeled as

$${\displaystyle \frac{\omega \left(s\right)}{E_a\left(s\right)}}={\displaystyle \frac{K_t}{L_{a}J{s}^2+(L_{a}B+R_{a}J)s+R_{a}B+K_t{K}_e}}$$

where $\omega$ (rad/s) is the load angular velocity, $L_a\left(H\right)$ is the armature inductance, $\mathrm{J}(\mathrm{kg}·{\mathrm{m}}^2)$ is the total rotational inertia of the motor shaft, ${\mathrm{R}}_{\mathrm{a}}\left(\mathsf{\Omega }\right)$ is the armature resistance, $\mathrm{B}(\mathrm{N}·\mathrm{m}·\mathrm{s}/\mathrm{rad})$ is the equivalent viscous damping coefficient of the motor shaft, $K_T(\mathrm{N}·\mathrm{m}/\mathrm{A})$ is the motor torque constant, $K_e(\mathrm{V}·\mathrm{s}/\mathrm{rad})$ is the motor back-EMF coefficient, and ${\mathrm{E}}_{\mathrm{a}}\left(\mathrm{V}\right)$ is the control voltage of the armature.

Set the system input voltage to u and the output angular velocity to y, and introduce a disturbance d to the system input. Then, the differential equation of the system can be expressed as

$$\ddot{y}+{\displaystyle \frac{L_{a}B+R_{a}J}{L_{a}J}}\dot{y}+{\displaystyle \frac{R_{a}B+K_T{K}_e}{L_{a}J}}y={\displaystyle \frac{K_T}{L_{a}J}}u+d$$

Since ${\displaystyle \frac{d{\theta }_M}{dt}}=\omega$, ${\theta }_M$ is the rotation angle of the motor, which is also the rotation angle of the prism. The transfer function of the load angle and the armature control voltage is modeled as

$${\displaystyle \frac{{\theta }_M\left(s\right)}{E_a\left(s\right)}}={\displaystyle \frac{K_T}{L_{a}J{s}^3+(L_{a}B+R_{a}J)s^2+(R_{a}B+K_T{K}_e)s}}$$

Models can be simplified for specific engineering applications. The armature inductance $L_a$ is usually small and can be ignored. The equation is simplified to

$$R_{a}J{\displaystyle \frac{d^2{\theta }_M}{d{t}^2}}+(R_{a}B+K_T{K}_e){\displaystyle \frac{d{\theta }_M}{dt}}=K_T{e}_a$$

Set the input voltage of the system to u and the angular displacement of the output to y, and introduce a disturbance d in the input of the system. The system can be expressed as

$$\ddot{y}+{\displaystyle \frac{R_{a}B+K_T{K}_e}{R_{a}J}}\dot{y}={\displaystyle \frac{K_T}{R_{a}J}}u+d$$

Therefore, the differential equations between the load angle, the load angular velocity, and the control voltage of the armature can be uniformly expressed as

$$\ddot{y}+{\eta }_1\dot{y}+{\eta }_{0}y=bu+d$$

where ${\eta }_1$, ${\eta }_0$, b are system parameters.

2.3. Risley Prism Beam Control Model

The overall Risley prism beam pointing control system is shown in Figure 2. Based on the required beam pointing information, the target azimuth angle $\theta$ and the deviation angle $\mathsf{\Phi }$ are input into the inverse solution model to solve for the target angle ${\theta }_{\mathrm{M}}$ that the two prisms need to rotate to. Then, the motor control model controls the two prisms to rotate to the target angle, while the regular solution model calculates the actual deviation angle and azimuth angle of the beam based on the rotation angles of the two prisms. Under the control of the controller, the actual deviation angle and azimuth angle of the beam tend to the target angle, ultimately achieving accurate pointing.

3. Methods

The fully actuated control method based on reinforcement learning mainly consists of two parts. The first part is a fully actuated controller established based on the motor model, which serves as the main controller. The second part is a reinforcement-learning-based compensation control item, which uses system errors and error change rates as reinforcement learning agent observation inputs. Through the design of a suitable Actor–Critic network, the reinforcement learning agent estimates system disturbances and nonlinear modeling uncertainties, and finally outputs a compensation value. The control amount of the fully actuated controller and the reinforcement learning compensation amount are used as the final control input amount.

3.1. Fully Actuated Controller Design

The fully actuated control system method is a general framework for the analysis and design of control systems proposed in recent years [37]. It can effectively solve nonlinear control problems, including robust control, adaptive control, disturbance suppression, optimal control, and tracking control.

A robust fully actuated control system was designed for the motor system model constructed above. It consists of two parts. The first part compensates for nonlinear terms in the system other than interference and then configures a linear system with the desired characteristic structure. The second part is a robust stabilization term to deal with the effects of system disturbances.

The first part can be designed as follows [38]:

$$u_1=\widehat{\ddot{y}}+{\eta }_1\dot{y}+{\eta }_{0}y+({\ddot{y}}_d-\widehat{\ddot{y}})+a_1({\dot{y}}_d-\dot{y})+a_0(y_d-y)$$

(8)

where $y_d$ is the target speed to be achieved, $a_0$ and $a_1$ are the parameters to be designed to configure the system characteristics, $\widehat{\ddot{y}}$ is the observed estimate of $\ddot{y}$, and in actual practice, the value of $\ddot{y}$ is difficult to measure directly, so an observer needs to be designed to estimate it:

$$\left\{\begin{array}{cc}\dot{z}=\mathrm{L}(y,\dot{y})\mathrm{z}-L(y,\dot{y})p(y,\dot{y})\hfill & \\ \widehat{\ddot{y}}=z-p(y,\dot{y})\hfill \end{array}\right.$$

where $p(y,\dot{y})\in {\mathbb{R}}^2$ is the vector to be designed and $z\in {\mathbb{R}}^2$ is the internal variable of the acceleration observer.

$$\mathrm{L}(y,\dot{y})={\displaystyle \frac{\partial p(y,\dot{y})}{\partial \dot{y}}},L_1(y,\dot{y})\in {\mathbb{R}}^{2\times 2}$$

Set an appropriate $p(y,\dot{y})$ such that $L(y,\dot{y})$ is symmetric and negative definite; then, $\widehat{\ddot{y}}$ can accurately estimate $\ddot{y}$.

The second part is designed as follows [38]:

$$u_2={\displaystyle \frac{1}{4\epsilon }}{\theta }^2{P}_L^{\mathrm{T}}\left(a\right)e$$

(9)

where $|\theta |\ge |d|$, $\epsilon$ is the accuracy parameter, $P_L^{\mathrm{T}}\left(a\right)$ is the parameter matrix, and $e=y_d-y$ is the error. It should be noted that the error does not need to be concerned with the positive or negative sign, nor does it need to be squared or taken as an absolute value, as it does not affect subsequent analysis and calculation in this paper.

In summary, the fully actuated control law of the system is $u_{\mathrm{FAC}}=b^{-1}(u_1+u_2)$, and, with Equations (8) and (9), we obtain

$$\begin{array}{cc}\hfill u_{\mathrm{FAC}}& =b^{-1}(u_1+u_2)\hfill \\ \hfill & =b^{-1}\left(\widehat{\ddot{y}}+{\eta }_1\dot{y}+{\eta }_{0}y+({\ddot{y}}_d-\widehat{\ddot{y}})+a_1({\dot{y}}_d-\dot{y})+a_0(y_d-y)\right)+{\displaystyle \frac{1}{4\epsilon }}{\theta }^2{P}_L^{\mathrm{T}}\left(a\right)e\hfill \end{array}$$

3.2. Reinforcement Learning Compensator Design

This paper employs the deep deterministic policy gradient (DDPG) algorithm to design a compensation controller. DDPG effectively addresses continuous control problems [39], enabling precise output of the compensation quantity ${\mathcal{U}}_{\mathrm{comp}}\in \mathbb{R}$, thus overcoming the quantization error issues associated with discrete action algorithms. Additionally, by learning end-to-end disturbance estimation strategies, explicit modeling of system uncertainties such as nonlinear friction and complex disturbances is unnecessary. The temporal difference learning mechanism of the critic network effectively links long-term performance with immediate compensation actions, addressing strategy optimization issues under sparse rewards. Furthermore, the algorithm uses delayed updates of the target networks ${\mu }^{\prime }$ and $Q^{\prime }$ to effectively alleviate the problem of overestimation of Q values caused by correlations of time series data. The overall implementation process of the reinforcement-learning-based compensation fully actuated control (RL+FAC) method proposed in this paper is shown in Figure 3. Its core consists of a high-performance fully actuated main controller (FAC) and a reinforcement learning (RL) compensator based on the DDPG algorithm connected in parallel. The process begins with the real-time acquisition of the system tracking error e and its rate of change $\dot{e}$, which together form the observation state of the reinforcement learning agent. The latest output of the fully actuated controller is introduced as a key reference information. This observation state is simultaneously fed into the Actor network and the Critic network: the Actor network acts as a policy function, mapping the current state to an optimal compensation control quantity, while the Critic network acts as a value function, evaluating the long-term expected return of executing this action under a specific state. Reinforcement learning outputs control compensation quantities through exploratory strategies, which are then added to the main controller’s output to form the final system control command. This command drives the controlled Risley prism system and obtains new system states to calculate the reward value r. The experience tuples generated from each interaction are stored in the experience replay buffer for subsequent random sampling from a large amount of historical data and batch learning. The learning process is achieved by minimizing the temporal difference error of the Critic network and updating parameters along the policy gradient direction of the Actor network, while synchronizing target network parameters using a soft update strategy to ensure training stability. Through this end-to-end learning mechanism, the RL compensator can adaptively estimate and offset unmodeled dynamics and disturbances within the system, thereby achieving higher-precision optimization on top of the robust foundation provided by the fully actuated controller.

3.2.1. Actor–Critic Network Design

Define the observation state of the reinforcement learning agent as

$$s={[e_t,{\dot{e}}_t,u_{FAC}]}^{\mathsf{T}}$$

The reward function is designed as

$$r={\displaystyle \frac{e_{FAC}-e_{RL+FAC}}{e_{FAC}+ϵ}}$$

where $e_t=y_d-y$ is the tracking error, $y_d$ is the desired trajectory, ${\dot{e}}_t={\displaystyle \frac{d{e}_t}{dt}}$ is the error change rate, $u_{FAC}$ is the current output of the fully actuated controller, $e_{FAC}$ is the error of the fully actuated controller, $e_{RL+FAC}$ is the error after reinforcement learning compensation, and $ϵ$ is set to a very small positive number to ensure that the denominator is not 0.

The Actor network uses a deep feedforward architecture to process three-dimensional state inputs. Features are extracted through two fully connected layers, each followed by ReLU activation and batch normalization operations to enhance generalization capabilities. Finally, a linear output layer with $tanh$ activation generates a bounded compensation quantity $u_{\mathrm{comp}}\in [\min ,\max ]$.

The design of the Critic network is based on a dual-stream architecture. The state branch and action branch process input features through independent fully connected layers, which are then concatenated and fused after ReLU activation. Finally, the joint layer regresses the action value function $Q(s,a|{\theta }^Q)$. Parameter updates are based on minimizing the temporal difference objective, and the chain rule drives the network to accurately evaluate the long-term benefits of compensation strategies.

3.2.2. Algorithm Implementation

The DDPG algorithm breaks sample correlation through experience replay to improve data utilization efficiency. Meanwhile, it uses a target network to calculate benchmark values to solve temporal difference instability. The Critic network is updated by minimizing temporal difference errors, and the Actor network optimizes strategy parameters using the policy gradient theorem. The target network uses soft updates to achieve gradual synchronization. During the learning process, it combines exponentially decaying Gaussian noise to dynamically balance exploration and exploitation, ensuring stable convergence of training.

The Experience Replay mechanism solves the problem of sequence sample relevance by storing and reusing historical experience data. In the specific implementation of this paper, the reinforcement learning agent generates a transition tuple at each step of the interaction process:

$${\tau }_t=(s_t,a_t,r_t,s_{t+1})$$

where $s_t$ denotes the current state vector, $a_t$ is the action selected by the agent, $r_t$ is the immediate reward, and $s_{t+1}$ is the new state after the transition. These transferred samples are stored in an orderly manner in a circular buffer $\mathcal{B}$, whose capacity is set to $|\mathcal{B}|={10}^7$. This design enables the algorithm to learn from a large number of independent and identically distributed samples, breaking time correlation and improving data utilization efficiency. When the network is updated, the system uniformly and randomly samples batch data from the buffer to ensure the stability of the training process.

Target value calculation is a core component of temporal difference learning, with the aim of providing reliable learning targets for the Critic network. The calculation formula is

$$y_i=r_i+\gamma Q^{\prime }({\mathbf{s}}_{i+1},{\mu }^{\prime }({\mathbf{s}}_{i+1}|{\theta }^{\mu \prime })|{\theta }^{Q\prime })$$

where $\gamma =0.99$ is the discount factor, $Q^{\prime }$ is the target Critic network, and ${\mu }^{\prime }$ is the target Actor network. Both use a delayed update mechanism to solve the instability problem of the bootstrapping method by separating the target network from the online network. The target value $y_i$ quantifies the immediate reward $r_i$ and the expected cumulative return of the next state, providing a benchmark reference for updating the Critic network.

The Critic network acts as an evaluator of system value, updating by minimizing the temporal difference error:

$$\mathcal{L}({\theta }^Q)={\displaystyle \frac{1}{N_b}}\sum _{i=1}^{N_b}{\left(y_i-Q({\mathbf{s}}_i,a_i|{\theta }^Q)\right)}^2$$

where $N_b=512$ is the batch size, ensuring that each update has sufficient samples.

Update parameters using gradient descent:

$${\theta }^Q\leftarrow {\theta }^Q-{\alpha }_c{\nabla }_{{\theta }^Q}\mathcal{L}\left({\theta }^Q\right)$$

The learning rate is set to ${\alpha }_c={10}^{-4}$, and gradient clipping constraints $\parallel {\nabla }_{{\theta }^Q}{\mathcal{L}\parallel }_2\le 1$ are implemented to prevent gradient explosion during backpropagation. The accurate evaluation of value by the Critic network is key to the strategy optimization and algorithm convergence of the Actor network.

The Actor network is a policy generator, and its optimization objective is to maximize the expected cumulative return. The policy gradient calculation uses the deterministic policy gradient theorem:

$${\nabla }_{{\theta }^{\mu }}J\approx {\displaystyle \frac{1}{N_b}}\sum _{i=1}^{N_b}{\nabla }_{a}Q\left(\mathbf{s},a|{\theta }^Q\right){|}_{a=\mu \left({\mathbf{s}}_i\right)}{\nabla }_{{\theta }^{\mu }}\mu ({\mathbf{s}}_i|{\theta }^{\mu })$$

The parameters are updated to

$${\theta }^{\mu }\leftarrow {\theta }^{\mu }+{\alpha }_a{\nabla }_{{\theta }^{\mu }}J$$

where ${\alpha }_a={10}^{-5}$ is the learning rate of the Actor network, which maintains a ratio of ${\alpha }_a/{\alpha }_c=0.1$ with the learning rate of the Critic network. This method ensures that the Critic network converges first, providing a more accurate value assessment for the policy gradient.

The target network adopts a soft update strategy:

$${\theta }^{Q\prime }\leftarrow \tau {\theta }^Q+(1-\tau ){\theta }^{Q\prime }$$

$${\theta }^{\mu \prime }\leftarrow \tau {\theta }^{\mu }+(1-\tau ){\theta }^{\mu \prime }$$

The update coefficient $\tau$ controls the update speed of the target network, allowing the target parameters to slowly track the online network parameters. This process is performed at each training step to maintain synchronization between the target network and the online network. The soft update mechanism introduces an inertial effect to improve the stability of the learning process and avoid problems such as policy oscillation and Q-value divergence.

In exploring strategy design, a time-decaying Gaussian noise injection mechanism is used to balance the conflict between exploration and exploitation in the reinforcement learning process:

$$a_t=\mu (s_t|{\theta }^{\mu })+\mathcal{N}(0,{\sigma }_t)$$

where ${\sigma }_t={\sigma }_{0}exp(-\beta t)$ is the adaptive noise standard deviation, the initial value ${\sigma }_0=0.1$ ensures that the state space can be fully explored during the early stages of training, and the decay coefficient $\beta ={10}^{-4}$ achieves exponential decay of the noise amplitude. As the number of training steps increases, the noise intensity ${\sigma }_t$ gradually approaches 0, causing the strategy to gradually shift from exploration to exploitation. This method effectively solves the problem of insufficient exploration in deterministic strategies while ensuring control accuracy in the later stages of training.

3.3. Stability Analysis

The second-order nonlinear system with uncertainty disturbances in Section 2.2 is represented as a fully driven system model:

$$\ddot{y}=f(y,\dot{y})+\Delta f(y,\dot{y})+bu$$

(10)

where $f(y,\dot{y})$ is a continuous vector function of the system, $\Delta f(y,\dot{y})$ is the uncertain disturbance to the system, b is the input gain of the system, y is the state of the system, and $u\in \mathbb{R}$ is the input of the system. Assume the existence of a non-negative continuous function $\rho (x,\dot{x})$ such that $\parallel \Delta f(y,\dot{y}))\parallel \le \rho (y,\dot{y})$.

The following important lemmas will be used in this proof [38]:

Lemma 1.

There exist two real numbers, x and y, with $y>0$. Then the following relation holds:

$$y\ge x-{\displaystyle \frac{x^2}{4y}}$$

Lemma 2.

Suppose that $A\in {\mathbb{R}}^{n\times n}$ satisfies $\mathrm{Re}{\lambda }_i\left(A\right)\le -{\displaystyle \frac{\mu }{2}},i=1,2,\dots ,n$, where $\mu >0$. Then there exists a positive definite matrix $P\in {\mathbb{R}}^{n\times n}$ satisfying

$$A^{\mathrm{T}}P+PA\le -\mu P$$

(11)

Lemma 3.

When (11) holds, for any $\mu >0$, there exists a positive definite matrix $P\left(A_{0\sim n-1}\right)=[P_1{P}_2\dots P_n]$, $P_i\in {\mathbb{R}}^{nr\times r}$ satisfying

$${\mathsf{\Phi }}^{\mathrm{T}}\left(A_{0\sim n-1}\right)P\left(A_{0\sim n-1}\right)+P\left(A_{0\sim n-1}\right)\mathsf{\Phi }\left(A_{0\sim n-1}\right)\le -\mu P\left(A_{0\sim n-1}\right)$$

For the sake of simplicity, introduce

$$P_L\left(A_{0\sim n-1}\right)=P\left(A_{0\sim n-1}\right)\left[\begin{array}{c}0\\ I_r\end{array}\right]=P_n$$

This paper uses the reinforcement learning DDPG algorithm to estimate system disturbances and modeling uncertainties $\Delta f$, with the estimated value as $\widehat{\Delta f}$.

The convergence of the algorithm can be ensured by rationally designing the network parameters and reward functions [40].

Assuming that the final convergence error is

$$\Delta f-\widehat{\Delta f}=\delta f<\Delta f(x,\dot{x})\le \rho$$

(12)

The fully actuated control design based on reinforcement learning compensation is as follows:

$$\left\{\begin{array}{cc}u=-b^{-1}\left(u_1+u_2\right)\hfill & \\ & \\ u_1=f(y,\dot{y})+A_{0\sim 1}{y}^{(0\sim 1)}\hfill & \\ & \\ u_2=-\widehat{\Delta f}+{\displaystyle \frac{1}{4\epsilon }}{\rho }^2(y,\dot{y})P_L^T\left(A_{0\sim 1}\right)y^{(0\sim 1)}\hfill \end{array}\right.$$

(13)

Substituting Equation (13) into Equation (10), the closed-loop system is obtained as follows:

$$\ddot{y}+A_1\dot{y}+A_{0}y=\Delta f(y,\dot{y})-\widehat{\Delta f}-{\displaystyle \frac{1}{4\epsilon }}{\rho }^2(y,\dot{y})P_L^{\mathrm{T}}\left(A_{0\sim 1}\right)y^{(0\sim 1)}$$

(14)

Convert the closed-loop system in Equation (14) to state-space equation form:

$$\left[\begin{array}{c}\dot{y}\\ \ddot{y}\end{array}\right]=\mathsf{\Phi }\left(A_{0\sim 1}\right)\left[\begin{array}{c}y\\ \dot{y}\end{array}\right]+\left[\begin{array}{c}0\\ \Delta f(y,\dot{y})-\widehat{\Delta f}-{\displaystyle \frac{1}{4\epsilon }}{\rho }^2(y,\dot{y})P_L^{\mathrm{T}}\left(A_{0\sim 1}\right)y^{(0\sim 1)}\end{array}\right]$$

(15)

where $\mathsf{\Phi }\left(A_{0\sim 1}\right)$ is the parameter matrix designed to give the system the desired feature structure. When Equation (11) is satisfied, there exists a positive definite matrix P such that Equation (6) holds. Then, construct the Lyapunov function:

$$V\left(y^{(0\sim 1)}\right)={\displaystyle \frac{1}{2}}{\left(y^{(0\sim 1)}\right)}^{\mathrm{T}}P{y}^{(0\sim 1)}$$

(16)

Substituting Equations (11) and (15) into Equation (16), the result is

$$\begin{array}{cc}\hfill \dot{V}\left(y^{(0\sim 1)}\right)& ={\displaystyle \frac{1}{2}}{\left({\dot{y}}^{(0\sim 1)}\right)}^{\mathrm{T}}P{y}^{(0\sim 1)}+{\displaystyle \frac{1}{2}}{\left(y^{(0\sim 1)}\right)}^{\mathrm{T}}P{\dot{y}}^{(0\sim 1)}\hfill \\ \hfill & ={\displaystyle \frac{1}{2}}{\left(\mathsf{\Phi }\left(A_{0\sim 1}\right)y^{(0\sim 1)}+\left[\Delta f(y,\dot{y})-\widehat{\Delta f}-{\displaystyle \frac{1}{4\epsilon }}{\rho }^2(y,\dot{y})P_L^T\left(A_{0\sim 1}\right)y^{(0\sim 1)}\right]\right)}^{\mathrm{T}}P{y}^{(0\sim 1)}\hfill \\ \hfill & +{\displaystyle \frac{1}{2}}{\left(y^{(0\sim 1)}\right)}^{\mathrm{T}}P\left(\mathsf{\Phi }\left(A_{0\sim 1}\right)y^{(0\sim 1)}+\left[\begin{array}{c}0\\ \delta f(y,\dot{y})-{\displaystyle \frac{1}{4\epsilon }}{\rho }^2(y,\dot{y})P_L^T\left(A_{0\sim 1}\right)y^{(0\sim 1)}\end{array}\right]\right)\hfill \\ \hfill & ={\displaystyle \frac{1}{2}}{\left(y^{(0\sim 1)}\right)}^{\mathrm{T}}\left({\mathsf{\Phi }}^{\mathrm{T}}\left(A_{0\sim 1}\right)P+P\mathsf{\Phi }\left(A_{0\sim 1}\right)\right)y^{(0\sim 1)}\hfill \\ \hfill & +{\left(y^{(0\sim 1)}\right)}^{\mathrm{T}}P\left[\delta f(y,\dot{y})-{\displaystyle \frac{1}{4\epsilon }}{\rho }^2(y,\dot{y})P_L^T\left(A_{0\sim 1}\right)y^{(0\sim 1)}\right]\hfill \end{array}$$

$$\begin{array}{cc}\hfill \dot{V}\left(y^{(0\sim 1)}\right)& =-{\displaystyle \frac{1}{2}}{\left(y^{(0\sim 1)}\right)}^{\mathrm{T}}\mu P{y}^{(0\sim 1)}\hfill \\ \hfill & +{\left(y^{(0\sim 1)}\right)}^{\mathrm{T}}{P}_L\left[\delta f(y,\dot{y})-{\displaystyle \frac{1}{4\epsilon }}{\rho }^2(y,\dot{y})P_L^{\mathrm{T}}\left(A_{0\sim 1}\right)y^{(0\sim 1)}\right]\hfill \\ \hfill & \le -{\displaystyle \frac{1}{2}}{\left(y^{(0\sim 1)}\right)}^{\mathrm{T}}\mu P{y}^{(0\sim 1)}\hfill \\ \hfill & +{\left(y^{(0\sim 1)}\right)}^{\mathrm{T}}{P}_L\left(A_{0\sim n-1}\right)\left[\delta f(y,\dot{y})-{\displaystyle \frac{1}{4\epsilon }}{\rho }^2(y,\dot{y})P_L^{\mathrm{T}}\left(A_{0\sim 1}\right)y^{(0\sim 1)}\right]\hfill \\ \hfill & \le -\mu V+{\left(y^{(0\sim 1)}\right)}^{\mathrm{T}}{P}_L\left(A_{0\sim n-1}\right)\delta f(y,\dot{y})\hfill \\ \hfill & -{\displaystyle \frac{{\rho }^2(y,\dot{y})}{4\epsilon }}{\left(y^{(0\sim 1)}\right)}^{\mathrm{T}}{P}_L\left(A_{0\sim n-1}\right)P_L^{\mathrm{T}}\left(A_{0\sim n-1}\right)y^{(0\sim 1)}\hfill \\ \hfill & \le -\mu V+∥{\left(y^{(0\sim 1)}\right)}^{\mathrm{T}}{P}_L\left(A_{0\sim n-1}\right)∥\delta f(y,\dot{y})\hfill \\ \hfill & -{\displaystyle \frac{{\rho }^2}{4\epsilon }}{∥{\left(y^{(0\sim 1)}\right)}^{\mathrm{T}}{P}_L\left(A_{0\sim n-1}\right)∥}^2\hfill \end{array}$$

From Lemma 1, it follows that

$$∥{\left({\mathrm{y}}^{(0\sim 1)}\right)}^{\mathrm{T}}{P}_L\left(A_{0\sim n-1}\right)∥\rho -{\displaystyle \frac{{\rho }^2}{4\epsilon }}{∥{\left({\mathrm{y}}^{(0\sim 1)}\right)}^{\mathrm{T}}{P}_L\left(A_{0\sim n-1}\right)∥}^2\le \epsilon$$

Because $\delta f<\rho$, therefore,

$$\begin{array}{cc}\hfill & ∥{\left(y^{(0\sim 1)}\right)}^{\mathrm{T}}{P}_L\left(A_{0\sim n-1}\right)∥\delta f(y,\dot{y})-{\displaystyle \frac{{\rho }^2}{4\epsilon }}{∥{\left(y^{(0\sim 1)}\right)}^{\mathrm{T}}{P}_L\left(A_{0\sim n-1}\right)∥}^2\hfill \\ \hfill & <∥{\left(x^{(0\sim 1)}\right)}^{\mathrm{T}}{P}_L\left(A_{0\sim n-1}\right)∥\rho -{\displaystyle \frac{{\rho }^2}{4\epsilon }}{∥{\left(x^{(0\sim 1)}\right)}^{\mathrm{T}}{P}_L\left(A_{0\sim n-1}\right)∥}^2\hfill \end{array}$$

$$∥{\left(y^{(0\sim 1)}\right)}^{\mathrm{T}}{P}_L\left(A_{0\sim n-1}\right)∥\delta f(y,\dot{y})-{\displaystyle \frac{{\rho }^2}{4\epsilon }}{∥{\left(y^{(0\sim 1)}\right)}^{\mathrm{T}}{P}_L\left(A_{0\sim n-1}\right)∥}^2<\epsilon$$

Let

$$∥{\left({\mathrm{y}}^{(0\sim 1)}\right)}^{\mathrm{T}}{P}_L\left(A_{0\sim n-1}\right)∥\delta f(y,\dot{y})-{\displaystyle \frac{{\rho }^2}{4\epsilon }}{∥{\left({\mathrm{y}}^{(0\sim 1)}\right)}^{\mathrm{T}}{P}_L\left(A_{0\sim n-1}\right)∥}^2=\eta$$

then the Lyapunov derivative satisfies

$$\dot{V}\left(x^{(0\sim 1)}\right)\le -\mu V+\eta$$

Solve the differential inequality:

$$V\left(t\right)\le {\displaystyle \frac{\eta }{\mu }}-{\displaystyle \frac{\eta }{\mu }}{e}^{-\mu t}+V\left(0\right)e^{-\mu t}$$

which ultimately converges to

$$V\left(t\right)\le {\displaystyle \frac{\eta }{\mu }}-{\displaystyle \frac{\eta }{\mu }}{e}^{-\mu t}+V\left(0\right)e^{-\mu t}\to {\displaystyle \frac{\eta }{\mu }},t\to \infty$$

In the original robust fully actuated control method, when the robust control term is $u_2={\displaystyle \frac{1}{4\epsilon }}{\rho }^2(y,\dot{y})P_L^T\left(A_{0\sim 1}\right){\mathrm{y}}^{(0\sim 1)}$, it can only converge to ${\displaystyle \frac{\epsilon }{\mu }}$. However, the robust control term $u_2=-\widehat{\Delta f}-{\displaystyle \frac{1}{4\epsilon }}{\rho }^2(y,\dot{y})P_L^T\left(A_{0\sim 1}\right){\mathrm{y}}^{(0\sim 1)}$ after reinforcement learning compensation can allow the system to ultimately converge to ${\displaystyle \frac{\eta }{\mu }}<{\displaystyle \frac{\epsilon }{\mu }}$.

4. Results and Discussion

This section verifies the effectiveness of the method described in this paper through experimental design. The fully actuated control and compensator are designed according to the method described in Section 3. In practical applications, there is usually a requirement for no blind spots near the optical axis, so this paper selects two prisms with the same wedge angles and index of refraction for testing. The model parameters are shown in

Table 1. System model parameters.

Parameters	Meaning	Value
B	$Damping coefficient$ (N·m·s/rad)	$6.5\times {10}^{-4}$
$R_a$	$Armature resistance \left(\mathsf{\Omega }\right)$	1
J	$Total moment of inertia$ (kg·m2)	$4.5\times {10}^{-4}$
$K_e$	$Damping coefficient$ (N·m·s/rad)	$0.12$
$K_t$	$Back EMF constant$ (V·s/rad)	$0.12$
n	$Index of Refraction$	$1.51$
$\alpha$	$Wedge Angle \left(\mathrm{deg}\right)$	16

.

In the control of the Risley prism system, the prism rotation angle is obtained by integrating the angular velocity of the motor. In many beam scanning and tracking tasks, high-speed prism rotation is required. This section first tests the angular velocity control performance of the motor model.

Figure 4 shows the results of the high-speed angular velocity signal tracking $1000sin(\pi t+{\displaystyle \frac{\pi }{2}})$, and Figure 5 shows the results of the low-speed angular velocity signal tracking $5sin(\pi t+{\displaystyle \frac{\pi }{2}})$. The beam pointing accuracy of the Risley prism system is extremely sensitive to motor speed fluctuations, as even small speed fluctuations are amplified by nonlinear refraction geometry into macroscopic beam jitter. The RL compensator uses an Actor–Critic network to estimate unmodeled disturbances in real time, dynamically compensating the output of the FAC controller to reduce angular velocity tracking error and cumulative absolute error. This demonstrates that the method effectively suppresses nonlinear factors in motor dynamics, laying the foundation for high-precision beam control.

The Risley prism beam control system is then tested. Six sets of tests are designed based on several different movements of the beam deviation angle and azimuth angle, which are shown in

Table 2. Risley prism control test information.

Test	Deviation Deflection Range	Azimuthal Deflection Range
Test 1 $(n=1.51,\alpha =16)$	$10°$	$100°$
Test 2 $(n=1.51,\alpha =16)$	$0°$–16°	80°
Test 3 $(n=1.51,\alpha =16)$	$8°$	$0°$–360°
Test 4 $(n=1.51,\alpha =16)$	$0°$–16°	$20°$–220°
Test 5 $(n=1.51,\alpha =16)$	$0°$–16°	$0°$–360°
Test 6 $(n=1.78,\alpha =10)$	$0°$–12°	$0°$–360°

:

Test 1: The target deviation angle and target azimuth angle are constant. Figure 6 shows the error in tracking the target pointing angle, and Figure 7 shows the absolute error integral during the tracking process. Both methods can converge the root error to near zero and remain stable in a short time. However, RL+FAC converges faster because the reinforcement learning compensator can quickly identify and compensate for static errors in the system, thereby stabilizing the beam at the target position in a short period of time. The reduction in cumulative absolute error indicates that this method can reduce long-term drift, which is critical for maintaining a stable optical link in laser communications. In addition, the FAC method can converge the tracking error of the deflection angle to within $1.00\times {10}^{-4}$, while the FAC+RL method can converge the error to within $1.00\times {10}^{-5}$, reducing the steady-state error by $90\%$. For azimuth, both methods can converge the error to within $1.00\times {10}^{-6}$.

Test 2 ∼ Test 3: Figure 8 and Figure 9 show the results of the target azimuth angle remaining constant while the deviation angle deviates within $0°$–$16°$. The deviation angle tracking signal is $8sin\left(2\pi t\right)+8$. Figure 10 and Figure 11 show the results of the target deviation angle remaining constant while the azimuth angle deviates within $0°$–$360°$. The azimuth tracking signal is $360sin\left(\pi t\right)$. In Test 2, for the deflection angle, the FAC method can converge the tracking error to within $1.00\times {10}^{-2}$, while the FAC+RL method can converge the tracking error to within $1.00\times {10}^{-4}$, representing a two-order-of-magnitude improvement. For the azimuth angle, the FAC method converges the tracking error to within $0.21$, while the FAC+RL method converges the tracking error to within $0.19$, reducing the steady-state error by $9.52\%$. In Test 3, for the deflection angle, the FAC method can converge the tracking error to within $8.00\times {10}^{-7}$, while the FAC+RL method can converge the tracking error to within $5.00\times {10}^{-7}$, reducing the steady-state error by $37.5\%$. For the azimuth angle, the FAC method converges the tracking error to within $3.00\times {10}^{-2}$, while the FAC+RL method converges the tracking error to within $2.00\times {10}^{-2}$, reducing the steady-state error by $33.3\%$. The results show that when the azimuth angle or deflection angle changes independently, the cumulative error of RL+FAC is smaller. This is due to the real-time learning of the compensator for nonlinear mapping: there is a complex trigonometric relationship between the prism rotation angle and the beam direction (Equations (1)–(7)). RL learns and compensates through low-dimensional FAC output, effectively suppressing the nonlinearity of the system.

Test 4 ∼ Test 5: The target deviation angle and azimuth angle move within a certain range. Figure 12 and Figure 13 show the effect of the azimuth angle moving within $20°$–$220°$ and the deviation angle moving within $0°$–$16°$. The azimuth angle tracking signal is $100sin\left(\pi t\right)+120$, and the deviation angle tracking signal is $8sin\left(2\pi t\right)+8$. Figure 14 and Figure 15 show the effect of the azimuth angle and the deviation angle simultaneously moving near the limit range. The azimuth angle moves within $0°$–$360°$, while the deviation angle moves within $0°$–$16°$. The azimuth angle tracking signal is $360sin\left(\pi t\right)$, and the deviation angle tracking signal is $8sin\left(2\pi t\right)+8$. In Test 4, for the deflection angle, the FAC method can converge the tracking error to within $4.00\times {10}^{-6}$, while the FAC+RL method can converge the tracking error to within $3.00\times {10}^{-6}$, reducing the steady-state error by $25\%$. For the azimuth angle, the FAC method converges the tracking error to within $0.15$, while the FAC+RL method converges the tracking error to within $0.1$, reducing the steady-state error by $33.3\%$. In Test 5, for the deflection angle, the FAC method can converge the tracking error to within $0.01$, while the FAC+RL method can converge the tracking error to within $1.00\times {10}^{-3}$, reducing the steady-state error by $90\%$. For the azimuth angle, the FAC method converges the tracking error to within $0.03$, while the FAC+RL method converges the tracking error to within $0.01$, reducing the steady-state error by $66.67\%$. From the results, it can be seen that reinforcement learning can effectively compensate for the controller, enabling the compensated RL+FAC method to maintain faster convergence speed, higher control accuracy and smaller cumulative error, both within a certain range of deflection and near the limit angle range. It is noted that the enlarged illustration in Figure 12a shows some transient rises or falls, which occur when the tracking trajectory reaches the maximum deflection angle. Taking the deflection angle as an example, the target signal is $8sin\left(2\pi t\right)+8$, which reaches a maximum value at $t=1.75$ s. This position corresponds to the region where the Risley prism system exhibits its strongest nonlinearity, making control more challenging. This jitter arises from the interaction between the FAC controller and the inherent strong nonlinearity of the Risley prism system, akin to introducing a significant internal disturbance into the control loop. This is precisely the manifestation of the FAC’s ability to suppress nonlinearity.

Test 6: Change the prism parameters for the experiment, change the index of refraction to $n=1.78$, and change the prism wedge angle to $\alpha =10°$. The azimuth angle moves within $0°$–$360°$, and the deviation angle moves within $0°$–$12°$. Figure 16 and Figure 17 show the experimental results. In Test 6, for the deflection angle, the FAC method converges the tracking error to within $3.00\times {10}^{-6}$, while the FAC+RL method converges the tracking error to within $2.00\times {10}^{-6}$, reducing the steady-state error by $33.3\%$. For the azimuth angle, the FAC method converges the tracking error to within $0.04$, while the FAC+RL method converges the tracking error to within $0.02$, reducing the steady-state error by $50\%$. The results show that even when the prism parameters are changed, causing changes in the system parameters, reinforcement learning can still effectively compensate for the control system, and RL+FAC still has a smaller overall error. This proves that the method is robust and does not rely on accurate models and can compensate for changes in optical characteristics caused by differences in prism manufacturing through online learning.

Regarding the stability and convergence analysis of the proposed FAC+RL method, a detailed theoretical proof based on Lyapunov stability theory has been provided in Section 3.3. To further evaluate the practical repeatability and robustness of the algorithm, we conducted 10 repeated tracking experiments under the same conditions. The average value of the integrated absolute error (IAE) was calculated, with the results shown in

Table 3. The average value of the integrated absolute error of 10 repeated experiments.

	IAE of Deflection Angle (deg)		IAE of Azimuth Angle (deg)
Test	FAC	FAC+RL	FAC	FAC+RL
Test 1	0.3743	0.3429	16.9534	15.8186
Test 2	0.1290	0.1161	13.6840	13.1700
Test 3	0.0436	0.0385	2.1370	1.9340
Test 4	0.1423	0.1383	21.2093	20.1940
Test 5	0.1308	0.1214	2.1394	1.9322
Test 6	0.0844	0.0736	2.5133	2.3081

. The statistical results indicate that the FAC+RL method exhibits lower cumulative error in all tests. This not only demonstrates that the method can improve the control accuracy of the system but also highlights the excellent repeatability and reliability of the proposed method.

Based on the above experiments, the fully actuated control method based on reinforcement learning outperformed ordinary fully actuated control in all tests. The reinforcement learning compensator designed in this paper uses the error and the FAC control amount before compensation as observations and uses the relative error with FAC as the reward function for strategy learning. Through the Actor–Critic network, the control strategy is dynamically adjusted, and dynamic compensation is performed based on the FAC method to predictively suppress nonlinear coupling effects, which can further reduce model errors. It also has a good compensation effect at positions where the beam deviation angle is large, which is where the nonlinearity is most significant. At the same time, the fully actuated controller output can be used as a low-dimensional observation signal to a certain extent as prior knowledge for reinforcement learning, alleviating the problem of high convergence training complexity.

There are also other control studies based on the Risley prism, such as the robust control method proposed by Ma et al. [24] based on an extended state observer. This method is fundamentally model-driven, with its core lying in precisely estimating and real-time compensating for total disturbances in the system through a linear observer. This is an efficient and mature reactive compensation scheme that can effectively reduce peak errors. However, its performance ceiling is limited by the accuracy of the offline-established mathematical model and the inherent strong nonlinearity exhibited by the Risley prism near its maximum deflection angle. The method proposed in this paper adopts a data-driven reinforcement learning (RL) architecture that does not rely on precise prior models. Instead, it actively learns the system’s nonlinear dynamics and disturbance patterns through online interaction, thereby generating predictive feedforward compensation signals. This means that for regular nonlinear behavior, the RL agent can learn and anticipate its impact rather than suppressing it after it causes errors. The goal is to optimize accuracy based on high-performance fully actuated control. The composite axis control system proposed by Xia et al. [41], based on the Risley prism and fast control mirror (FSM), provides an efficient solution to the contradiction between tracking accuracy and response speed from a system architecture perspective. It uses the FSM’s high bandwidth to suppress high-frequency disturbances and feeds the FSM deflection angle back to the Risley prism for collaborative compensation, ultimately achieving ultra-high tracking accuracy at the sub-arcsecond level. Unlike the core approach of this study, this work represents an increase in system performance through increased hardware. In contrast, the RL+FAC control strategy studied in this paper deeply explores the performance potential of the existing single Risley prism hardware architecture through data-driven reinforcement learning algorithms without altering the hardware architecture. The former demonstrates significant advantages in terms of absolute accuracy metrics but introduces higher system complexity and cost; the latter focuses on enhancing system performance by intelligently compensating for nonlinearities and disturbances within a single Risley system architecture.

5. Conclusions

High-precision pointing control of the beam is one of the key technologies in fields such as laser communication. This paper proposes a fully actuated control method based on reinforcement learning compensation for nonlinear control problems in Risley prism systems. This method combines fully actuated control theory with reinforcement learning methods: reinforcement learning methods are based on Actor–Critic networks, which further estimate model nonlinearity and uncertainty to compensate for the controller and improve control performance. At the same time, the output and error information of the fully actuated controller are used as reinforcement learning observations to reduce learning complexity. This method takes advantage of the strengths of both methods in dealing with nonlinear problems while alleviating the difficulties of insufficient control accuracy and complex convergence of reinforcement learning in fully actuated control methods. The stability of the method is verified through detailed theoretical analysis. Finally, the fully actuated control method based on reinforcement learning compensation is verified experimentally to have smaller cumulative error, faster convergence speed, and higher accuracy.