Vibration and Position Control of a Two-Link Flexible Manipulator Using Reinforcement Learning

Abstract

In recent years, industries have increasingly emphasized the need for high-speed, energy-efficient, and cost-effective solutions. As a result, there has been growing interest in developing flexible link manipulator robots to meet these requirements. However, reducing the weight of the manipulator leads to increased flexibility which, in turn, causes vibrations. This research paper introduces a novel approach for controlling the vibration and motion of a two-link flexible manipulator using reinforcement learning. The proposed system utilizes trust region policy optimization to train the manipulator’s end effector to reach a desired target position, while minimizing vibration and strain at the root of the link. To achieve the research objectives, a 3D model of the flexible-link manipulator is designed, and an optimal reward function is identified to guide the learning process. The results demonstrate that the proposed approach successfully suppresses vibration and strain when moving the end effector to the target position. Furthermore, the trained model is applied to a physical flexible manipulator for real-world control verification. However, it is observed that the performance of the trained model does not meet expectations, due to simulation-to-real challenges. These challenges may include unanticipated differences in dynamics, calibration issues, actuator limitations, or other factors that affect the performance and behavior of the system in the real world. Therefore, further investigations and improvements are recommended to bridge this gap and enhance the applicability of the proposed approach.

1. Introduction

Current industrial robots generally have their work-plan programmed or hardcoded, so that it is necessary to program the robot from scratch each time the production process changes. This process is time consuming and costly, requiring elaborate technical expertise [1,2]. In addition, conventional industrial robots have high rigidity so that sufficient positioning accuracy can be obtained but, in recent years, high speed and energy efficiency, as well as a reduction in cost for work, is required in the industries. This has led to increasing interest in the production of flexible link manipulator robots. There are several advantages stemming from the use of light-weight flexible-link manipulators, such as faster operation, lower energy consumption, and higher load-carrying capacity for the amount of energy expended [3,4]. Weight reduction is desirable; however, when the manipulator is made lighter, it becomes inevitably flexible, increasing vibration [5,6]. Therefore, it is necessary to consider the flexure and vibration due to the elasticity of the robot-arm.

Vibration and position control has been performed using varying strategies and methodologies, according to the literature. The authors of [7] developed a model-based controller for a flexible manipulator. PID and other hybrid strategies have been applied in [7,8,9]. Other strategies, like fuzzy logic, shape memory alloy (SMA), have also been utilized [7,10,11]. Conventional strategies for position and vibration control, such as PID control, model-based control, and frequency-based control, rely on mathematical models and predefined control algorithms. They require knowledge of system dynamics and parameter tuning. On the other hand, machine learning approaches learn control policies directly from data without explicitly modeling the system. Machine learning methods offer adaptability, the ability to handle complex systems, and do not require precise system models. However, the algorithms have higher implementation complexity, and may require extensive training. The choice between conventional strategies and machine learning models depends on the specific control problem, available system knowledge, and desired performance. In this paper, the focus is on the use of machine learning, specifically, reinforcement learning.

In recent years, reinforcement learning has been attracting attention in robotics and manufacturing industries [12,13,14]. Reinforcement learning (RL) is a type of machine learning capable of complex control schemes without the need for correct labels and large amounts of data. Consequently, RL has proven to be valuable in various applications in robotics. Researchers have explored the use of RL algorithms to enhance control, trajectory planning, grasping, and vibration control, amongst other in the robotics. Some studies in this field include the work by Xiaoming et al. [15], who investigated the optimal control problem of heavy material handling manipulators for agricultural robots with unknown robot parameters.

Xie et al. developed a sliding mode controller using a decoupling method to simultaneously control the angular movements of a three-rigid-link robotic manipulator and the attitude and position of an 8U CubeSat [16]. Zhi-cheng et al. devised a hybrid control strategy combining motion trajectory optimization and piezoelectric active control for a flexible hinged plate [17]. They utilized finite element modeling and experimental system identification to obtain accurate models of the piezoelectric driving and motor acceleration driving of the joint, and trained modal active controllers using an auto-soft-actor-critic RL algorithm.

Authors in [11] tackled the continuous-time optimal tracking control problem of an SMA actuated manipulator with prescribed tracking error constraints and unknown system dynamics. They employed data-driven RL to approximate the solution iteratively, avoiding the need for an SMA manipulator model. Another study is reported in [17] where the authors applied an RL-based active vibration control algorithm to suppress the coupling vibration of a multi-flexible beam coupling system. They developed an accurate model of the system using a combination of finite element modeling and experimental identification. The model was utilized as an RL environment, and a deep deterministic policy gradient RL algorithm based on priority experience replay to train modal controllers.

In a previous study, we proposed a search algorithm that employed convolutional neural networks to map real-world observations (images) to policy-equivalent images trained by RL in a simulated environment [18]. The system was trained in two steps, involving RL policy and a mapping model, which mitigated the challenges associated with sim–real transfer using solely simulated data.

Yuncheng et al. performed vibration control of a flexible manipulator by reinforcement learning. In this case, the control target was a one-link flexible manipulator, which has reduced complexity in the controller [19]. A study reported in [20] used two reinforcement-learning-based compensation methods that enhance control performance by adding a learned correction signal to the nominal input, which is evaluated on aUR5 manipulator with 6-DoF. The authors reported better results when comparing the performance with nominal strategies like model-based prediction algorithms. It is worth noting that the industrial robot in the study is structurally different, carrying loads of up to 5 Kg. Similarly, the authors of [21] utilized a structurally rigid rotary flexible beams to implement reinforcement learning algorithm in a manipulator.

Besides RL, artificial neural networks has been employed in the control of flexible manipulators. Njeri et al. presented a self-tuning control system for a two-link flexible manipulator using neural networks [22]. The neural networks learned the gains of PI controllers to suppress vibrations and track desired joint angles. Their numerical and simulation results demonstrated the effectiveness of the self-tuning control system for high-speed control of a 3D, two-link flexible manipulator. Fuzzy logic, autoregressive models, and other variants of neural networks have also been used as reported in multiple studies [23,24,25]. Other research approaches and considerations towards utilization of RL in robotics can be found in surveys reported in [26,27].

From the literature review, most of these RL adaptations to robots are based on a rigid or semi-rigid robot-arm models. In the cases where RL is utilized in flexible manipulators, the focus has been on single links, as discussed. Thus, little research has been conducted on RL with a low-rigidity two-link robot arm. Therefore, in this paper, the objective is to perform vibration and position control of a flexible manipulator using RL. First, a simulation model is developed that mimics the response and behavior of an actual two-link flexible manipulator. To this end, a 3D model is created considering the characteristics of the physical manipulator in the lab. Next, in order to perform position control, the modeled simulation is used to train the controller with virtual scenarios by reinforcement learning, with the reward set to the distance between the target point and the end effector. Finally, in order to suppress vibrations, simulations with strains added to the learning reward settings are performed to examine the effectiveness of the model in vibration control for a low-rigidity robot arm.

The rest of this paper is divided as follows. Section 2 gives the methods utilized touching on RL formulations. Section 3 gives the simulation of virtual flexible manipulator model with RL, while Section 4 gives the comparison of simulated vibration control with experimental data. Section 5 draws the conclusion of the study.

2. Materials and Methods

The paper proposes usage of reinforcement learning (RL), a type of machine learning, where the agent observes the current state in a certain environment and acquires the optimal action by trial and error [12,28]. In RL, if a certain action is stochastically taken from the current state, a rewarded is issued by the environment, and hence the learning process. RL takes the action that maximizes the overall reward, not the reward alone. Considering time t and discount rate γ, the total reward $G_t$ is the summation of the future discounted rewards as shown in (1). In this case, r_t is the reward at time t.

$$G_t\stackrel{\scriptscriptstyle\mathrm{def}}{=}{r}_{t+1}+\gamma r_{t+2}+{\gamma }^2{r}_{t+3}+\cdots +{\gamma }^{T-t-1}{r}_T.$$

(1)

Success in RL is therefore dependent on the algorithm used to find the policy that maximizes expected returns.

2.1. Policy Gradient Method

The policy gradient method is a technique in RL that relies on optimization of the parameterized policy to maximize cumulative (expected) reward, using a gradient method [29,30]. Given a set of Actions a, states s, and policy ${\pi }_{\theta }$, the policy is assumed to be parameterized by the model parameter $\theta$. The probability of transition to state s according to the policy is $d^{{\pi }_{\theta }}\left(s\right).$ The probability of taking action a in state s (action probability) is ${\pi }_{\theta }\left(a|s\right)$, and the action value is $Q^{{\pi }_{\theta }}(s,a)$. Let (s, a) be the slope of the expected value $J\left(\theta \right)$ of the policy. It can be expressed as:

$$\nabla J\left(\theta \right)=\sum _{s\in S}{d}^{{\pi }_{\theta }}\left(s\right)\sum _{a\in A}{\nabla \pi }_{\theta }\left(a|s\right)Q^{{\pi }_{\theta }}\left(s,a\right)$$

(2)

The model parameter $\theta$ of the policy is updated using the learning rate $\alpha$ and the gradient $\nabla J\left(\theta \right)$, as shown in (3).

$${\theta }_{t+1}={\theta }_t+\alpha \nabla J\left(\theta \right)$$

(3)

In this research, the policy gradient method is used because the action space is continuous. The policy ${\pi }_{\theta }$ uses a Gaussian policy, defined as a multidimensional normal distribution. The Gaussian policy is expressed in (4), with d as the number of dimensions.

$$\begin{array}{c}\pi \left(a|s,\theta \right)=\frac{1}{{\left(2\pi \right)}^{\frac{d}{2}}{\left|{\sum }_{\theta }\left(s\right)\right|}^{\frac{1}{2}}}\times \\ \exp \left(-\frac{1}{2}{\left(a-{\mu }_{\theta }\left(s\right)\right)}^T{\sum }_{\theta }^{-1}\left(s\right)\left(a-{\mu }_{\theta }\left(s\right)\right)\right)\end{array}$$

(4)

The state $s$ is input, the mean ${\mu }_{\theta }$ and the variance ${\sigma }_{\theta }$ are output, and the action $a$ is sampled from the 12-dimensional normal distribution created by the mean and variance. Other input parametric settings of the policy are shown in

Table 1. Policy parameter settings.

Parameter	Value
Input channels.	215
Dimensions of the action space.	12
Hidden channels.	200
Hidden layers.	2
Scale of weight initialization of the mean layer.	0.01
Initial value the var parameter.	0

.

One of the drawbacks associated with the Gaussian policy gradient method is over-learning [30]. For this reason, trust region policy optimization is applied to remedy for overlearning challenge in the model.

2.2. Trust Region Policy Optimization

Trust region policy optimization (TRPO) is a reinforcement learning algorithm that addresses the issue of over-learning in the Gaussian policy gradient method. Over-learning occurs when the policy is updated too aggressively, leading to instability and convergence problems. TRPO solves this challenge by imposing a trust region with a surrogate objective function that approximates the expected return. To find the policy update that maximizes the surrogate objective while staying within the trust region, TRPO solves a constrained optimization problem using techniques like conjugate gradients. It also employs a backtracking line search to gradually reduce the step size of the update until it satisfies the trust region constraint.

TRPO addresses overfitting in the policy function by constraining the Kullback–Leibler (KL) divergence between the old and updated policy distributions [31]. The equation for the KL updated is shown in (5), where E is the expectation in time t.

$$E_t\left[KL\left[{\pi }_{{\theta }_{old}}\left(·|s_t\right),{\pi }_{\theta }\left(·|s_t\right)\right]\right]\le \mathsf{\delta }$$

(5)

The relative value of action $A_t$ can be expressed as (6) using the state value $V^{\pi }\left(s\right)$. A_t represents the advantage of taking action a_t in state s_t compared to the baseline value V^π (s).

$$A_t=Q^{{\pi }_{\theta }}\left(s,a\right)-V^{\pi }\left(s\right)$$

(6)

Under the KL constraint, TRPO aims to maximize the expectation of the ratio between the current policy π_θ (a_t│s_t) and the old policy π__θold (a_t│s_t) multiplied by the advantage A_t, as depicted in Equation (7). This maximization encourages an increase in the action value A_t while respecting the policy divergence constraint.

$$\underset{\theta }{\max }{E}_t\left[\frac{{\pi }_{\theta }\left(a_t|s_t\right)}{{\pi }_{{\theta }_{old}}\left(a_t|s_t\right)}{A}_t\right]$$

(7)

Parameter settings with the optimization algorithm are shown in

Table 2. TRPO parameter settings.

Parameter	Value
Interval steps of TRPO iterations.	5000
Maximum number of iterations in the conjugate gradient method.	20
Damping factor used in the conjugate gradient method.	0.1
Discount factor $\in$ [0, 1].	0.995
Lambda-return factor $\in$ [0, 1].	0.97
Number of epochs for the value function.	5
Weight coefficient for entropy bonus $\in$ [0, inf).	0
Batch size of stochastic gradient descent method for value function.	64
Maximum number of backtracking.	10
Window size used to compute statistics of value predictions.	1000
Window size used to compute statistics of entropy of action distributions.	1000
Window size used to compute statistics of KL divergence between old and new policies.	100
Window size used to compute statistics of step sizes of policy updates.	100

. Figure 1 shows the flowchart of reinforcement learning with a Gaussian policy model optimized by TRPO algorithm. In this case, a reward was given that was inversely proportional to the distance between the manipulator tip and the target object. The episode ended if the tip touched the target or the episode duration expired.

2.3. Activation Functions

Different activation functions have been proposed in the field of machine learning [32,33]. The choice of the right activation function is dependent on the targeted use scenario of the learning algorithm. In this research, different activation functions were employed to identify the most appropriate function for use. The evaluated activation functions and corresponding computation expressions are shown in

Table 3. Evaluated activation functions.

Function Name	Expression
ReLU	$f(x)=max(0,x)=\left\{\begin{array}{c}{x}_i \left(x\ge 0\right)\\ 0 \left(x<0\right)\end{array}\right.$
Sigmoid	$f(x)=\frac{1}{1+e^{-x}}$
SoftMax	$f(x_i)=\frac{e^{x_i}}{{\sum }_{j=1}^n{e}^{x_j}}(\mathrm{i}=\mathrm{1,2},\cdots ,\mathrm{n})$
SoftPlus	$f(x)=\log \left(1+e^x\right)$
Radial Basis Function (RBF)	$f(x)=\frac{1}{\sqrt{2\pi {\sigma }^2}}\mathrm{e}\mathrm{x}\mathrm{p}(-\frac{{(x-\mu )}^2}{2{\sigma }^2})$

. The performance of the functions is as shown in Figure 2, Figure 3, Figure 4, Figure 5 and Figure 6. From the figures, ReLU, RBF, and Sigmoid reported the best training performance, as shown in the mean reward of Figure 2, Figure 3 and Figure 4, respectively.

The performance of tip position using RBF and ReLU is shown in Figure 7. From the figure, position in x-axis and z-axis are slightly different but position in y-axis is significantly different and does not improve over time. This implies that the tip did not arrive at the target position. In the figure, 1 point represents 50 episodes. This was plotted to improve visualization quality of the plotted data. During testing, the ReLU trained model was found to have the best performance and, hence, it is maintained as the activation function in the rest of the paper. The simplified activation function is shown in (8). In this case, the Adam optimizer is applied for stochastic gradient descent during training [34].

$$\mathrm{y}=\left\{\begin{array}{c}x \left(x>0\right)\\ 0 \left(x\le 0\right)\end{array}\right.$$

(8)

3. Simulation of the Flexible Manipulator Model

To analyze the performance of the learning algorithm, robot simulator software was used to ascertain the position of the end effector with reference to the target. For this purpose, Multi-Joint dynamics with Contact (MuJoCo) physics simulator was used [35]. MuJoCo, produced by Roboti LLC^® (Redmond, WA, USA), is free for students, and is a 3D physics engine developed for research and development in robotics targeting model-based optimization through contacts.

3.1. Actual Manipulator Model

To assess the RL outcomes for the flexible manipulator, a reference model was required. Figure 8a displays the physical implementation of the flexible manipulator used in the study, consisting of two links and three degrees of freedom for manipulation. The connection setup, including the boards and signaling components, is depicted in Figure 8b. For analog-to-digital conversion (ADC) and digital-to-analog conversion (DAC) processes, a dSPACE 1103 controller board was utilized. This board facilitated device driving and communication by converting analog signals to digital format and vice versa. The DC motor servo driver, connected to the dSPACE signal processing board, drove the motors responsible for manipulator movement. These motors were equipped with optical encoders, providing feedback on the angles of rotation with a pulse output of 1000 pulses per revolution. The angle information from the encoders was sent back to the servo drive amplifier and then transmitted to the dSPACE board. To reduce joint speed, harmonic drives were employed, effectively lowering the speed of the joints by a factor of 100. Specifically, Joint 1 exhibited twisting motion perpendicular to the ground, while Joints 2 and 3 allowed bending rotation. This configuration allowed the flexible manipulator to perform its intended movements and tasks.

3.2. Virtual Manipulator Model

Figure 9 shows the schematic diagram of the proposed flexible manipulator. The design is inspired by a physical robot shown in Figure 8. From the figure, there are three motors (motor 1, 2, and 3) as shown. Motor 1 has z-axis rotation, while motor 2 and 3 have x-axis rotation. A mass with weight, w is attached to the tip, and the side length is h. There are two cylindrical rods which formulate Link 1 and Link 2 from the bottom. The length of each link $L_i$($i=\mathrm{1,2}$), diameter $d_i$($i=\mathrm{1,2}$), mass $W_i$($i=\mathrm{1,2}$), and the longitudinal coefficient $E_i$($i=\mathrm{1,2}$) are listed in

Table 4. Parameters of the flexible manipulator.

Parameter	Value
${\mathit{L}}_1$	0.44 [m]
${\mathit{L}}_2$	0.44 [m]
${\mathit{W}}_1$	0.027 [kg]
${\mathit{W}}_2$	0.059 [kg]
${\mathit{E}}_1$	$19.3\times {10}^{10}$$[\mathrm{N}/{\mathrm{m}}^3$]
${\mathit{E}}_2$	$72.6\times {10}^9$$[\mathrm{N}/{\mathrm{m}}^3$]
${\mathit{d}}_1$	0.01 [m]
${\mathit{d}}_2$	0.008 [m]
w	0.1 [kg]
h	0.04 [m]

. Link 1 is assumed to be stainless steel, and Link 2 is assumed to be aluminum. Also, each link reproduces flexibility by connecting 11 cylinders with ball joints (Figure 10). Specific manipulator parameters are as shown in Table 4. Figure 11 shows the implemented design in MuJoCo, with the target object shown in red.

The strain $\mathsf{\epsilon }$ at the root of Link 1 is measured. A sensor that measures the force $\mathit{F}={\left[F_x ,F_y ,F_z\right]}^T$ is attached 0.02 m above the root of Link 1. The strain in the y-axis direction, which is the out-of-plane direction, is parameter of consideration in this research. The calculation method of strain $\mathsf{\epsilon }$ from the measured force is as shown in (9).

$$\mathsf{\epsilon }=\frac{M}{Z{E}_1}$$

(9)

where M is the moment, given by (10):

$$M=F_y\times 0.02$$

(10)

The section modulus, $Z$ can be written as shown in (11). In this case, d₁ is the diameter of Link 1.

$$Z=\frac{{d_1}^3\pi }{32}$$

(11)

3.3. Experiment Setup

The flexible manipulator in the virtual and actual model is used to verify the performance. In the RL training phase, the target position shown in red (Figure 11) moves randomly within the reachable space of the manipulator. The distance between the tip of the manipulator and the target is used as a reward, in that the RL agent is rewarded for action-set that decreases the distance.

After training, the model is evaluated in an actual manipulator. For that, a predefined X, Y, Z position is supplied to the model as the target. The model outputs a corresponding input signal that is fed to each of the joints to realize the targets. For verification, the study also conducted an experiment where each joint is moved 20 degrees on the actual machine without the controller. This is the same end-effector position as that of the RL controller model. In the no-controller experiment, the manipulator starts with a vertical posture at t = 0.

Next, a characteristic comparison of the designed model with a physical two-link flexible manipulator used in the laboratory was performed. In a physical robot, when each motor is moved 20 degrees, the resonance frequency is 3 Hz for about 10 s until the vibration dies out for forward reverse motion. The steady strain ${\mathsf{\epsilon }}_{st}$ is $-2.982\times {10}^{-6}$.

Figure 12 shows the modelled trajectory of the end effector of the designed model with no vibration control mechanism. Figure 13 is the Link 1 and Link 2 root strain response of the modeled and actual manipulators. The experimental and simulation results are in almost good agreement, and the modeling is considered valid. The performance is confirmed to have similar characteristics to the physical model reported in [36]. The model is designed to have a sampling rate of 0.002 s (1 step = 0.002 s).

4. Results and Discussion

The overall goal of the trained model is to control operations to bring the end effector closer to the target coordinates. If the target coordinate is at $\mathit{g}={\left[g_x ,g_y ,g_z\right]}^T$ and the end effector coordinate is at $\mathit{e}={\left[e_x ,e_y ,e_z\right]}^T$, then the reward function R should be the distance between two points as shown in (12). The reward R is highest when the end effector is closer to the target coordinate and vibration is suppressed.

$$R=\frac{1}{1+‖\mathit{e}-\mathit{g}‖}$$

(12)

4.1. Position Control Results

Figure 14 shows the trajectory of the end effector after the learning phase. In this case, one episode is 500 steps. Compared with Figure 12, it can be seen that vibration has been suppressed. However, although the end effector is moved to the target position, it is clear from Figure 15 that strain has not been suppressed.

4.2. Vibration/Strain Control Results

From Figure 14, the position of the tip of the flexible manipulator has been suppressed effectively with the exception of strain (Figure 15). To remedy for strain distortion, a new training model is performed where strain error is added to reward equations. The modified reward functions are shown in (13) and (14). The distortion amplification gain is set to $1\times {10}^4.$ The results of trained model data is shown in Figure 16 and Figure 17 for end effector position and strain, respectively.

$$R=\frac{1}{1+‖\mathit{e}-\mathit{g}‖}+\frac{1}{1+1\times {10}^4\times \left|{\mathsf{\epsilon }-\mathsf{\epsilon }}_{st}\right|}$$

(13)

$$R=\frac{1}{1+‖\mathit{e}-\mathit{g}‖}\times \frac{1}{1+1\times {10}^4\times \left|{\mathsf{\epsilon }-\mathsf{\epsilon }}_{st}\right|}$$

(14)

From the figures, both vibrations and strain were suppressed. As shown in Figure 17, the multiplicative model (Equation (14)) had better strain results compared to the additive model (Equation (13)). This is seen as a reduction in distortion (vibration) of the resultant strain. The magnitude of the target strain after reaching target coordinate was set in (12) to be $-5\times {10}^{-6}\le \mathsf{\epsilon }\le -0.5\times {10}^{-6}$. For (14), it was set at $-3.5\times {10}^{-6}\le \mathsf{\epsilon }\le -2\times {10}^{-6}$. Since (13) is an addition, it is argued that the overall reward increases as the distance from the target coordinate decreases. On the other hand, for (14), the distance from the target coordinate is short, and the total sum of rewards does not increase unless the distortion is suppressed. Equation (14) can be expressed as shown in (15).

$$R=\frac{1}{1+‖\mathit{e}-\mathit{g}‖}\times \frac{1}{1+\beta \times \left|{\mathsf{\epsilon }-\mathsf{\epsilon }}_{st}\right|}$$

(15)

Building from (15), the premise is that, as gain $\beta$ increases, the effect of the overall reward is propagated more, thereby reducing strain. To this end, $\beta$ was varied from $1\times {10}^4$, as shown in (14), to $3.5\times {10}^5$ to arrive at an optimal gain for the reward function.

Figure 18, Figure 19 and Figure 20 show the results of the trajectories and strains for various $\beta$ values. From the results, the end effector reached the target coordinates in all reward functions. Notably, for higher $\beta$ values, slightly more time was needed before the position settled. This is as seen in Figure 17 where position of end effector in z-axis settles at around 300^th episode for $\beta =3.5\times {10}^5$, compared with $\beta =5\times {10}^4$, which settles at approximately 150^th episode.

Regarding the strain, different values of $\beta$ resulted in different strain suppressions, as shown in Figure 18 and Figure 19. When $\beta$ was made larger than $25\times {10}^4,$ the strain distortion diverged, as shown in Figure 20. This is considered to be the effects of distortions caused on the overall reward being too large. From the figure, Equation (15) with $\beta$ set to $2.5\times {10}^5$ was considered the optimal reward function.

4.3. Experiment on the Actual Flexible Manipulator

Figure 21 shows the time response of the root strain voltage of Link 1, Link 2, and Link 3, respectively. From Figure 21, the joint angles did not converge to the target values. In addition, from Figure 22, the steady strain in the in-plane direction of Link 1 and Link 2 is the same, and vibration tends to be suppressed to some extent.

The power spectral densities of the strain at the base of each joint is shown in Figure 23. From the figure, we note that there are peaks around 3 Hz and 10 Hz, which are the resonance frequency components in the actual machine experiment similar to what is reported in [22,37]. By vibration mode analysis, among the resonance frequency components, the components near 3 Hz and 10 Hz are vibration modes in the driving direction of Joint 2 and Joint 3, and the components near 20 Hz are in the torsional direction, that is, in the torsional direction of Joint 1. It has been found that the vibration mode is in the rotational direction.

Analyzing the changes in the distortion of Link 1 after applying reinforcement learning, as depicted in Figure 22, we observed favorable results in the numerical simulation. However, when implementing these learned policies on the actual machine, there is a possibility of encountering issues related to the motor’s ability to follow the desired trajectories. Previous research has indicated that subtle differences between the numerical simulation environment and the real-world experimental environment can significantly impact the learning outcomes in a negative manner, what is referred to as sim–real challenges [18,38,39].

The sim–real problem, characterized by the disparities between the simulated and experimental environments, likely played a significant role in the notable discrepancy observed between the modeled reinforcement learning outcomes and the experimental data. Some of the key causes of sim–real problems include:

• Model inaccuracies: The simulation model may not accurately capture all aspects of the real-world system, such as dynamics, friction, or other physical properties. Assumptions and simplifications made during modeling may have led to discrepancies when compared to the actual system;
• Sensor and actuator variability: Sensors and actuators used in the real-world setup may exhibit variations and uncertainties that are not adequately represented in the simulation. Differences in sensor accuracy, noise levels, actuator response, and other characteristics can contribute to deviations between the simulation and reality, leading to sim–real problems;
• Unmodeled dynamics and nonlinearities: Complex interactions and nonlinear dynamics within the system may not be fully captured in the simulation model. Structural deformations, coupling between joints, dynamic effects, and other unmodeled phenomena can result in discrepancies between the simulated and real-world behavior, leading to sim–real problems.

This emphasizes the need for further investigation through implementation experiments to thoroughly understand and address these challenges. By conducting additional research and experimentation, we hope to can gain insights into the specific issues arising from the sim–real problem and improve the adaptability and performance of RL model when applied to real-world systems.

5. Conclusions

In this paper, we successfully applied reinforcement learning to control the vibration of a two-link flexible manipulator. Through simulations of a 3D model in MuJuCo, we validated the position and vibration control, closely resembling our physical manipulator used in the laboratory.

During training, our objective was to minimize the distance between the end effector and target coordinates, incorporating displacement and strain into the reward function. To achieve this, we incorporated displacement and strain into the reward function using a multiplicative model (Equation (15)) with β set to 2.5 × 10⁵. The results demonstrated the effective suppression of vibration at the tip of the flexible manipulator, as well as mitigated strain distortions.

However, when applying the trained model to the physical manipulator, we encountered a sim–real challenge, resulting in a discrepancy between the simulation and experimental environments. This discrepancy may be attributed to modeling assumptions, sensor and actuator variability, environmental factors, calibration and setup differences, and unaccounted dynamics and nonlinearities.

To address these challenges in future studies, we propose remedies such as improving simulation fidelity, utilizing transfer learning techniques, iterative refinement through physical testing, and incorporating hardware-in-the-loop setups for real-time validation. By addressing the sim–real challenge and bridging the gap between simulation and reality, future research can achieve better alignment and enhance the robustness and reliability of flexible manipulator control using reinforcement learning.