Enhancing Quadcopter Autonomy: Implementing Advanced Control Strategies and Intelligent Trajectory Planning

Abstract

In this work, an in-depth investigation into enhancing quadcopter autonomy and control capabilities is presented. The focus lies on the development and implementation of three conventional control strategies to regulate the behavior of quadcopter UAVs: a proportional–integral–derivative (PID) controller, a sliding mode controller, and a fractional-order PID (FOPID) controller. Utilizing careful adjustments and fine-tuning, each control strategy is customized to attain the desired dynamic response and stability during quadcopter flight. Additionally, an approach called Dyna-Q learning for obstacle avoidance is introduced and seamlessly integrated into the control system. Leveraging MATLAB as a powerful tool, the quadcopter is empowered to autonomously navigate complex environments, adeptly avoiding obstacles through real-time learning and decision-making processes. Extensive simulation experiments and evaluations, conducted in MATLAB 2018a, precisely compare the performance of the different control strategies, including the Dyna-Q learning-based obstacle avoidance technique. This comprehensive analysis allows us to understand the strengths and limitations of each approach, guiding the selection of the most effective control strategy for specific application scenarios. Overall, this research presents valuable insights and solutions for optimizing flight stability and enabling secure and efficient operations in diverse real-world scenarios.

1. Introduction

As technology continues to evolve, the domain of drones presents a fascinating landscape of innovation. Among the diverse types of drones available, quadcopters stand out as highly promising and versatile unmanned aerial vehicles (UAVs), known for their exceptional maneuverability and ability to navigate challenging terrain. They have become invaluable tools in diverse applications, such as surveillance and disaster response [1,2]. To fully harness their capabilities, it is essential to develop advanced control strategies and trajectory planning techniques that can effectively adapt to dynamic environments and avoid obstacles [3,4,5,6,7].

For implementation and simulation, various linear, nonlinear, and robust controllers have been studied [8,9,10]. The PID and FOPID controllers have been applied on height, x and y positions, and roll, pitch, and yaw angles. Li, J. et al. proposed a trajectory-tracking control of a quadrotor based on a fractional-order S-Plane model; their study focused on the low tracking accuracy and weak anti-interference ability of quadcopter drones in trajectory-tracking control [11]. Ademola, A. et al.’s study investigated the problem of the nonlinear quadcopter system’s mathematical modeling and control for stabilization and trajectory tracking using the feedback linearization (FBL) technique combined with the PD controller [12]. The implementation of a robust sliding mode controller (SMC) for trajectory tracking has been carried out by Yih, C.-C. et al. They demonstrate from the Lyapunov stability theorem that the proposed control scheme can guarantee the asymptotic stability of the tilt-rotor quadcopter in terms of position and attitude following control allocation [13]. The Dyna-Q learning-based obstacle avoidance technique was the subject of many research works and was applied to a wide range of quadcopter controllers [14,15,16]. Budiyanto, A. et al. proposed the application of the Deep Dyna-Q algorithm in formation control in both simulations and actual experiments [15].

The main objectives of this work revolve around enhancing quadcopter autonomy and control capabilities through comprehensive investigation. The first section of this work explores the principles of flight dynamics, encompassing the mechanics that govern quadcopter movement. The second section delves into the modeling aspect, presenting a detailed mathematical representation of quadcopter dynamics. This comprehensive quadcopter model covers both kinematics and dynamics, serving as a robust framework for controller design and stability analysis. Utilizing this model, we proceed to implement three distinct control approaches: the proportional–integral–derivative (PID) controller, the fractional-order PID controller, and the sliding mode controller (SMC).

The PID controller, widely known for its simplicity and widespread use, serves as the baseline for comparison among the control strategies. Additionally, the project explores the integration of fractional calculus into the PID controller to enhance control performance, particularly in managing system nonlinearities and uncertainties [6]. Furthermore, the sliding mode controller’s robustness against disturbances was evaluated.

In the final section of this work, the crucial aspect of obstacle avoidance is addressed through the development of an intelligent trajectory planning approach using Dyna-Q learning. This approach leverages real-time environmental data to generate safe and efficient paths for quadcopters, enabling them to autonomously navigate complex environments while intelligently avoiding obstacles. The proposed control strategies, together with trajectory planning using Dyna-Q learning, undergo extensive validation through simulations employing various performance metrics.

The anticipated outcomes of this research include significant advancements in quadcopter autonomy, enabling their deployment in increasingly challenging scenarios. By integrating advanced control strategies and intelligent trajectory planning, we aim to enhance the quadcopter’s adaptability and responsiveness, making them indispensable assets in applications such as search and rescue missions, precision agriculture, and infrastructure inspection. Through an in-depth understanding of the strengths and limitations of various control strategies and innovative approaches like Dyna-Q learning, we strive to unlock the full potential of quadcopters and pave the way for their widespread adoption in diverse applications.

2. Quadcopter State Space Model

The state space vector X describes the position of the quadcopter in space and its linear and angular velocities as follows [5,6,7,8,17,18,19,20]:

$$X={\left[\begin{array}{c}\hspace{0.25em}\hspace{0.25em}x\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\dot{x}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}y\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\dot{y}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}z\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\dot{z} \varphi \hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\dot{\varphi }\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\theta \hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\dot{\mathsf{\theta }}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\psi \hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\dot{\psi }\end{array}\right]}^T$$

(1)

The control input vector for a quadcopter state space model is represented as follows:

U = [U₁ U₂ U₃ U₄] T

(2)

where U₁ controls the quadcopter’s total thrust that is responsible for quadcopter altitude, while U₂, U₃, and U₄ control the moments that result in roll, pitch, and yaw angles, respectively, enabling precise control and stabilization of the quadcopter in response to external disturbances.

Figure 1 illustrates the movement of a quadcopter in space. The x, y, and z variables represent the displacements of the quadcopter’s center of mass from an Earth-fixed inertial frame along the respective x, y, and z axes. The quadcopter’s orientation is represented by the three Euler angles: φ represents the roll angle around the x-axis, θ represents the pitch angle around the y-axis, and ψ represents the yaw angle around the z-axis [5,6,7].

In order to obtain the state space representation of the quadcopter, the following equations that describe the translational and rotational motion of the quadcopter are used [17,18,19,20]:

$$\left\{\begin{array}{c}\ddot{x}=\frac{U_1}{m_t}(\sin \varphi \sin \psi +\cos \varphi \mathrm{sin\theta }cos\psi )-\frac{A_x\dot{x}}{m_t}\\ \ddot{y}=\frac{U_1}{m_t}\left(\cos \varphi \mathrm{s}\mathrm{i}\mathrm{n}\mathsf{\theta }\mathrm{s}\mathrm{i}\mathrm{n}\psi -\mathrm{s}\mathrm{i}\mathrm{n}\varphi \mathrm{c}\mathrm{o}\mathrm{s}\psi \right)-\frac{A_y\dot{y}}{m_t}\\ \ddot{z}=\frac{U_1}{m_t}\left(\cos \varphi cos\mathsf{\theta }\right)-g-\frac{A_z\dot{z}}{m_t}\end{array}\right.$$

(3)

The translation equation of motion in terms of control input is shown above;

$$\left\{\begin{array}{c}\ddot{\mathsf{\varphi }}=\dot{\mathsf{\theta }}\dot{\varphi }{a}_1\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}+b_1{lU}_2-\dot{\mathsf{\theta }}{a}_2{\omega }_{rz}\hspace{0.25em}\hspace{0.25em}-K_{ax}{b}_1\dot{\varphi }\\ \ddot{\mathsf{\theta }}=\dot{\varphi }\dot{\mathsf{\psi }}{a}_3+{b_{2}lU}_3-\dot{\varphi }{a}_4{\omega }_{rz}\hspace{0.25em}\hspace{0.25em}-K_{ay}{b}_2\dot{\mathsf{\theta }}\\ \ddot{\mathsf{\psi }}={b_{3}U}_4-K_{az}{b}_3\dot{\mathsf{\psi }}\end{array}\right.$$

(4)

And the rotational equation of motion in terms of control input is shown above,

where:

• m_t is the total mass of the quadcopter;
• $K_{ax}$ and $K_{ay}{,K}_{az}\mathrm{a}\mathrm{r}\mathrm{e}$ the aerodynamics rotation coefficients matrix;
• ${\omega }_{rz}$ is the rotor’s angular velocities about the axis where the rotation occurs (z-axis).

And

$$a_1=\frac{I_{yy}-I_{zz}}{I_{xx}}\hspace{0.25em}\hspace{0.25em},a_2=\frac{I_r}{I_{xx}}\hspace{0.25em},a_3=\frac{I_{zz}-I_{xx}}{I_{yy}},a_4=\frac{I_r}{I_{yy}}\hspace{0.25em},b_1=\frac{1}{I_{xx}}\hspace{0.25em},b_2=\frac{1}{I_{yy}}\hspace{0.25em},b_3=\frac{1}{I_{zz}}\hspace{0.25em}$$

$I_{xx}$, $I_{yy}$, and $I_{zz}$ are the mass moments of inertia about the three principal axes in the body frame.

The nonlinear state differential equations are expressed according to the following:

$$\left\{\begin{array}{c}\dot{x_1}=x_2\\ \dot{x_2}=\frac{U_1}{m_t}(\sin x_7\sin x_{11}+\cos x_7\sin x_{9}cos{x}_{11})-\frac{A_x{x}_2}{m_t}\\ \dot{x_3}=x_4\\ \dot{x_4}=\frac{U_1}{m_t}\left(\cos x_7\mathrm{s}\mathrm{i}\mathrm{n}{x}_9\mathrm{s}\mathrm{i}\mathrm{n}{x}_{11}-\mathrm{s}\mathrm{i}\mathrm{n}{x}_7\mathrm{c}\mathrm{o}\mathrm{s}{x}_{11}\right)-\frac{A_y{x}_4}{m_t}\\ \dot{x_5}=x_6\\ \dot{x_6}=\frac{U_1}{m_t}\left(\cos x_{7}cos{x}_9\right)-g-\frac{A_z{x}_6}{m_t}\\ \dot{x_7}=x_8\\ \dot{x_8}=x_{10}{x}_8{a}_1+b_1{lU}_2-x_{10}{a}_2{\omega }_{rz}-K_{ax}{b}_1{x}_8\\ \dot{x_9}=x_{10}\\ \dot{x_{10}}=x_8{x}_{12}{a}_3+{b_{2}lU}_3-x_8{a}_4{\omega }_{rz}-K_{ay}{b}_2{x}_{10}\\ \dot{x_{11}}=x_{12}\\ \dot{x_{12}}={b_{3}U}_4-K_{az}{b}_3{x}_{12}\end{array}\right.$$

(5)

3. Quadcopter Control Methods

3.1. PID Controller

PID controllers, known as proportional–integral–derivative controllers, are widely employed due to their simplicity and high efficiency in numerous industrial applications [19,20,21,22]. To design the PID controller for the nonlinear system, an initial step involves designing and tuning a controller for the linearized model. Subsequently, the designed controller is then implemented on the nonlinear quadcopter system. This approach simplifies the control design process by utilizing classical control techniques suitable for linear systems, allowing for an effective starting point in achieving stabilization. However, it is crucial to acknowledge that the linearized model is an approximation and may not fully capture all the complexities of the quadcopter’s nonlinear dynamics. Therefore, further adjustments and fine-tuning of the actual nonlinear system may be required to optimize stability and overall performance in varying operating conditions.

Altitude Controller: A PID controller is designed to control the quadcopter’s altitude, generating the control input U₁ that governs the quadcopter’s altitude as shown below:

$$U_1=\left(K_P{e}_z+K_i\int e_z+K_D\frac{d{e}_z}{dt}+g\right)m_t$$

(6)

Roll Controller: A PID controller is designed to generate the control input U₂ that governs the quadcopter’s roll motion. The control law is presented by the following:

$$U_2=K_P{e}_{\varphi }+K_i\int e_{\varphi }+K_D\frac{d{e}_{\varphi }}{dt}$$

(7)

Pitch Controller: a PID controller is designed to generate the control input U₃ that governs the quadcopter’s pitch motion. The control law is presented by the following:

$$U_3=K_P{e}_{\mathsf{\theta }}+K_i\int e_{\mathsf{\theta }}+K_D\frac{d{e}_{\mathsf{\theta }}}{dt}$$

(8)

Heading Control: A PID controller is designed to generate the control input U₄ that governs the quadcopter’s yaw motion. The control law is presented by the following:

$$U_4=K_P{e}_{\mathsf{\psi }}+K_i\int e_{\mathsf{\psi }}+K_D\frac{d{e}_{\mathsf{\psi }}}{dt}$$

(9)

Position Control: To control the position, two PID controllers are designed to generate the control signals, U_x and U_y, which represent the accelerations, $\ddot{x}$ and $\ddot{y}$, respectively. The formulated control laws are presented by the following:

$$\left\{\begin{array}{c}{U}_x=\ddot{x}=K_P{e}_{\mathrm{x}}+K_i\int e_{\mathrm{x}}+K_D\frac{d{e}_{\mathrm{x}}}{dt}\\ U_y=\ddot{y}=K_P{e}_{\mathrm{y}}+K_i\int e_{\mathrm{y}}+K_D\frac{d{e}_{\mathrm{y}}}{dt}\end{array}\right.$$

(10)

3.2. Fractional-Order Controller

The nonlinear fractional controller, known as the ${PI}^{\lambda }{D}^{\mu }$ controller, represents a broader form of the classical PID controller. It expands the conventional integral and differential orders, λ and μ, beyond integer values into the real and complex domains [11,23,24].

Table 1. Extended types of classical PID controllers.

Type of Controller	λ	μ
${\mathbf{P}\mathbf{I}}^{\mathsf{\lambda }}$	$0<\mathsf{\lambda }<$1	0
${\mathbf{P}\mathbf{I}}^{\mathsf{\lambda }}\mathbf{D}$	$0<\mathsf{\lambda }<$1	1
${\mathbf{P}\mathbf{D}}^{\mathsf{\mu }}$	0	$0<\mathsf{\mu }<$1
${\mathbf{P}\mathbf{I}\mathbf{D}}^{\mathsf{\mu }}$	1	$0<\mathsf{\mu }<$1
${\mathbf{P}\mathbf{I}}^{\mathsf{\lambda }}{\mathbf{D}}^{\mathsf{\mu }}$	$0<\mathsf{\lambda }<$1	$0<\mathsf{\mu }<$1

resumes all possible configurations for the types of classical PID controllers, which are determined by the values of μ and λ:

The transfer function of a fractional-order controller has the following form:

$$U\left(s\right)=E\left(s\right)\left[K_P+K_I{s}^{-\lambda }+K_D{s}^{\mu }\right]$$

(11)

The equation for the ${PI}^{\lambda }{D}^{\mu }$ controller output in the time domain is as follows:

$${u\left(t\right)=K}_{P}e\left(t\right)+K_I{D}^{-\mathsf{\lambda }}e\left(t\right)+K_D{D}^{\mu }e\left(t\right)$$

(12)

$K_P,K_I,\mathrm{a}\mathrm{n}\mathrm{d}{K}_D$ represent the proportional, integral, and derivative gains, respectively, and $e\left(t\right)$ denotes the error between the desired and the obtained results. In addition to these parameters, the FOPID controller introduces two additional degrees of freedom, μ and λ, which play a crucial role in enhancing the controller’s performance and providing greater flexibility in its design. These extra degrees of freedom allow for superior control capabilities and improved adaptability to various control tasks. After obtaining the appropriate PID gains for the nonlinear system, we proceeded to implement these gains into the fractional-order PID (FOPID) controller. The FOPID controller is designed using the FOMCON toolbox in MATLAB 2018a, which allows for tuning the parameters μ and λ for each controller. Through a systematic process, we fine-tuned these parameters and made necessary adjustments to the gains to account for any observed system oscillations. This iterative tuning procedure aimed to achieve improved control performance and stability for the quadcopter system under consideration.

3.3. Sliding Mode Controller

The sliding mode controller (SMC) technique is a nonlinear control approach that alters the dynamics of a nonlinear system using a discontinuous control signal. This signal compels the system to slide along a predefined cross-section of its normal behavior, showcasing remarkable accuracy and robustness [12,25,26]. The SMC technique comprises two essential components. Firstly, a discontinuous control law is employed to drive the error vector towards a specific decision rule, referred to as the sliding surface. Once the error vector reaches this surface, a continuous component of the controller takes over to follow the system dynamics defined by the equations characterizing the sliding surface. The selection of the sliding surface or decision rule is a critical aspect and is based on performance criteria since it determines the system’s dynamics. Therefore, it can be expressed as follows [12]:

$$s=\dot{e}+\lambda e$$

(13)

where S is the sliding surface, λ is a positive tuning parameter, and e is the tracking error;

$$\dot{s}=-K1sign\left(s\right)-K2s$$

(14)

and where K1, K2 are positive tuning parameters.

The sliding surface, its derivative, develops controllers for attitude, altitude, position, and heading.

Altitude sliding mode controller:

Considering the error between the desired altitude and the actual one provides the following equation:

$$e_z=z_d-z$$

(15)

The error in Equation (13) is replaced in Equations (14) and (15), and thus the following equations are obtained:

$$\left\{\begin{array}{c}{s}_z=(\dot{z_d}-\dot{z})+\lambda (z_d-z)\\ \dot{s_z}=\ddot{z_d}-\ddot{z}+\lambda (\dot{z_d}-\dot{z})\\ \dot{s_z}=-k1sign\left(s_z\right)-k2\left(s_z\right)\end{array}\right.$$

(16)

Equation (16) can be written as follows:

$$\ddot{z_d}-\ddot{z}+\mathsf{\lambda }(\dot{z_d}-\dot{z})=-k1sign\left(s_z\right)-k2\left(s_z\right)$$

(17)

The control input U1 law is obtained and represented by the following equation:

$$U1=\frac{m\ddot{(z_d}+g+\frac{A_z\dot{z}}{m}+\mathsf{\lambda }\left(\dot{z_d}-\dot{z}\right)+k1sign\left(s_z\right)+k2\left(s_z\right))}{\cos \varphi cos\mathsf{\theta }}$$

(18)

The attitude sliding mode controllers are presented below:

Roll controller:

$$\ddot{{\mathsf{\varphi }}_d}-\ddot{\mathsf{\varphi }}+\mathsf{\lambda }(\dot{{\mathsf{\varphi }}_d}-\dot{\mathsf{\varphi }})=-k1sign\left(s_{\mathsf{\varphi }}\right)-k2\left(s_{\mathsf{\varphi }}\right)$$

(19)

The control input U2 law is obtained and represented by the following equation:

$$U_2=\frac{\ddot{{\mathsf{\varphi }}_d}-\dot{\mathsf{\theta }}\dot{\varphi }{a}_1\hspace{0.25em}\hspace{0.25em}+\dot{\mathsf{\theta }}{a}_2{\omega }_{rz}\hspace{0.25em}\hspace{0.25em}+K_{ax}{b}_1\dot{\varphi }+\mathsf{\lambda }\left(\dot{{\mathsf{\varphi }}_d}-\dot{\mathsf{\varphi }}\right)+k1sign\left(s_{\mathsf{\varphi }}\right)+k2\left(s_{\mathsf{\varphi }}\right)}{b_{1}l}$$

(20)

Pitch controller:

$$\ddot{{\mathsf{\theta }}_d}-\ddot{\mathsf{\theta }}+\mathsf{\lambda }(\dot{{\mathsf{\theta }}_d}-\dot{\mathsf{\theta }})=-k1sign\left(s_{\mathsf{\theta }}\right)-k2\left(s_{\mathsf{\theta }}\right)$$

(21)

The control input U3 law is obtained and represented by the following equation:

$$U_3=\frac{\ddot{{\mathsf{\theta }}_d}-\dot{\varphi }\dot{\mathsf{\psi }}{a}_3+\dot{\varphi }{a}_4{\omega }_{rz}\hspace{0.25em}\hspace{0.25em}+K_{ay}{b}_2\dot{\mathsf{\theta }}+\mathsf{\lambda }\left(\dot{{\mathsf{\theta }}_d}-\dot{\mathsf{\theta }}\right)+k1sign\left(s_{\mathsf{\theta }}\right)+k2\left(s_{\mathsf{\theta }}\right)}{b_{2}l}$$

(22)

Heading sliding mode controller:

$$\ddot{{\psi }_d}-\ddot{\psi }+\mathsf{\lambda }(\dot{{\psi }_d}-\dot{\psi })=-k1sign\left(s_{\psi }\right)-k2\left(s_{\psi }\right)$$

(23)

The control input law is obtained and represented as follows:

$$U_4=\frac{\ddot{{\psi }_d}+K_{az}{b}_3\dot{\mathsf{\psi }}+\mathsf{\lambda }\left(\dot{{\psi }_d}-\dot{\psi }\right)+k1sign\left(s_{\psi }\right)+k2\left(s_{\psi }\right)}{b_3}$$

(24)

Position sliding mode controller:

$$\left\{\begin{array}{c}\ddot{x}=\ddot{x_d}+\lambda \left(\dot{x_d}-\dot{x}\right)+k1sign\left(s_x\right)+k2\left(s_x\right)\\ \ddot{y}=\ddot{y_d}+\lambda \left(\dot{y_d}-\dot{y}\right)+k1sign\left(s_y\right)+k2\left(s_y\right)\end{array}\right.$$

(25)

In this part, the trajectory of the quadcopter was refined using MATLAB and explored three distinct controllers: PID, fractional-order PID, and sliding mode. By simulating the quadcopter’s dynamics and subjecting these controllers to rigorous testing, our objective was to significantly enhance its path-tracking performance. Meticulous adjustments were systematically implemented to improve the quadcopter’s precision in trajectory tracking. This part underscores our practical approach to trajectory enhancement, seamlessly integrating simulated dynamics with real-world control experimentation.

3.4. Results

The presented results encompass the performance of the PID, fractional-order PID, and sliding mode controllers. The simulation was conducted under various scenarios to assess their effectiveness in trajectory regulation and error minimization. The outcomes underline the adaptability of these control methods to the nonlinear model, as well as their robustness in handling disturbances. Before delving into the results of controllers both with and without disturbances, it is important to discuss the reason behind introducing disturbances into the simulation. The inclusion of disturbances in our study serves two significant purposes: firstly, to emulate real-world scenarios where systems often encounter unpredictable external influences, and secondly, to evaluate the controllers’ ability to manage such disturbances and maintain stable performance. By subjecting the controllers to varying disturbance levels, we gain valuable insights into their robustness and adaptability, allowing us to make well-informed assessments of their practical efficacy.

3.4.1. PID Controller Results

• Linear model control

The fully tuned PID controller for the linear model displayed accurate trajectory regulation and effective control over the system’s behavior. The obtained trajectory closely approximates the desired trajectory, exhibiting a small deviation in its initial phase. Therefore, the tuning process is validated. Figure 2 shows the obtained and desired trajectories.

Figure 3 illustrates the control inputs.

Figure 4 shows how the errors of positions x, y, and z and orientations roll, pitch, and yaw change with respect to time.

Table 2. PID controller gains, settling time, and overshoot for the linear model.

Controller	P	I	D	Settling Time	Overshoot
x	0.18858	0.0025421	3.1082	1.28 s	0.737%
y	0.38107	0.0053418	3.2245	4.95 s	4.42%
z	11	0.034082	15	0.863 s	2.61%
Roll	0.2564	0.025926	0.5634	5.15 s	5.58%
Pitch	0.9788	0.19295	1.1033	0.001 s	4.25%
Yaw	3.2682	5.5053	0.2235	0.909	8.07%

showcases the PID gains, settling time, and percentage overshoot for positions x, y, and z and orientations roll, pitch, and yaw. These parameters provide crucial insights into the PID controller’s performance, stability, and accuracy in regulating the system.

• Nonlinear model control

The PID gains obtained from the linear model are used as a starting point for the PID gains of the nonlinear model.

Table 3. PID controller gains for the nonlinear model.

PID	P	I	D
x	0.25	0.003	3.5
y	0.92	0.01	2
z	150	50	30
Roll	9	0.05	1.2
Pitch	7	0.2	1
Yaw	3.26	5.5	0.22

displays the PID gains for positions x, y, and z and orientations roll, pitch, and yaw in the nonlinear model.

Using the PID gains derived from the linear model as a starting point, we fine-tuned the PID controller for the nonlinear model and obtained the following results shown in Figure 5:

Figure 6 illustrates the control inputs from the PID controller without disturbance.

Figure 7 shows how the errors of positions x, y, and z and orientations roll, pitch, and yaw change with respect to time.

Disturbance was introduced to the closed loop system in the interval t = {10 s–13 s}. The results are shown in Figure 8.

Figure 9 illustrates the control inputs from the PID controller with a disturbance.

Figure 10 shows how the errors of positions x, y, and z and orientations roll, pitch, and yaw change with respect to time:

3.4.2. Fractional-Order PID Controller Results

The initial gains were obtained from the PID controller for the nonlinear model. Subsequently, these gains were further adjusted to optimize the performance of the fractional-order PID controller. The evaluation of the fractional-order PID controller was carried out under both disturbance and non-disturbance conditions to highlight its adaptability and effectiveness in handling external influences. The following section dives into the results of these simulations, providing insights into the performance and robustness of the fractional-order PID controller in real-world scenarios.

Table 4. FOPID controller gains for the nonlinear model.

FOPID	KP	KI	KD	λ	μ
x	1	0.01	15	0.8	0.785
y	1	0.01	4	0.8	0.64
z	140	50	30	1	1
Roll	1	0.03	1.9	0.6	0.8
Pitch	1	0.2	1	0.5	0.8
Yaw	3.26	5.5	1	0.8	0.9

displays the fractional-order PID controller gains for positions x, y, and z and orientations roll, pitch, and yaw, as well as the parameters µ and λ. These optimized gains are essential in ensuring accurate and stable control of the system, allowing the fractional-order PID controller to regulate the trajectory effectively under the influence of nonlinear dynamics.

Figure 11 shows some modifications of the gains obtained from the previous PID controller and by tuning the parameters of the fractional-order PID controller.

Figure 12 depicts the control inputs from the fractional-order PID controller without disturbance.

Figure 13 shows how the errors of positions x, y, and z and orientations roll, pitch, and yaw change with respect to time.

After introducing disturbance to the closed loop system in the interval t={10s-13s}, the following results shown in Figure 14 were obtained:

Figure 15 depicts the control inputs from the FOPID controller with disturbance.

Figure 16 shows how the errors of positions x, y, and z and orientations roll, pitch, and yaw change with respect to time.

3.4.3. Sliding Mode Controller Results

The simulation of the sliding mode controller starts with initial gains set to “ones”, which is then fine-tuned to achieve optimal values, ensuring the controller’s robustness and adaptability. Through a comprehensive exploration of various scenarios, with and without disturbances, we examine the controller’s adeptness in regulating trajectory outputs and ensuring system stability. This section presents a detailed analysis of the simulation results, providing valuable insights into the sliding mode controller’s efficiency in handling intricate dynamics and external perturbations.

Table 5. The SMC gains for the nonlinear model.

Controller	λ	K1	K2
x	10	0.5	0.9
y	14.5	0.1	1.5
z	5	0.1	10
$\mathsf{\varphi }$	10.2	0.1	7.5
$\mathsf{\theta }$	5	0.01	110
$\psi$	30	0.1	30

displays the sliding mode gains for positions x, y, and z, and orientations roll, pitch, and yaw for the nonlinear model.

These optimized gains are essential in ensuring accurate and stable control of the system, allowing the sliding mode controller to regulate the trajectory effectively under the influence of nonlinear dynamics.

Through careful adjustment of the gains, the results presented in Figure 17 demonstrate effective and stable control in the absence of disturbances. These outcomes serve as proof of the sliding mode controller’s adaptability in successfully handling nonlinear dynamics.

Figure 18 depicts the control inputs from a sliding mode controller without disturbance.

Figure 19 shows how the errors of positions x, y, and z and orientations roll, pitch, and yaw change with respect to time.

Despite the presence of disturbances, the sliding mode controller in Figure 20 demonstrates exceptional adaptability by effectively handling nonlinear dynamics, maintaining stability, and achieving precise trajectory tracking.

Figure 21 depicts the control inputs from the SMC with the presence of disturbance.

Figure 22 displays how the errors of positions x, y, and z and orientations roll, pitch, and yaw change with respect to time.

3.5. Discussion

Each controller has its advantages and suitability for specific applications. The PID controller is reliable and widely used, while the fractional-order PID controller provides enhanced flexibility. Meanwhile, the sliding mode controller excels in handling disturbances and uncertainties. The choice of the controller depends on the specific requirements and characteristics of the control system. Further research and experimentation are encouraged to explore these controllers’ performance in different applications and real-world scenarios.

The PID controller, originally designed for the linear model, initially exhibited a slight perturbation in trajectory due to a disparity between the desired y trajectory’s starting point (set at 5) and the quadcopter’s actual initial position (which is 0). Additionally, a minor delay of approximately 1 s was observed in trajectory tracking, which was attributed to the system’s inherent oscillatory behavior. When applied to the nonlinear model, the PID controller displayed an initial time lag aligning with the trajectory. It manifested noticeable perturbations at the beginning of the trajectory: around 14 s for the x component and 5 s for the y component. However, beyond this initial phase, the controller gradually stabilized, closely approaching the desired path, while exhibiting a slight delay and a minor error. The PID controller demonstrated an acceptable performance in handling disturbances. It effectively managed the impact of disturbances on the y component. However, it exhibited a delayed reaction in addressing the deviations in the x component. It is worth noting the consistent presence of a delay in trajectory tracking, as well as a minor error that persisted along the trajectory. Regarding the altitude and heading components, the PID controller’s performance was notably satisfactory both in the presence and absence of disturbances. Despite these observations, the PID controller’s handling of disturbances remained acceptable overall.

3.6. Comparison

After evaluating the PID, fractional-order PID (FOPID), and sliding mode controllers for trajectory tracking, distinct performance characteristics came to light: The PID controller displayed effective disturbance handling even though there were considerable delays and overshoots. The fractional-order PID controller exhibited remarkable performance in rejecting disturbances; however, there is still some presence of error.

On the contrary, the sliding mode controller excelled comprehensively. The sliding mode controller not only achieved accurate trajectory tracking with minimal delays but also robustly handled disturbances, maintained minimal errors, achieved quick settling times, and demonstrated negligible overshoot. Taking these attributes into account, the sliding mode controller emerges as the most efficient choice, showcasing exceptional performance and suitability for achieving precise and effective trajectory tracking across many scenarios.

This section has provided a comprehensive evaluation of three controllers: PID, fractional-order PID, and sliding mode.

The PID controller demonstrated reliability and adaptability, yet with delays and overshoots. The fractional-order PID exhibited potential and robustness, but there were still slight errors present, along with initial instability. In contrast, the sliding mode controller stood out with precise trajectory tracking, robust disturbance handling, rapid settling times, and minimal overshoot.

This exceptional performance positions the sliding mode controller as the preferred choice for real-world applications requiring accurate trajectory tracking and offering valuable insights for practical implementation.

When controlling a quadcopter, the PID, FOPID, and sliding mode controllers (SMCs) offer different advantages and challenges. The PID controller is simple and widely used but struggles with nonlinearities and disturbances. The FOPID controller, with fractional calculus, provides better flexibility and robustness but is complex to design and computationally intensive. The SMC excels in robustness and handling nonlinearities, making it ideal for quadcopters, but it suffers from implementation complexity and chattering issues. The choice of control method hinges on balancing performance requirements, system complexity, and implementation feasibility.

4. Enhancing Quadcopter Trajectory Tracking through Dyna-Q Learning

Trajectory planning and obstacle avoidance are important for quadcopter navigation, ensuring safe and efficient paths in complex environments. By employing reinforcement learning, particularly the Dyna-Q approach, quadcopters can enhance their decision-making and adapt their flight trajectories. This combination of strategic path planning and adaptive obstacle avoidance, aided by advanced machine learning, allows quadcopters to optimize their operations, prevent collisions, and maintain stability while dynamically adjusting to their surroundings and achieving mission objectives [27,28,29,30,31].

4.1. Reinforcement Learning Approaches

Reinforcement learning is a type of machine learning approach where an agent learns to make decisions by interacting with an environment [12,32]. Figure 23 shows the reinforcement learning block diagram.

The agent learns and makes decisions and takes actions to maximize cumulative rewards over time, adjusting its behavior based on the reward received from the environment that indicates how good or bad the actions were. This feedback loop helps the agent learn optimal strategies for achieving specific goals, making it well suited for tasks that involve sequential decision-making in dynamic environments [12,32].

Reinforcement learning approaches consist of model-free and model-based approaches. The model-based approach includes learning the model or being provided with the model, while the model-free approach involves policy optimization and Q-learning techniques. Dyna-Q learning combines both learning the model and Q-learning to optimize the learning process effectively. In reinforcement learning, the Markov Decision Process (MDP) is used to model the interactions between an agent and the environment, helping the agent maximize cumulative rewards in uncertain environments [12,27,29,32,33]. MDPs aim to determine policies that guide the agent’s actions:

-

The deterministic policy specifies a single action for each state; for every state (s) there is a clear action choice π ∶ S → A that the agent follows.

-

The stochastic policy assigns a distribution over actions to each state following the policy π such that π: S → proba(A), where the agent decides actions based on probabilities for each state (s). This way, the agent can choose different actions in a state, with each option having its own chance of being selected.

Environments can also be deterministic (outcomes of actions are predictable) and stochastic (outcomes of actions are probabilistic and uncertain). Dyna-Q learning effectively navigates both deterministic and stochastic environments by optimizing the agent’s decision.

4.2. Q-Learning Algorithms

Q-learning is a useful technique for improving how quadcopters are controlled. Scientists have tried different versions of Q-learning to solve issues in quadcopter control, such as not often receiving rewards or dealing with complicated situations. By using Q-learning, quadcopters can quickly change how they act, deal with different situations, and perform tasks on their own. This has the potential to enable these aerial robots to perform more advanced tasks in the future. The Q-learning technique leads the quadcopter to the development of a value function that helps make smart decisions based on the rewards they expect to receive [28,30,33,34,35].

The Q-value Qπ(s, a) estimates the expected reward starting from state (s), taking action (a), and following policy π, while the V-function Vπ(s) estimates the cumulative reward starting from state (s) under policy π. The optimal policy π∗ maximizes the expected cumulative rewards. The agent faces the exploration–exploitation dilemma, where it must balance trying new actions (exploration) to learn about the environment and selecting known actions (exploitation) to maximize rewards. The epsilon-greedy strategy manages this balance by occasionally choosing random actions to explore while mostly selecting the best-known actions. This process continuously updates the Q-values, guiding the agent towards optimal decisions.

In the domain of reinforcement learning, the temporal difference (TD) error plays a crucial role in updating the expected return of an agent’s actions as it transitions from one state to another. The TD error captures the difference between the current estimate of the Q-value of a state–action pair and the updated estimate based on observed outcomes. Mathematically, the TD error is defined as follows [12,28]:

$$TDerror=R(s,a)+\gamma {max}_{a’}Q(s’,a’)–Q(s,a)$$

(26)

The temporal difference (TD) update equation for Q-learning, which is a model-free reinforcement learning algorithm, is used to update the Q-values based on the observed rewards and transitions. The equations are as follows [15,34]:

$$Q(s,a) ⃪Q(s,a)+\alpha TD$$

(27)

$$Q\left(s,a\right) ⃪Q(s,a)+\alpha (R\left(s,a\right)+\gamma {max}_{a’}Q\left(s’,a’\right)-Q(s,a\left)\right)$$

(28)

where α is the learning rate:

• High Learning Rate (α near 1): The agent will be highly responsive to the most recent experiences.
• Low Learning Rate (α near 0): The agent will be less responsive to new experiences and will rely more on existing knowledge.

Figure 24 shows the Q-learning algorithm block diagram.

4.3. Implementation of Dyna-Q Learning for Trajectory Planning

Reinforcement learning begins with an agent’s Q-table containing initial values. The agent explores and refines Q-values through interactions, guiding actions for higher rewards. Dyna-Q learning combines real experiences with simulations, helping the agent navigate complex environments efficiently by learning to avoid obstacles and achieve goals [10,12]. It maintains a Q-table and uses an environment model to accelerate learning. This approach strikes a balance between real and simulated experiences [15,16,34,35,36].

In reinforcement learning for optimal policy derivation, the Bellman equation for the state–value function V(s) is defined as follows:

V(s) = r(s) + γ  max ∑^H T(s, a, s′) V(s′)

(29)

This leads to the Bellman equation for the action–value function:

Q(s, a) = r(s, a) + γ ∑^H T(s, a, s′) [max_a Q(s′, a′)]

(30)

To create a transition matrix: T[s, a, s′] = P(s′ | s, a)

(31)

The agent’s possible states and actions are enumerated, and transition probabilities are assigned; the agent randomly selects an action while it is in a random state, then transitions to the next state according to the updated transition matrix, measures the reward according to the reward function, and updates the Q-values. This process is repeated in the m planning steps.

Figure 25 illustrates the Dyna-Q learning algorithm block diagram.

The TD (temporal difference) error for a state–action pair (s, a) in Dyna-Q is expressed as follows:

$$TDerror(s,a)=R(s,a)+\gamma \left(\sum s^{\prime }T\left(s,a,s^{\prime }\right){max}_{a^{\prime }}Q(s\prime ,a^{\prime })\right)-Q(s,a)$$

(32)

The temporal difference (TD) update equation for Q-learning is used to update the Q-values based on the observed rewards and transitions.

The equation becomes the following:

$$Q(s,a)\leftarrow Q(s,a)+\alpha \left[R\right(s,a)+\gamma (\Sigma s^{\prime }T\left(s,a,s^{\prime }\right){max}_{a^{\prime }}Q(s^{\prime },a^{\prime }))-Q(s,a\left)\right]$$

(33)

For obstacle avoidance:

-

A lower learning rate $\alpha$ is better for obstacle avoidance in uncertain environments. It allows the agent to be cautious in updating its Q-values based on new experiences.

-

A higher discount factor $\gamma$ is better for obstacle avoidance. It encourages the agent to consider long-term consequences and plan for the future, which is important when navigating around obstacles and finding safe paths.

-

A lower exploration rate ε is better for obstacle avoidance during the initial stages of learning.

4.4. Results

Table 6. The values of the parameters used in the improved Dyna-Q learning algorithm.

Number of Episodes	Learning Rate	Discount Factor	Exploration Rate	Horizon	Obstacle Hit Reward	Reach Goal Reward	Regular Step
3000	0.1	0.8	0.2	200	−100	100	−2

summarizes the chosen values of the parameters used in the improved Dyna-Q learning algorithm.

4.4.1. Deterministic and Stochastic Environments Results

Real-world environments often have some level of uncertainty, and incorporating stochastic elements in the learning process can be beneficial to train agents that perform well in uncertain and dynamic scenarios. Ultimately, the choice between deterministic and stochastic environments depends on the application and the learning goals.

The total reward and steps in the deterministic environment are shown in Figure 26.

The trajectory path planning is represented in Figure 27 in 2D and 3D spaces.

The total reward and steps in the stochastic environment are shown in Figure 28.

The trajectory path planning is represented in 2D and 3D spaces in Figure 29.

The stochastic environment seems to yield a more direct path from the starting point to the goal while avoiding obstacles effectively. This could be because the stochastic environment encourages the agent to explore and find efficient paths rather than taking a roundabout route. If this behavior aligns with your goals and the real-world scenario you are simulating, the stochastic environment might be more appropriate.

4.4.2. Dyna-Q Learning with Sliding Mode Controller

After using the path determined by Dyna-Q learning as the planned path and employing a sliding mode controller to manage the quadcopter’s movements, the resulting flight path can be observed in both 3D and 2D spaces in Figure 30.

Table 7. SM gains for the resulting path from Dyna-Q learning.

Controller	λ	K1	K2
x	1.5	0.08	0.4
y	1.6	0.07	0.6
z	0.5	0.08	60
$\mathsf{\varphi }$	40	0.5	14
$\mathsf{\theta }$	49	0.3	11
$\psi$	0.5	1	4

displays the sliding mode gains for positions x, y, and z and orientations roll, pitch, and yaw for the obtained trajectory from using Dyna-Q learning.

The following results in Figure 31 were obtained:

Figure 32 depicts the control inputs from the sliding mode controller.

Figure 33 shows how the errors of positions x, y, and z and orientations roll, pitch, and yaw change with respect to time.

4.5. Discussion

The obtained results from the integration of the sliding mode controller with the Dyna-Q learning-based obstacle avoidance system were indeed remarkable. The Dyna-Q learning agent demonstrated impressive performance in learning optimal collision-free paths for the quadcopter in complex environments. Through iterations of simulation and interaction with various obstacles, the agent effectively applied a set of trajectories that enabled safe and efficient navigation. Employing these learned trajectories as the desired path for the quadcopter, the sliding mode controller showcased its ability to accurately track the trajectory, maintaining a high level of precision.

The sliding mode controller offers an excellent solution for trajectory planning and obstacle avoidance due to its robustness, adaptability, and ability to manage constraints. Its inherent capability to swiftly adjust control inputs to follow optimized trajectories, as demonstrated by its integration with Dyna-Q learning for obstacle avoidance, makes it a prime choice for achieving high-performance autonomous navigation. The SMC’s capacity to ensure accurate and safe trajectory tracking, even in the presence of disturbances and uncertainties, makes it a key choice for enhancing the capabilities of autonomous systems operating in complex and obstacle-rich environments.

At last, the integration of Dyna-Q learning and the sliding mode controller presents a powerful solution for obstacle avoidance in autonomous quadcopter navigation. This approach, driven by data-driven reinforcement learning and robust real-time control, demonstrates remarkable efficiency, safety, and adaptability. By enabling quadcopters to learn optimal trajectories and swiftly respond to dynamic environments, this approach holds great promise for enhancing quadcopter autonomy and successfully navigating intricate space.

5. Conclusions

The objective of this study was to formulate a precise mathematical model for a quadcopter and to devise three distinct control strategies: linear PID, nonlinear fractional-order PID, and sliding mode controllers, with the aim of stabilizing the quadcopter’s behavior. Through extensive simulations and precisely refining these controllers, important results were concluded. Remarkably, the fractional-order PID controller demonstrated superior performance compared to the conventional PID controller when it came to accurately tracking flight paths that exhibited dynamic changes. Additionally, the sliding mode controller showcased exceptional proficiency in handling the complexities of nonlinear dynamics and external disturbances. The utilization of the sliding mode controller extended to trajectory planning. Notably, through the integration of Dyna-Q learning, the quadcopter’s navigational capabilities were augmented, setting the stage for enhanced autonomous navigation. Since the sliding mode controller (SMC) offered the best results among the three controllers in nonlinearity handling, it was chosen to be combined with Dyna-Q learning for quadcopter trajectory planning. Together, the SMC’s precise control and Dyna-Q’s adaptive planning create a powerful system for achieving efficient and reliable trajectory planning in dynamic environments.

In essence, this study successfully accomplished its dual objectives by formulating an adept mathematical model for the quadcopter and introducing three controllers. The fractional-order PID controller emerged as a frontrunner in adapting to varying flight paths, while the sliding mode controller excelled in managing complexities. This study’s insights contribute to the advancement of both quadcopter control methodologies and trajectory planning, with potential applications in autonomous aerial navigation.