Adaptive Control Strategy for the PI Parameters of Modular Multilevel Converters Based on Dual-Agent Deep Reinforcement Learning

Abstract

As renewable energy sources are integrated into power grids on a large scale, modular multilevel converter-high voltage direct current (MMC-HVDC) systems face two significant challenges: traditional PI (proportional integral) controllers have limited dynamic regulation capabilities due to their fixed parameters, while improved PI controllers encounter implementation difficulties stemming from the complexity of their control strategies. This article proposes a dual-agent adaptive control framework based on the twin delayed deep deterministic policy gradient (TD3) algorithm. This framework facilitates the dynamic adjustment of PI parameters for both voltage and current dual-loop control and capacitor voltage balancing, utilizing a collaboratively optimized agent architecture without reliance on complex control logic or precise mathematical models. Simulation results demonstrate that, compared with fixed-parameter PI controllers, the proposed method significantly reduces DC voltage regulation time while achieving precise dynamic balance control of capacitor voltage and effective suppression of circulating current, thereby notably enhancing system stability and dynamic response characteristics. This approach offers new solutions for dynamic optimization control in MMC-HVDC systems.

1. Introduction

In the context of the global energy structure transitioning towards a high proportion of renewable energy, high voltage direct current (HVDC) transmission, characterized by its large capacity and long-distance capabilities, has emerged as a core solution for integrating renewable energy into the grid [1]. HVDC systems based on modular multilevel converters (MMC) have been widely deployed in offshore wind power and desert photovoltaic applications [2]. However, their dynamic control faces significant challenges. (1) The stochasticity and intermittency of renewable energy generation can lead to grid power imbalance, subsequently triggering zero-sequence circulating currents and DC voltage fluctuations, which render the operational states of MMC highly dynamic and complex [3]. (2) Conventional controllers fail to meet MMCs’ control requirements under diverse operating conditions [4]. (3) Both the topological structure and mathematical models of MMCs exhibit significant complexity [5]. These issues lead to a substantial increase in the complexity of MMC system control.

Researchers have developed a series of control methodologies specifically designed for the MMC to enhance its control effectiveness. From the perspective of the control hierarchy, these methodologies can be categorized into high-level control and low-level control. The high-level controller is mainly responsible for modulating external variables such as active power [6], reactive power [7], DC voltage [8,9], and AC voltage and frequency [10]. In contrast, the low-level controller focuses on regulating internal variables within the MMC, including the regulation of submodule capacitor voltage and energy balance [11], circulating current suppression [12], and arm energy balancing [13]. In summary, all these control strategies rely on accurate mathematical modeling of the MMC for targeted design and place stringent demands on the performance of the employed controllers.

To address the stringent control requirements of MMC, researchers have proposed a variety of controllers, including sliding mode controllers (SMCs), model predictive controllers, proportional-resonant (PR) controllers, reduced-order generalized integrators, and PI controllers [14]. Despite challenges such as complex parameter tuning and relatively slow dynamic response, the structural simplicity and strong robustness of PI controllers ensure their widespread use in various control strategies and practical engineering applications. Therefore, research on and improvement of PI controllers hold significant theoretical importance and practical value for enhancing system control precision and engineering reliability.

Compared with conventional proportional integral (PI) controllers, fuzzy PID controllers demonstrate enhanced transient performance with accelerated response characteristics and mitigated steady-state errors [15]. However, their reliance on predefined rules limits their dynamic regulation capabilities. Moreover, intelligent optimization algorithms, such as ant colony optimization [16], particle swarm optimization [17], water cycle algorithms [18], and genetic algorithms [19], have been applied to the real-time tuning of PI controller parameters, addressing the limitations of traditional static parameter adjustment methods. Nevertheless, when dealing with complex, strongly coupled system models, these algorithms may suffer from limited convergence rates, potentially compromising control performance. Another approach involves combining PI controllers with other control strategies to form hybrid controllers. For example, [20] integrates a PI controller with an SMC to enhance system stability; [21] combines a bang-bang funnel controller with a PI controller to improve fault ride-through capability; and [22] incorporates a PR controller with a PI controller to suppress circulating currents. However, the parameter settings in these hybrid controllers still heavily depend on empirical adjustments, which makes it challenging to achieve globally optimal control performance. Moreover, most hybrid control architectures feature fixed structures that are incapable of adaptively adjusting controller parameters based on operational states, thereby constraining their ability to satisfy dynamic demands under complex operating conditions.

In recent years, the rapid proliferation of renewable energy sources, such as photovoltaic and wind power [23], has significantly increased the complexity and uncertainty of power grid operating environments. To address this challenge effectively, artificial intelligence technologies, especially deep reinforcement learning (DRL), have become indispensable tools in power system control [24]. By leveraging its end-to-end environmental perception and adaptive policy optimization capabilities, DRL has demonstrated remarkable performance in various scenarios, including voltage regulation [25], load frequency control [26,27], and emergency control of power systems [28,29]. From the perspective of algorithm architecture evolution, the Deep Q-Network has shown notable proficiency in tasks with discrete action spaces [30]. However, it is unsuitable to meet the continuous control requirements of MMC. The Deep Deterministic Policy Gradient (DDPG) algorithm supports continuous control but faces challenges such as Q-value overestimation and a tendency to converge to local optima during practical applications [31]. The Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm addresses these limitations by incorporating a dual Q-network structure and delayed update mechanism, thereby enhancing convergence stability and learning efficiency [32]. Moreover, DRL includes other significant algorithms, such as proximal policy optimization [33] and soft actor-critic, each offering unique advantages across different application scenarios. Compared to traditional single-agent architectures, multi-agent DRL can more efficiently handle high-dimensional nonlinear systems like MMCs, which consist of strongly coupled submodules. This capability is achieved through distributed policy optimization and local-global reward coordination mechanisms [34].

To address the insufficient regulation capability of traditional controllers under complex MMC operating conditions, as well as the challenges of intricate design and heavy reliance on precise mathematical models, this article proposes a dual-agent adaptive PI control architecture based on the TD3 algorithm (DA-TD3). The core innovations are summarized as follows:

(1) A dual-agent framework incorporating a collaborative optimization mechanism is developed. Agent 1 specializes in generating PI control parameters for voltage and current dual-loop control, while Agent 2 focuses on tuning PI parameters for capacitor voltage balance control. Both agents share key state variables (DC voltage error) and utilize differentiated weight coefficients in their respective reward functions to ensure DC voltage stability. This design not only achieves a specialized division of labor in subsystem control but also reinforces the global constraint on the primary control objective (DC voltage stability) through hierarchical weighting and shared observations. By establishing collaborative strategies among agents, the trade-off between local optimization and global stability in multi-objective control scenarios is effectively addressed, leading to a notable improvement in the algorithm’s convergence speed.

(2) Leveraging the data-driven nature of the TD3 algorithm, the DA-TD3 architecture enables online adaptation to complex operating conditions through end-to-end autonomous learning. Moreover, this architecture overcomes the reliance on precise mathematical models inherent in traditional methods and avoids the limitations of manual parameter tuning based on empirical experience. Ultimately, the trained agents can detect real-time changes in operating conditions and generate continuously optimal PI controller parameters, thereby achieving rapid and stable control of the MMC system.

The remainder of this article is structured as follows. Section 2 describes the MMC topology and its fundamental control principles. Section 3 elaborates on the MMC control strategy based on the DA-TD3 algorithm. Section 4 validates the effectiveness of the proposed algorithm through simulation experiments. Section 5 summarizes the entire article.

2. Topology and Control Principles of MMCs

2.1. Topological Structure and Mathematical Modeling of MMCs

Figure 1 depicts the topology of a single-phase MMC, which comprises upper and lower arms. Each arm comprises N identical submodules (SM) connected in series. The submodules adopt a half-bridge structure, which includes two insulated gate bipolar transistors (T₁/T₂), two reverse-parallel diodes (D₁/D₂), and a submodule capacitor (C). By controlling the switching states of the IGBTs, the submodule can either output the capacitor voltage V_c or be bypassed. Furthermore, arm inductors L_arm and resistors R_arm are incorporated to mitigate circulating currents and limit fault current transients.

The mathematical model of the three-phase MMC can be derived via single-phase analysis, leveraging the system’s symmetry. As shown in Figure 2, the equivalent circuit model of a single-phase MMC is presented. For each phase j (where j = a, b, c), the electrical potentials across the upper and lower arms are respectively designated by the symbolic representations V_{u_j} and V_{l_j}. The voltage drop across the arm impedance is denoted as V_{diff_j}. The three-phase AC grid voltage, equivalent grid resistance, and equivalent grid inductance are represented by V_{g_j}, R_s, and L_s, respectively. The neutral points are N_dc (DC-side neutral) and N_ac (AC-side grounded neutral).

The dynamic equations for the upper and lower arms can be formulated as follows:

$$\left\{\begin{array}{c}{\displaystyle \frac{V_{dc}}{2}}-V_{u\_j}-R_{arm}{i}_{u\_j}-L_{arm}{\displaystyle \frac{d_{iu\_j}}{dt}}+R_s{i}_j+L_s{\displaystyle \frac{d{i}_j}{dt}}-V_{g\_j}=0\\ -{\displaystyle \frac{V_{dc}}{2}}+V_{l\_j}+R_{arm}{i}_{l\_j}+L_{arm}{\displaystyle \frac{d{i}_{l\_j}}{dt}}+R_s{i}_j+L_s{\displaystyle \frac{d{i}_j}{dt}}-V_{g\_j}=0\end{array}\right.$$

(1)

To achieve balanced three-phase power, the DC I_dc is equally distributed across three phases, while the output current undergoes symmetrical division between the upper and lower arms. This current allocation principle leads to the following relationships for the arm currents:

$$\left\{\begin{array}{l}{i}_{u\_j}={\displaystyle \frac{I_{dc}}{3}}+i_{diff\_j}+{\displaystyle \frac{i_j}{2}}\hfill \\ i_{l\_j}={\displaystyle \frac{I_{dc}}{3}}+i_{diff\_j}-{\displaystyle \frac{i_j}{2}}\hfill \end{array}\right.$$

(2)

In Equation (2), i_{diff_j} represents the circulating current that consists solely of the AC component. Therefore, the mathematical representation of circulating current (denoted as i_{cir_j} is derivable through the following analytical formulation:

$$\left\{\begin{array}{l}{i}_{cir\_j}=i_{u\_j}-{\displaystyle \frac{i_j}{2}}=i_{l\_j}+{\displaystyle \frac{i_j}{2}}\hfill \\ i_j=i_{u\_j}-i_{l\_j}\hfill \end{array}\right.$$

(3)

By summing up the two equations presented in (1) and substituting the resultant expression into i_j in (2), the external dynamic equation of the MMC can be systematically derived as follows:

$$\left(L_{arm}+2L_s\right){\displaystyle \frac{d{i}_j}{dt}}=-\left(R_{arm}+2R_s\right)i_j+V_{u\_j}-V_{l\_j}+2V_{g\_j}$$

(4)

By subtracting the two equations in (1) and substituting the resultant expression into i_{diff_j} in (2), the internal dynamic equation of the MMC can consequently be formulated as follows:

$$V_{diff\_j}=L_{arm}{\displaystyle \frac{{di}_{diff\_j}}{dt}}+R_{arm}{i}_{diff\_j}+R_{arm}{\displaystyle \frac{I_{dc}}{3}}={\displaystyle \frac{V_{dc}}{2}}-{\displaystyle \frac{V_{u\_j}+V_{l\_j}}{2}}$$

(5)

Equations (4) and (5) collectively form the theoretical foundation of the MMC control strategy.

2.2. The Control Strategy of MMCs

Figure 3 illustrates the voltage and current dual-loop structure, which is widely employed in MMC control [35]. In this structure, i_d and i_q represent the d-axis and q-axis components of the three-phase current, while V_d and V_q correspond to the d-axis and q-axis components of the three-phase voltage. Additionally, the DC output voltage is represented by V_dc, while the phase-locked loop generates the angular frequency ω.

Capacitor voltage balancing control consists of two components: energy averaging and voltage equalization. Given the three-phase symmetry, the subsequent analysis focuses on phase A as a representative example.

As shown in Figure 4, the energy averaging control regulates the average submodule capacitor voltage V_{c_av} through the outer voltage loop to track the reference value V_{c_ref}. The output of this loop serves as the reference input i_{cir_ref} for the inner circulating current controller, which suppresses circulating currents and ensures uniform energy distribution among the submodules [36]. When V_{c_ref} exceeds V_{c_av}, the circulating current reference i_{cir_ref} increases, promoting capacitor charging; conversely, a reduction in i_{cir_ref} induces capacitor discharging. By forcing the actual circulating current to track the reference value through feedback control, the independent circulating current loop ensures stable regulation of V_{c_av} under load disturbances. The formula for calculating V_{c_av} is provided below, with N corresponding to the number of submodules per arm.

$$V_{c\_av}={\displaystyle \frac{1}{2N}}{\displaystyle \sum _{n=1}^{2N}{V}_{c\_n}}$$

(6)

Voltage equalization control is determined by both the polarity of arm currents and the magnitude of capacitor voltage tracking errors in submodules. As shown in Figure 5, a proportional (P) controller modulates these reference voltages to enforce convergence toward their target values, with the proportional gain determining the trade-off between dynamic response and steady-state error. This closed-loop architecture guarantees submodule voltage uniformity under both transient and steady-state operating conditions.

Taking the upper arm of Phase A as an example, the capacitor voltage equalization control is adjusted based on the directions of the currents in both the upper and lower arms. Consequently, the polarity of V_{B_n_ref} is determined by i_{u_a} and i_{l_a}. When V_{c_ref} is greater than V_{c_n}, the converter must acquire energy from the DC side to charge the arm capacitor. Specifically, if i_{u_a} is greater than 0, V_{B_n_ref} becomes positive, and power flows into the converter; if i_{u_a} is less than 0, V_{B_n_ref} becomes negative, and power still flows into the converter. Conversely, when V_{c_ref} is less than V_{c_n}, the converter supplies energy to the DC side, causing the upper arm capacitor to discharge. In this case, if i_{u_a} is greater than 0, V_{B_n_ref} becomes negative, and power flows to the DC side; if i_{u_a} is less than 0, V_{B_n_ref} becomes positive, and power also flows to the DC side. The specific adjustment quantity for the capacitor voltage of the upper arm is given as follows:

$$V_{B\_n\_ref}=\left\{\begin{array}{l}{K}_1\left(V_{c\_ref}-V_{c\_n}\right),\hspace{1em}\hspace{0.17em}{i}_{u\_a}>0\hfill \\ -K_1\left(V_{c\_ref}-V_{c\_n}\right),\hspace{1em}{i}_{u\_a}<0\hfill \end{array}\right.\left(n=1,\dots N\right)$$

(7)

Similarly, the adjustment amount for the capacitor voltage in the lower arm is given as follows:

$$V_{B\_n\_ref}=\left\{\begin{array}{l}{K}_1\left(V_{c\_ref}-V_{c\_n}\right),\hspace{1em}\hspace{0.17em}{i}_{l\_a}>0\hfill \\ -K_1\left(V_{c\_ref}-V_{c\_n}\right),\hspace{1em}{i}_{l\_a}<0\hfill \end{array}\right.\left(n=1,\dots N\right)$$

(8)

Based on the single-phase equivalent circuit illustrated in Figure 2, the reference capacitor voltage value V_{C_n_ref} for each submodule can be derived through the following relationship:

$$\left\{\begin{array}{l}{V}_{C\_n\_ref}=-{\displaystyle \frac{V_{t\_a}}{N}}+{\displaystyle \frac{V_{dc}}{2N}}\hspace{1em}(\mathrm{Upper}\mathrm{arm})\hfill \\ V_{C\_n\_ref}={\displaystyle \frac{V_{t\_a}}{N}}+{\displaystyle \frac{V_{dc}}{2N}}\hspace{1em}\hspace{1em}(\mathrm{Lower}\mathrm{arm})\hfill \end{array}\right.$$

(9)

Ultimately, the amplitude of the modulation wave for implementing the carrier phase-shifting strategy can be determined:

$$V_{n\_ref}=V_{A\_ref}+V_{B\_n\_ref}+V_{C\_n\_ref}$$

(10)

Consequently, the corresponding PWM signals can be generated to combine the output voltages of activated submodules, thereby synthesizing the arm output voltage waveform.

Most control strategies rely on PI controllers to achieve their functional objectives. Although they can achieve the objectives of stabilizing the DC-side output voltage of the MMC and balancing the capacitor voltages of the submodules, manual tuning of the PI controller parameters is required, and fixed parameters are insufficient to adapt to the complex operating conditions of the MMC. Notably, during dynamic processes, traditional PI control demonstrates limitations in response speed and overshoot suppression, which may lead to exacerbated voltage fluctuations in the submodules or failure in circulating current suppression. To address these limitations, this article proposes a DA-TD3 structure to dynamically adjust PI controller parameters. The methodology capitalizes on the end-to-end environmental perception and adaptive policy optimization capabilities of DRL, enabling effective management of the MMC’s complex and variable operating conditions. Section 3 provides a detailed explanation of the DA-TD3 structure’s principle and training parameters.

3. DA-TD3 Algorithm for Optimized Control of MMCs

The TD3 algorithm builds upon the DDPG algorithm, significantly improving stability and robustness. Furthermore, the dual-agent architecture attains equilibrium between dynamic response speed and steady-state accuracy through collaborative mechanisms [37]. The combination of these two approaches is thus better suited for addressing complex decision-making problems characterized by high coupling and multiple variables.

3.1. The Core Principles Underlying the TD3 Algorithm

In the TD3 algorithm, two independent critic networks (Q₁ and Q₂) are employed to estimate Q-values, which are parameterized by θ₁ and θ₂, respectively. In this article, a dual-agent framework is adopted, consisting of two actor networks (k = 1, 2) and four critic networks (j = 1, 2, 3, 4). Each agent (Agent 1 and Agent 2) utilizes two dedicated critic networks—Q₁ and Q₂ for the first agent and Q₃ and Q₄ for the second agent––with each critic network parameterized by its corresponding θ_j. All critic networks are updated synchronously through the recursive Bellman equation, expressed as follows:

$$Q^{\pi }({s^k}_t,{a^k}_t)=r({s^k}_t,{a^k}_t)+{\gamma }^k{Q}^{\pi }\left({s^k}_{t+1},\pi ({s^k}_{t+1})\right),k=1,2$$

(11)

To approximate the optimal Q function, the mean squared Bellman error function is utilized to quantify the degree to which θ_j satisfies the Bellman equation. The specific expression is as follows:

$$L({\theta }_j,\mathcal{R})={\mathbb{E}}_{(s,a,r,s^{\prime })~\Re }\left[{\left(Q_{{\theta }_j}(s^k,a^k)-y^k(r^k,s^k\prime )\right)}^2\right],k=1,2; j=1,2,3,4$$

(12)

By employing clipped double Q-learning to compute Q-values and select the minimum target estimate, this dual mechanism effectively alleviates Q-value overestimation. Another key enhancement is target policy smoothing regularization, which involves adding a small random noise component ε to the target action, allowing the policy to explore the Q value function error, smooth the variations in actions, and prevent the critic network from overfitting at the narrow peaks of Q values [38]. Here, Gaussian noise is utilized, and the specific expression is as follows:

$${\epsilon }^k~\mathrm{clip}({\mathcal{N}}^k(0,{\sigma }^k),-c^k,c^k),k=1,2$$

(13)

Based on the above theory, the parameter update formula for the comment network is obtained as follows:

$$\left\{\begin{array}{l}{y}^k(r^k,s^k\prime )=r^k(s^k,a^k)+{\gamma }^{k}mi{n}_{j=1,2,3,4}{Q}_{{{\theta }^{\prime }}_j}\left(s^k\prime ,{\pi }_{{\varphi }^k\prime }(s^k\prime )+{\epsilon }^k\right)\hfill \\ {\epsilon }^k~clip({\mathcal{N}}^k(0,{\sigma }^k),-c^k,c^k)\hfill \end{array},k=1,2\right.$$

(14)

Here, r^k(s^k, a^k) denotes the instantaneous reward, γ^k ∈ [0, 1] is the discount factor, σ^k specifies the standard deviation of Gaussian noise, and c^k defines the truncation boundary.

The actor network can be updated via gradient ascent based on the expected return ∇ϕ^kJ(ϕ^k), which is formulated as follows:

$${\nabla }_{{\varphi }^k}J({\varphi }^k)={\mathbb{E}}_{s^k~\mathcal{R}}\left[{\nabla }_{a^k}{Q}_{{\theta }_j}(s^k,a^k){|}_{a^k=\pi (s^k)}{\nabla }_{{\varphi }^k}{\pi }_{{\varphi }^k}(s^k)\right],k=1,2 j=1,3$$

(15)

The delayed policy update pertains to the circumstance where the actor network is updated only once for each d update of the critic network, with d being equal to 2 in this article, aiming to mitigate policy oscillation. Additionally, the TD3 algorithm likewise employs the soft update of the target network. The target network parameters are gradually synchronized using the soft update coefficient τ^k, enhancing stability. The soft weights (θ_j, ϕ^k) are updated according to the following expression:

$$Soft update\left\{\begin{array}{ll}{\theta }_j\prime \leftarrow {\tau }^k{\theta }_j+(1-{\tau }^k){\theta }_j\prime ,\hfill & j=1,2,3,4\hfill \\ {\varphi }^k\prime \leftarrow {\tau }^k{\varphi }^k+(1-{\tau }^k){\varphi }^k\prime \hfill & k=1,2\hfill \end{array}\right.$$

(16)

3.2. Dual-Agent Training Mechanism in DA-TD3

This article proposes a DA-TD3 collaborative control framework, as depicted in Figure 6, to address the multi-objective control requirements of MMCs. The architecture is developed based on the multi-agent Markov game framework [39] and incorporates three key collaborative mechanisms tailored to the multi-objective and strongly coupled nature of MMCs, as follows:

(1) Task decoupling mechanism: By isolating voltage and current dual-loop control (Agent 1) from capacitor voltage balance control (Agent 2), the traditional high-dimensional coupled control problem is transformed into two independent optimization tasks. Specifically, Agent 1 specializes in generating optimal double-loop PI parameters for dynamic control, while Agent 2 independently tunes capacitor voltage balancing control parameters to maintain steady-state equilibrium. Through orthogonal design of the action space, these agents achieve a specialized division of labor, effectively reducing control complexity.

(2) Target negotiation mechanism: A hidden communication channel between agents is established through the shared DC voltage error ΔV_dc, its integral term, and asymmetric reward weights, enabling autonomous trade-offs between local optimization and global stability. This ensures that each agent can pursue its objectives while collaboratively maintaining the overall system performance.

(3) Collaborative parameter optimization mechanism: By differentially configuring hyperparameters, such as learning rates and exploration noises, and adjusting the network structures for the two agents, specialized control objectives are achieved. Specifically, Agent 1 employs high learning rates and exploration noise levels to rapidly track DC voltage dynamics and ensure transient stability, whereas Agent 2 operates with low learning rates and minimized exploration noise to maintain capacitor–voltage equilibrium while ensuring long-term steady-state accuracy.

Figure 7 illustrates the actor-critic network architecture of the agent. The subsequent sections will systematically elaborate on and analyze the construction of dual intelligent agents and their collaborative mechanisms.

For Agent 1 (voltage and current dual-loop control), the state space can be defined as S₁ = [Δi_d, Δi_q, ΔV_dc, ∫Δi_d dt, ∫Δi_q dt, ∫ΔV_dc dt]. Specifically, Δi_d and Δi_q represent the dq-axis current errors, while ΔV_dc denotes the DC voltage tracking error. These errors capture the complete dynamic characteristics of both the dq-axis currents and the DC voltage regulation. The error components (Δi_d, Δi_q, ΔV_dc) directly quantify transient control deviations, serving as real-time system state indicators. The integral components (∫Δi_d dt, ∫Δi_q dt, ∫ΔV_dc dt) incorporate accumulated historical error information, thereby enhancing sensitivity to steady-state deviations.

The action space of Agent 1 can be defined as A₁ = [K_p1, K_i1, K_p2, K_i2], where the voltage outer-loop PI parameters (K_p1, K_i1) and current inner-loop PI parameters (K_p2, K_i2) are dynamically adjusted through TD3’s policy in a continuous action space, enabling real-time parameter adaptation for enhanced control performance. During exploration, the agent improves through continuous trial-and-error learning, making the selection of an appropriate reward function R critical to guide its learning process. The reward function R₁ (ΔV_dc, Δi_d, Δi_q, V_{dc_ov}) for Agent 1 is defined as follows:

$$R_1=-\left({\alpha }_1|\Delta V_{dc}|+{\alpha }_2|\Delta i_d|+{\alpha }_3|\Delta i_q|\right)-\beta \cdot V_{d\mathrm{c}\text{\_}\mathrm{ov}}$$

(17)

Here, both α and β represent weighting coefficients, with V_{dc_ov} quantifying DC voltage overshoot. Given the critical role of DC voltage (V_dc) stability in determining the power transmission capacity of the MMC, the weighting coefficient α₁ is assigned the highest weight to prioritize voltage regulation. Considering the dominant role of active current i_d in DC voltage regulation compared to i_q, the secondary coefficient α₂ is weighted heavier than α₃. Furthermore, the reward function implements an overshoot penalty mechanism: when V_{dc_ov} exceeds the threshold V_{ov_th}, the penalty term β· V_{dc_ov} is activated, effectively constraining the overshoot. Based on this design, and considering both control error minimization and training efficiency, after iteratively training the agent with various weightings, the final weighting ratio is determined as α₁:α₂:α₃ = 10:3:2, β = 20, and V_{ov_th} = 10%. Under these conditions, an agent with superior comprehensive performance can be achieved.

For Agent 2 (capacitor voltage balance control), the state space is defined as S₂ = [ΔV_dc, ΔV_c, Δi_cir, ∫ΔV_dc dt, ∫ ΔV_c dt, ∫ Δi_cir dt]. Here, the DC voltage error ΔV_dc serves as a shared state variable for the two agents, enabling coordinated control between them. ΔV_c represents the capacitor voltage error, which reflects the energy imbalance of the submodules in the MMC. Δi_cir represents the circulating current error, which characterizes the energy exchange between the arms.

The action A₂ = [K_pu, K_iu, K_p, K_i] output by Agent 2 corresponds respectively to the PI parameters of the voltage equalization control and the circulating current suppression control. The formulation of the reward function R₂(ΔV_c, Δi_cir, ΔV_dc, V_{dc_ov}) is described as follows:

$$R_2=-\left({\gamma }_1|\Delta V_c|+{\gamma }_2|\Delta i_{cir}|+{\gamma }_3|\Delta V_{dc}|\right)-\beta \cdot V_{d\mathrm{c}\text{\_}\mathrm{ov}}$$

(18)

Among the terms, γ and β serve as weighting coefficients, while V_{dc_ov} denotes the DC voltage overshoot. Given that capacitor voltage balancing is essential for the reliable operation of the MMC and therefore serves as the primary control objective of Agent 2, the weighting coefficient γ₁ associated with capacitor voltage balancing is assigned the highest priority among all control targets, followed by γ₂ for circulating current suppression, which holds secondary priority in the control hierarchy. To mitigate inter-agent control conflicts with Agent 1, the weighting coefficient γ₃ for DC voltage error is appropriately reduced. After extensive training and parameter tuning, the final weighting ratio is determined as γ₁:γ₂:γ₃ = 20:10:1. Under these conditions, an agent with superior comprehensive performance can be achieved. Moreover, Agent 2 adopts the same overshoot penalty mechanism as Agent 1, where β = 20 and V_{ov_th} = 10%. This design ensures that Agent 2 achieves capacitor voltage equilibrium while maintaining the stability of the main circuit.

Furthermore, the network architecture design in the TD3 algorithm constitutes a critical governing factor for agent training effectiveness. Structurally, the actor network of Agent 1 employs two hidden layers with 256 and 128 neurons, respectively. Critic network 1 utilizes a single hidden layer containing 128 neurons, while critic network 2 adopts three hidden layers with 64 neurons each. The actor network architecture of Agent 2 is identical to that of Agent 1. Additionally, both critic network 3 and critic network 4 incorporate two hidden layers with 256 and 128 neurons, respectively. The rectified linear unit (ReLU) is utilized as the activation function across all hidden layers, whereas the hyperbolic tangent (tanh) is employed for the output layer in the actor network. The training process of the DA-TD3 algorithm is detailed in Algorithm 1. In each episode, the states S_i (i = 1, 2) are randomly sampled from the corresponding environment, and the action A_i is selected from the continuous action space.

Building upon the theoretical foundation established above, Figure 8 illustrates the overall control block diagram of the MMC system under DA-TD3 control. Through the collaborative mechanisms discussed in this section, DA-TD3 achieves real-time, precise tuning of PI parameters under complex MMC operating conditions. The framework demonstrates its core advantage by decoupling the high-dimensional coupled optimization problem in traditional centralized strategies into two coordinated low-dimensional subspace tasks. Moreover, through autonomous policy optimization, DA-TD3 overcomes the reliance on precise mathematical models and manual parameter adjustments while maintaining the engineering applicability of PI controllers.

Algorithm 1. DA-TD3 algorithm [32]

Input: Operating environment of the MMC

1: Initialize the actor network μϕ1 and critic networks Qθ1, Qθ2 for Agent 1. Initialize the actor network μϕ2 and critic networks Qθ3, Qθ4 for Agent 2.

2: Initialize target network parameters:          ${\phi }_1^{\prime }\leftarrow {\phi }_1, {\theta }_1^{\prime }\leftarrow {\theta }_1, {\theta }_2^{\prime }\leftarrow {\theta }_2$ ${\phi }_2^{\prime }\leftarrow {\phi }_2, {\theta }_3^{\prime }\leftarrow {\theta }_3, {\theta }_4^{\prime }\leftarrow {\theta }_4$

3: Initialize independent replay buffers M1 and M2.

4: for episode = 1, 2, … N do

5:  Agents 1 and 2 select actions with exploration noise:            $a^k~{\pi }_{\varphi k}(s^k)+{\epsilon }^k,{\epsilon }^k~\mathcal{N}(0,{\sigma }^k)$

6:  Perform joint action (a1, a2), observe rewards r1, r2, and new states s1′, s2′.

7:  Store interaction data in the reply buffers:            $M_k\leftarrow (s^k,a^k,r^k,s^k\prime ),k=1,2$

8:  Sample mini-batch of m1, m2 transition from reply buffers M1, M2.

9:  Compute greedy actions for the next states through         ${\tilde{a}}^k\leftarrow {\pi }_{{\varphi }^k\prime }(s^k\prime )+{\epsilon }^k,\mathrm{where}{\epsilon }^k~\mathrm{clip}\left(\mathcal{N}(0,{\tilde{\sigma }}^k),-c^k,c^k\right),k=1,2$

10:   Compete target Q-values according to (14)

11:   Update critic networks through:         ${\theta }_j\leftarrow {\mathrm{argmin}}_{\theta j}{m_k}^{-1}{\displaystyle \sum {\left(y^k-Q_{{\theta }_j}(s^k,a^k)\right)}^2},k=1,j=1,2\mathrm{and}k=2,j=3,4$

12:   if Episode mod 2 = 0 then

13:   Update actor networks ϕ1, ϕ2 by deterministic policy gradient according to (15)

14:   Update target networks according to (16)

15:   end if

16: end for

4. Algorithm Training and Simulation Analysis

4.1. Deep Reinforcement Learning Algorithm Training

To validate the effectiveness of the proposed methodology, an MMC model utilizing the DA-TD3 algorithm was constructed within the MATLAB/Simulink simulation platform. The hyperparameter settings for reinforcement learning training are detailed in

Table 1. Main hyperparameters of the training process.

Parameter		Value
Episode number (N)		3000
Sample time (T1, T2)		[1 × 10−3, 1 × 10−3]
Learning rate for the actor [λa1, λa2]		[1 × 10−4, 1 × 10−4]
Learning rate for the critic [λc1, λc2]		[2 × 10−3, 5 × 10−4]
Discount factor [γ1, γ2]		[0.995, 0.999]
Replay buffer size [M1, M2]		[1 × 104, 1 × 104]
Mini batch size [m1, m2]		[128, 128]
Target update frequency [FT1, FT2]		[2, 2]
Soft target update factor [τ1, τ2]		[1 × 10−3, 5 × 10−3]
Policy updates frequency [d1, d2]		[2, 2]
Exploration model		Gaussian action noise
Initial value of noise standard deviation		[0.3, 0.1]
Decay rate of the standard deviation		[1 × 10−4, 1 × 10−5]
Agent 1PI parameter range [Kp1, Ki1, Kp2, Ki2]
0.05 ≤ Kp1 ≤ 1	2 ≤ Ki1 ≤ 20	10 ≤ Kp2 ≤ 50	100 ≤ Ki2 ≤ 500
Agent 2 PI parameter range [Kpu, Kiu, Kp, Ki]
0.5 ≤ Kpu ≤ 12	10 ≤ Kiu ≤ 500	1 ≤ Kp ≤ 20	10 ≤ Ki ≤ 600

, while the main parameters of the MMC system are listed in

Table 2. System specifications of the MMC.

Parameter	Value
Grid voltage Vg_abc (kV)	3.3
DC output voltage Vdc (kV)	6
Submodule capacitor C (mF)	7
Arm resistance Rarm (Ω)	0.04
Arm inductance Larm (mH)	13.5
Grid equivalent resistance Rs (Ω)	0.04
Grid equivalent inductance Ls (mH)	10

.

4.2. Simulation Analysis

This article establishes two operating conditions for analysis: (1) steady-state operation and (2) grid voltage sag. Under these operating conditions, the control performance of the intelligent agent trained with the DA-TD3 framework, the intelligent agent trained with DDPG algorithms, the PI controller based on a fuzzy algorithm, and the fixed-parameter PI controller for MMC systems was evaluated.

4.2.1. Steady-State Operating Condition

As illustrated in Figure 9, the steady-state performance of the DA-TD3 agent, DDPG agent, fuzzy PI controller, and conventional PI controller are comparatively analyzed through MMC simulations. The results indicate that DRL-based control strategies exhibit significantly enhanced dynamic response characteristics compared to traditional methods, though the DDPG agent demonstrates marginally inferior robustness relative to conventional approaches. The detailed findings are as follows:

The DA-TD3 agent achieves DC voltage regulation in 0.05 s, representing a 61.5% improvement over the conventional PI controller (0.13 s). It also outperforms both the DDPG agent (0.08 s) and fuzzy PI controller (0.1 s), confirming its superior rapid-response capability for complex operating conditions.

The DA-TD3 agent demonstrates smaller overshoot amplitudes compared to both fuzzy PI controllers and conventional PI controllers. While the DDPG agent achieves the minimal overshoot amplitude, its control system displays oscillations before reaching a steady state, suggesting compromised stability compared to the DA-TD3 agent.

In critical metrics such as submodule capacitor voltage, bridge arm current, and circulating current suppression, the MMC system controlled by the DA-TD3 agent demonstrates the best overall performance with minimal electrical quantity fluctuations and no abrupt changes. This further validates the DA-TD3 agent’s superiority in multi-task control scenarios.

In summary, the DA-TD3 agent achieves enhanced steady-state control performance compared to the DDPG agent, fuzzy PI controller, and conventional PI controller, delivering accelerated dynamic response and improved control precision.

4.2.2. Grid Voltage Sag Operating Condition

As illustrated in Figure 10, when the grid voltage drops to 0.7 pu at 0.3 s, the dynamic responses of critical electrical quantities in the MMC system are compared under control by the DA-TD3 agent, DDPG agent, fuzzy PI controller, and conventional PI controller. The results show that the DA-TD3-regulated DC voltage stabilizes within 0.05 s post-disturbance, maintaining consistency with its steady-state reference value, while the fuzzy PI controller requires 0.1 s for stabilization. In contrast, both the DDPG agent and conventional PI controller exhibit sustained DC voltage oscillations with slower response speeds. Regarding key indicators such as submodule capacitor voltage, arm current, and circulating current suppression, the DA-TD3 agent demonstrates the best overall performance.

In conclusion, the DA-TD3 agent exhibits superior dynamic response characteristics and anti-interference capabilities under grid voltage fluctuations, showcasing greater engineering application potential in complex MMC control scenarios.

5. Conclusions

This article addresses the following challenges in MMC dynamic control: complex and variable operating conditions, the inability of traditional controllers to fully meet control requirements due to reliance on empirical parameter tuning, and the difficulties in dynamic optimization of high-dimensional coupled multi-variable systems with complex mathematical models. A DA-TD3-based adaptive PI parameter control strategy is proposed. Through theoretical analysis and simulation verification, the following conclusions are drawn:

(1) Compared with traditional PI and fuzzy PI controllers, the DA-TD3 framework achieves dynamic optimization of PI parameters through its end-to-end autonomous learning mechanism, eliminating reliance on empirical parameter tuning. Its data-driven characteristics significantly enhance the controller’s adaptability to random disturbances and nonlinear operating conditions, addressing the dynamic performance limitations caused by fixed parameters in traditional methods.

(2) The DA-TD3 framework overcomes the shortcomings of DDPG algorithms, such as susceptibility to local optima and unstable convergence in complex systems, through the task decoupling mechanism, target negotiation mechanism, and collaborative parameter optimization mechanism between dual agents. Simulation results demonstrate that the DA-TD3 architecture exhibits superior robustness and dynamic response efficiency in multi-variable cooperative control, validating its potential in complex systems.

(3) The DA-TD3 framework avoids complex modeling and parameter derivation processes by leveraging DRL for autonomous policy optimization, eliminating dependence on precise mathematical models.

This framework provides an innovative solution for dynamic optimization control of MMC, demonstrating broad research prospects. Future research will focus on exploring technical solutions for algorithm lightweight design and migration learning-accelerated training while promoting iterative validation and scenario-specific deployment of the framework in hardware-in-the-loop simulation experiments and real-world engineering applications to expand its potential in engineering fields.