Implementation of a Modular Distributed Fault-Tolerant Controller for MMC Applications

Abstract

Centralized control algorithm limits the hardware flexibility of a modular multilevel converter (MMC). Therefore, distributed control structure has recently started to be seen in the industry application. Even though distributed controller reduces a single point of failure risk compared to the centralized controller, the failure risk of the entire control systems increases due to the number of local controllers. However, the distributed controller can be programmed in such a way as to replace the faulty local controller and sustain the MMC operation. In this paper, the distributed modular fault-tolerant controller is implemented in a laboratory-scale MMC prototype. The controller is built to control four SMs per phase for the proof-of-concept. Therefore, the MMC prototype is also built by two SMs per arm. The controller capability is validated with experimental and the Opal-RT result-time simulator results in a control-hardware-in-loop (CHIL) environment.

1. Introduction

Global energy demand increases daily, but the rise in energy generation does not increase as fast as the demand. The number of clean and environmentally friendly energy generation sites should be increased to meet the demand. However, most of the electric energy demand is still supplied by burning fossil fuels, so pollution keeps increasing. Energy demand should be supplied through sustainable energy sources to reduce the carbon footprint and eliminate the harmful effects of fossil fuel burning. Due to their complex and traditional structure, existing utility grid infrastructures are not yet ready to integrate many sustainable sources such as solar, wind, etc. [1]. In addition, sustainable sources vary naturally, so energy storage systems may be needed for stable integration. Undoubtedly, the need for clean, affordable, and sustainable energy is more significant than ever to protect the environment and meet the demand. Therefore, conventional AC power grid infrastructures should be modernized and strengthened to integrate bulk power from renewable energy sources. Additionally, energy storage systems should be developed to reduce the variability effect of renewable energy sources for oscillation-free integration [2]. However, modernizing the existing power grid structure might be challenging, time-consuming, and extremely costly.

On the other hand, high voltage direct current (HVDC) transmission is a versatile alternative to AC transmission for carrying more power from remote locations with fewer conductors and lower losses. HVDC transmission is more cost-effective for longer distances because it has minor capacitive losses than the HVAC, especially when the conductors are placed closer to the ground. Hence, reactive power compensation is not required along with the transmission as opposed to AC transmission. Unlike DC cables, HVAC lines are subject to corona discharge, so the conductors are bundled to increase the effective radius. However, bundling the conductors increases the overall line capacitance and reactive power consumption [3,4,5,6,7].

The HVDC grid is an excellent candidate to integrate renewable energy, yet high voltage transmission is necessary to lower transmission losses. Raising the voltage to hundreds of kV is challenging with two- or three-level converter topologies due to the limited number of series-connected power modules. Therefore, modular multilevel converter (MMC) has recently been the primary selection for high-voltage transmission applications due to its scalable and modular structure. An MMC can easily increase or decrease the voltage level with the series-connected power modules, called sub-module (SM). However, the performance of an MMC highly depends on the control structure as each SM capacitor voltage needs to be monitored and controlled at all times [8,9].

The MMC controller should manage AC and DC side voltage and internal MMC current. These tasks can be categorized under higher-level and lower-level control, respectively. In general, higher-level control aims to control the DC link voltage or the output power of the converter. The lower-level control is much faster than the higher-level control and manages the inner current, capacitor voltage balancing and averaging, and modulation. The circulating current (CC) control can also be categorized under lower-level control. These tasks can be performed by a central control unit (CCU) or distributed controllers.

In the case of a CCU, all computation is performed based on the measured signals and control commands in a central control unit such as a digital signal processor (DSP), field programmable gate array (FPGA), or application-specific integrated circuit (ASIC) chip. The centralized control algorithm is relatively fast with no significant control delays, but it limits hardware flexibility. In addition to the computational burden, modification of the controller and the communication links can be challenging for high-voltage MMC applications due to the vast number of SMs. A centralized controller is generally adopted for MMC-based medium voltage motor drive applications [10] or laboratory-scale prototype applications [11]. The CCU limits the scalability and modularity features of an MMC because the entire control structure needs modification in case of any hardware changes. A significant computational burden on CCU might be experienced in case of a high number of SM, and an undesirable overrun for each control cycle might be observed as a result.

On the other hand, the distributed control algorithm divides the tasks between the local controllers (LC) and a master controller. MMC-based industry applications have recently started moving towards a distributed-based control structure due to a flexible and easily modified structure [12,13]. In distributed control structure, LCs of the SMs can make the necessary computations with minimal communication with the master controller. Therefore, the central processor can significantly reduce communication bandwidth and computational load.

However, precise synchronization between the LCs and master controller is necessary for proper operation. Otherwise, improperly controlled power modules may cause unstable operation for an MMC application.

A distributed controller can significantly reduce the probability of a single point of failure. At the same time, the failure risk of the entire control system increases due to the increased number of LCs. Nevertheless, the distributed controller can be programmed in such a way that any faulty controller can be bypassed, and the adjacent LC can take care of the SM. Therefore, a well-structured distributed controller can handle the failure of a local controller(s).

This paper implements a distributed fault-tolerant controller for a downscaled MMC prototype. The aim is to prove the concept of MMC operation under LC failures. Controller failure may not be as frequent as other components in an MMC, such as IGBTs or SM capacitors, yet a controller can fail or malfunction due to lead corrosion, radiation damage, or latencies in the system. This paper shows continuous MMC operation under controller(s) failure. The results are validated with the experimental setup and the Opal-RT real-time simulator using the same controller. The rest of this paper is followed by the modular distributed fault-tolerant control algorithm for MMC applications in Section 2, experimental and real-time results in Section 3, and the conclusion in Section 4.

2. Modular Distributed Fault-Tolerant Controller Algorithm for MMC Applications

A three-phase MMC consists of six identical arms, and each arm accommodates a series of connected sub-modules (SM) and an arm inductor ($L_{arm})$. Various SM types are available for MMC applications, but half-bridge sm (HBSM), seen in Figure 1, is preferred in this paper. Each HBSM has one capacitor $(C_{SM})$, and two IGBT switches. Controlling the switches in the HBSM, each capacitor can be connected in series with the arm inductor or bypassed from the circuit. The switches can be controlled by a CCU or an LC. However, in the case of an MMC application with a high number of SM, such as 400 SM per arm, it might be challenging for a CCU to control all the SMs. Additionally, high-bandwidth communication between the CCU and the SMs is required due to the volume of the SMs.

On the other hand, the control tasks can be divided between the master controller and a bunch of LCs in the distributed control algorithm. In the case of the distributed controller, each SM or a group of SMs has a dedicated LC, which allows more straightforward modification in the control algorithm compared to the CCU. Distributed controllers for multilevel converters have become an appealing research topic due to easy modification, less data transfer, and faster computation. In addition, the distributed controller reduces the risks of the single point of failure.

Regardless of the controller scheme, arm voltages are determined based on the number of inserted SMs. Thus, increasing or decreasing the voltage level of an MMC application depends on the number of inserted modular SMs. The reference voltage for the upper ($v_{u,x})$ and the lower ($v_{l,x})$ arm can be determined based on AC and DC side dynamics as Equation (1) and Equation (2), respectively. $V_{DC}$ represents the DC link voltage, $V_{mx}$ represents the AC side voltage and $V_{cx}$ represents the circulating voltage Equation (3) induced on the arm inductors due to circulating current [14,15,16,17]. Circulating current occurs due to voltage differences between the arms.

$$v_{u,x}=\frac{V_{DC}}{2}-V_{mx}-V_{cx}$$

(1)

$$v_{u,x}=\frac{V_{DC}}{2}-V_{mx}-V_{cx}$$

(2)

$$v_{z,x}=L_o\frac{d{i}_{Z,x}}{dt}$$

(3)

$$n_{u,x}=\frac{1}{V_{cu}^x}{V}_{u,x}$$

(4)

$$n_{l,x}=\frac{1}{V_{cl}^x}{V}_{l,x}$$

(5)

$$i_{u,x}=i_{z,x}+\frac{i_x}{2}$$

(6)

$$i_{l,x}=i_{z,x}-\frac{i_x}{2}$$

(7)

$$i_{z,x}=\frac{i_{u,x}+i_{l,x}}{2}=i_{dc,x}+i_{circ,x}$$

(8)

The modulation index determines the number of inserted or bypassed capacitors based on the arm’s reference voltages. During a steady state, the insertion index for the upper and lower arm is determined in Equation (4) and Equation (5), respectively, where $V_{cu, l}^x$ represents individual capacitor voltage in the arms. Similarly, upper ($i_{u,x})$ and lower ($i_{l,x})$ arm current can be expressed in terms of differential current and AC side current Equation (6) and Equation (7), respectively. Differential current ($i_{z,x})$ Equation (8) has AC and DC parts. The AC part is the circulating current, the same for all the phases, whereas the DC part is one-third of the DC current per phase under a steady state. As seen in Figure 1, the series connected SMs are the backbone of the MMC structure. Increasing or decreasing the number of SMs allows for adjusting the generating voltage level. In case of a high number of MMC applications with a centralized controller scheme, any changes in the hardware require extensive control algorithm modification. Therefore, a centralized controller indirectly limits hardware flexibility. Unlike the centralized control scheme, distributed control scheme helps reduce the burden on the master controller and assigns some of the tasks to local controllers. The distributed modular controller is designed based on one master and dedicated four LCs for the one phase of the MMC. The master controller mainly controls the total energy in the MMC arms by measuring the AC and the DC side and communicates with the local controllers for the capacitor voltage balancing. An illustration of the modular distributed fault-tolerant controller can be seen in Figure 2. Averaging (phase balancing) controller, seen in Figure 3, ensures that phase voltages are distributed equally based on Equation (9) and the reference voltage generated. Differential current compensates for the error when there is a mismatch between the average arm voltage and the individual SM voltage. The master controller synchronizes the LCs based on their reference frequency. The synchronization process compares the reference frequency of the master controller and the local controller, and the error is compensated based on Equation (10) where $f_m$ is the reference frequency, $f_e$ is the error and $f_l$ is the frequency of a local controller. Similarly, the time error can be determined in Equation (11) as a function of frequency and compensated accordingly.

$$V_{dc}={\displaystyle \sum }_{n=1}^4{V}_{SMn,x ul}+L_{arm}\frac{d\left(i_{u,x}+i_{l,x}\right)}{dt}$$

(9)

$$f_e=f_m-f_l=w_e=2\pi f_e$$

(10)

$$t_e=\frac{f_e}{f_m}t$$

(11)

$$CA{M}_{i,1}=\{\begin{array}{c}0 if [F_{LC, i}\wedge F_{LC, i+1}\wedge F_{LC, i-1}]=1\\ 1 Otherwise \end{array}$$

(12)

In the literature, researchers have focused on increasing the resiliency of an MMC under SM-level failures, as the semi-conductor switches such as IGBTs or MOSFETs are more prone to open or short-circuit failures [18,19,20]. Therefore, various research methods are available to sustain the MMC operation under SM failure or shut down the system gracefully [21,22,23]. On the other hand, controller failure is not frequently expected as much, yet even unexpected latencies might cause a system failure. Therefore, the reliability of the DSPs is quantified with failure rates. Authors in [24] indicate that the total failure rate of a DSP board is 8.84 × 10⁻⁶ hour. Some potential failure causes are lead corrosion, radiation damage, undetected corruption, or non-initialization of pointers [25,26,27]. Therefore, controller failure should be considered, especially for safety-critical applications, and a fault-tolerant algorithm is necessary to sustain the MMC operation under a controller failure. This paper implements the distributed modular controller, seen in Figure 4, to handle LC failures. In case of an LC failure, the adjacent controller takes care of the faulty controller’s SM to sustain the MMC operation. The controller is built based on a two-dimensional redundant architecture [28], so each local (slave) controller has a one-dimensional array synchronized with the master controller. As seen in Figure 5, the controller has one master and twelve LCs. Each controller is implemented using Texas Instrument’s F28379D Digital Signal Processor (DSP) card. The LCs are connected to the master controller through a shared serial-communication bus. The output of these controllers is fed into dedicated Altera’s ACM-204-40C8 FPGA. The role of the FPGAs is to detect fail-over of the LCs. Therefore, each LC checks the neighbor controllers’ health status for every control cycle. If there is an LC failure, the faulty LC is bypassed, and the neighbor LC takes care of the faulty controller’s SM. In this design, each LC is directly connected with all the LCs, and data are transferred using a series data port. Therefore, each LC controls an SM and communicates with the neighbor LCs. Thus, the measured signals and the computation results are compared with the neighbor LCs to validate the accuracy of the output based on the neighbor LC. In case of an LC failure, both the neighbor controllers compare the errors to detect the failure, as in Figure 6. Then, one adjacent LC takes the faulty LC’s SM based on the FPGA’s decision. The controller availability matrix (CAM) is shown (12) to determine if there is any LC available to take the vacant SM where $F_{LC, i}$ is the failed controller, $F_{LC, i+1}$ is the upper controller, and $F_{LC, i-1}$ is the lower controller. The dedicated FPGA runs the matrix and chooses the necessary controller for the vacant module. Each FPGA has six PWM signals as inputs from three controllers, the main and two neighbor controllers.

In addition, two error signals are fed into the FPGAs to determine the health status. Two PWM signals from two LCs are compared to check the mismatch. If the PWM signals of the neighbor LCs are not the same at an instant, the counter, seen in Figure 6, exceeds a predefined threshold, and the error signal goes high. This event is named a fail-over condition, and it has two steps. The first is to check the fault signal to decide if any of the main controllers are faulty. The main controller’s output is compared with the adjacent controllers. If the outputs of these comparisons are different, the main controller does not create the appropriate output signal. This situation may occur if the main processor gets faulty or the controller has no supply voltage. If such a case occurs and the output signal comparison of the adjacent LCs is not the same, the fail-over signal gets high, and one adjacent LC replaces the faulty LC. If both the adjacent LCs are faulty based on the health status check seen in Figure 7, then there is no available LC to take care of the SM. In such a case, the SM will be bypassed. The scope of this paper covers an LC failure, so bypassing the SM through the controller will be validated in a separate research paper.

The role of the master controller is to control the AC voltage, phase circulating current, and capacitor voltage averaging for each phase of the MMC. In addition, it is responsible for regulating output voltage through the active and reactive current based on the modulation signal and generate the reference voltage for the power-electronics switches, as seen in Figure 8. SM capacitor voltages are averaged based on the sensor reading of each phase to regulate the average voltage of each converter leg. On the other hand, each LC is responsible for balancing the capacitor voltages and generating the PWM signals for the switches [29,30,31]. Additionally, each LC manages the capacitor voltage balancing as in Figure 9. Then, the output of the balancing controller is fed into the PWM generation block.

3. Experimental and Real-Time Simulator Results

A three-phase MMC prototype with two SM per arm is designed to test the controller, which has four local controllers per phase. The test setup is developed to validate the modular fault-tolerant distributed controller under steady state and local controller failure conditions. Each phase leg of the MMC prototype is connected to a constant DC power supply, an N5700 series 150 V, 10 A, 1500 W from Keysight Technologies. The DC source provides stable output power, and the MMC is operated in an inverter mode. The AC side of the porotype is connected to an inductive load, represented by a 2.5 mH inductor and 20 Ω resistor. An illustration of the experimental setup can be seen in Figure 10, and a photo of the test setup is shown in Figure 11.

A list of parameters of the prototype can be found in

Table 1. System parameters.

DC Voltage	80–100 V	Number of SM per arm	2
AC Frequency	60 Hz	Capacitor Ref. Voltage	50 V
Switching Frequency	10 kHz	SM Capacitance	3.9 mF
Load Inductance	2.5 mH	Load Resistance	22 Ω
Arm Inductance	1 mH	IGBT at Tc = 25 °C	60 A, 650 V

. A photo of the MMC phase module can be seen in Figure 12; each phase leg has four SMs and an isolated gate driver. Each HBSM has a 60 A, 650 V field stop trench gate IGBT (STGW60H65DRF) with an ultrafast diode attached to a heat sink and a 3.9 mF snap-in capacitor. LEM ^® sensors, placed on a single board, are used to measure the signals from the SMs. The measured signals are fed into a signal conditioning board, seen in Figure 13, to filter the analog signal and amplify the useful signals for the controller cards. Figure 14 shows the experimental results for the SM capacitor when the DC voltage is changed from 0 p.u to 0.8 p.u. It can be seen that all four capacitors share the DC bus voltage almost equally. The capacitor voltages’ summation is not quite equal to the DC link voltage due to the voltage drop on the relatively large arm inductor (1 mH), resulting in $2V_{cx}$ based on Equation (1) and Equation (2). The DC link voltage is increased to 100 V in Figure 15. The load current and the capacitor voltages of the upper SMs can be seen accordingly. Capacitor voltages of the SMs show that the balancing controller is effective.

The main scope of this paper is to test the controller capability under LC failure. Hence, the prototype is operated at 80 V DC link voltage to test LC failing scenarios. The arm, load, circulating current, and SM capacitor voltage can be seen in Figure 16 and Figure 17 during the steady state. The circulating current controller is disabled, and the dominant second-order harmonic can be seen in the circulating current. In addition, switching frequency circulating current is observed in the circulating current. As seen from the SM voltages, the balancing controller effectively balances the capacitor voltages.

The LC of the second SM (LC2) failed by cutting the supply voltage. The controller’s health is immediately reported based on Figure 7 to manage the SM through the adjacent controllers. The circulating current and the capacitor voltage momentarily respond while the adjacent controller takes over the SM2. Due to the dynamic response of the controller, there is a rise in capacitor voltage during the take-over, which is less than 5% of the rate value seen in Figure 18.

The circulating current shows a high magnitude due to the time delays involved during the detection and switching of control inputs from failed LC to healthy LC. The detection delay is around 500 µsec as the synchronization check occurs once every control cycle. The FPGA allocated for the LC failure detects the failure based on the loss of communication signal from the faulty LC.

Additionally, a delay is associated with deciding the available healthy LC for the particular SM. Therefore, momentary loss of control occurs due to the delays and a high magnitude of circulating current is observed as a result. Although there is a transition delay, the take-over process is still fast enough to observe no effect on the load current and the SM capacitor voltage balancing. Similarly, two LC failures are tested simultaneously by cutting the supply voltage. The health status of the controllers is reported, and the failure detection occurs within 500 µsec. The process of two LC failures is the same as in the previous case, but the number of control delays is increased. As a result, higher oscillation is observed in the circulating current in Figure 19. Still, the controller can sustain the SM capacitors with two healthy LCs. There is a rise in capacitor voltage during the take-over, which is around 5% of the rate value due to the dynamic response of the controller. The take-over process repeats itself until a healthy LC is available to take over an SM.

Table 2. The take-over chart during fail-over condition.

LC1	LC2	LC2	LC4	System Status
Failed	Takeover	No Change	No Change	Up
Takeover *	Failed	Takeover *	No Change	Up
No Change	Takeover *	Failed	Takeover *	Up
No Change	No Change	Takeover	Failed	Up
Takeover	Failed	Failed	Takeover	Up
Failed	Takeover	Takeover	Failed	Up
Failed	Takeover *	Failed	Takeover *	Up
Failed	Failed	No Change	No Change	Down
No Change	No Change	Failed	Failed	Down

* the dedicated phase FPGA decides which LC takes over based on the health status.

shows the algorithm of the take-over process during a fail-over condition. System status becomes down if there is no available LC to replace the faulty LC(s).

As can be seen, the controller can sustain the operation under any one LC failure. Further, the controller can sustain the operation under two LCs failures as long as the failures are not located in the middle and the corner LCs. For instance, the controller is still operational if LC2 and LC3 fail because LC1 and LC4 can take over the faulty LCs. However, if LC1 and LC2 fail simultaneously, the system must shut down. The reason is that there is no available neighbor controller to take over the LC1 in this case. The experimental results are verified with the Opal-RT real-time simulator, as seen in Figure 20. A control-hardware-in-loop (C-HIL) environment is developed with a modulation index of 0.92 using the same modular distributed fault-tolerant controller. The same power circuit is modeled in the Virtex 7 FPGA using MATLAB Simulink^®.

Due to the high switching frequency operation of the controller, the required simulation time-step is less than 1 µs. Although the time-step around 1µs satisfies the communication, this will significantly increase the simulation time. Therefore, power electronics components are chosen from the RT-event^® library to reduce the simulation time step and overall time. The C-HIL setup aims to test the controller capability before testing the hardware setup. The behavior of the current and voltages are compared and validated with the hardware result. The circulating current observed in the OPAL-RT result is more significant than in the hardware result. This is because the circulating current is fed to the controller during the OPAL-RT simulation and calculated as quantity using signals sampled in the OPAL-RT system’s CPU. In the prototype, the circulating current is measured through the arm currents using LEM sensors for each phase, but it is not possible to place a current measurement sensor in OPAL-RT that directly measures the circulating current. Due to this limitation, the arm currents are measured from the simulation in the CPU and processed to calculate the circulating current based on the discrete-time simulation. This process has communication delays between the simulation FPGA and the CPU. Therefore, the circulating current in the OPAL-RT results cannot be correlated directly to the measurement result because it is not a direct measurement. Yet, the aim is to compare the behavior of the current under LC failure. Figure 21 shows phase B’s load, circulating current, and upper SM capacitor voltages. This figure validates the experimental results shown in Figure 17 at 80 V DC link voltage. Similarly, Figure 22 shows the same result pattern as Figure 18 when LC2 is hard failed. Lastly, Figure 23 validates the experimental results under the failure of LC2 and LC3, as seen in Figure 19.

4. Conclusions

The MMC is one of the suitable topologies for medium and high-voltage applications. One of the reasons is that an MMC can generate almost a sinusoidal voltage with several hundred series connected SMs per arm. In addition, the number of SMs can be easily modified to adjust the system’s rating. However, the controller of an MMC should be modified accordingly based on the rating. The modification for hundreds of SMs might be challenging and time-consuming with a centralized control structure.

On the other hand, a distributed control structure is more flexible than a centralized controller. The reason is that a distributed controller has multi-control points, such as the master controller and the LCs. Hence, modifying control points is relatively easier than modifying a central controller with hundreds of control lines. Additionally, a distributed controller significantly eliminates the single point of failure compared to a centralized controller. On the other hand, the failure risk of the controller increases due to the increased number of LCs. However, a distributed controller can be programmed with resiliency.

This paper implements the distributed modular fault-tolerant controller for an MMC prototype. The controller is built to control four SMs per phase for the proof-of-concept. Therefore, the MMC prototype is built by two SMs per arm. The experimental results are validated using the same controller with the Opal-RT result-time simulator in a control-hardware-in-loop (CHIL) environment.

The results show that a distributed controller can sustain the MMC operation despite two local controller failures. However, the oscillatory circulating current behavior is observed due to the control, communication, and sampling delays. Despite the oscillatory behavior and delays in the distributed controller, the MMC sustains the operation without significant interruption. It is worth mentioning that the oscillatory behavior of the circulating current can be reduced with the circulating current suppression controller. In addition, the oscillation effects will be much less for an MMC with a high number of SMs if the controller is extended for such an application.

Lastly, the scope of this paper only covers the fail-over feature of the distributed modular fault-tolerant controller. In other words, only the LC failure test cases are validated with the experimental and real-time simulator results. The controller can also sustain the MMC operation under SM failure, but this feature will be covered in another research paper using the same MMC prototype in the future.