Non-Intrusive Delay-Based Model Partitioning for Distributed Real-Time Simulation

Abstract

The work in this article analyses the impact of time-delays on distributed real-time simulation stability and accuracy with respect to different decoupling points as well as the impact of decoupling point selection on system modes. We perform analysis of the system modes and participation matrix of the system and determine suitable points that negligibly modify the system modes to decouple the original system. From this analysis, a non-intrusive delay-based model partitioning method for distributing real-time simulations that exploits the flexibility in the context of selecting decoupling points is developed.

1. Introduction

With the surge of power electronics and renewable energy sources integration in modern power systems, the need for real-time power system simulations and Power Hardware-in-the-Loop (PHiL) studies is constantly increased [1]. Real-time simulations in modern power systems are considered as an essential tool for the grid and device testing [2,3,4]. However, this constant increase of the device number and the sizes of grids simulated in real-time environment poses a challenge to simulating models in one simulation unit.

The issue of splitting into subsystems and assigning subsystems to different processing resources while ensuring simulation stability and the required level of simulation fidelity in distributed real-time power system simulations is a challenging task and an open research question. Distributed simulations are used when we want or need to have a real-time simulation of a system that cannot be simulated on just one OPAL-RT target or RTDS rack.

This is the case even for simulations running on different CPU cores in one simulator unit since at least a one time step delay between partitions must to be considered. The impact of that one time step delay on the simulation performance is not negligible. In the following, both terms—a partition and a subsystem—are going to be used interchangeably. The contribution of this paper is the method for non-intrusive delay-based model partitioning of distributed real-time simulations developed via analysis of the delay impact on distributed real-time simulation modes at different decoupling points.

2. Literature Review and Related Work on System Partitioning

The most common real-time power system solvers fall into three main categories: nodal analysis-based (as in Hypersim of OPAL-RT and RTDS real-time simulator), state-space formulation solvers (as in Simscape Power System from MATLAB and eMEGAsim from OPAL-RT) and the most recent state-space nodal (SSN) [5,6] formulation that combines the previously mentioned two approaches. The method in this paper performs system analysis using the state-space system formulation.

Research efforts have previously been made to convert nodal power system representation to the state-space formulation if the method is going to be applied in an automated manner on the nodal based power system simulators [7]. Different algorithms and methods have been proposed in the literature for power system simulation partitioning in different contexts. Partitioning methods can be classified based on the used methodology, application and partitioning objective. The most commonly used methodologies are graph-based and eigenvalue-based.

From the application perspective, three main groups are present: methods for parallel computing of the power system solution where the partitioning objective is the reduction of computational complexity when distributing the simulation, methods for the partitioning of multi-rate simulations where the partitioning objectives are the stability and accuracy of the multi-rate simulation and methods for a distributed system simulation where the focus lies on the system simulation stability and accuracy as in methods for multi-rate simulations.

The main difference between distributed system simulations and parallel system simulations is the inclusion of the delays between partitions in the case of distributed system simulations. While the methods dealing with parallel computing partition the system to parallelize the computing of a network solution on one simulation unit, assuming no delays between partitions in one simulation time step, distributed system simulations simulate partitions (representing one monolithic system) on different computational units.

They are, therefore, affected by time delays occurring between computational units. The main focus of this paper is system partitioning for distributed simulations. Therefore, our literature review focuses on the methods already existing for this application, the explanation for why some other currently existing partitioning algorithms cannot be directly used in this application, and papers that have similar methodologies to the one analysed in this paper.

The standard method for power system model partitioning for real-time applications is the travelling-wave method. Here, decoupling is achieved via a travelling-wave transmission line model whose travel time is greater than or equal to the simulation time step used [8,9]. Therefore, natural decoupling of the system is achieved. Assuming the speed of light and simulation time step of $T_s=50\mathsf{\mu }$s, the line length should be at least 15 km in order to naturally decouple the system.

For many transmission networks, this method was sufficient. However, in distribution networks where longer transmission lines do not exist, this method is inapplicable, and decoupling of the system poses greater challenges. Artificially increasing the line length brings in large capacitance and inaccurate results—for example, grid voltages becoming unusually high [10]. The standard method for system partitioning for parallel processing of the system solution is diakoptics, which was developed to reduce the computational burden of the matrix inversion by tearing the power system into sub-problems that are solved independently with the introduction of additional calculations [11,12].

In the original form, as with other parallel processing partitioning methods, diakoptics does not include a delay that could occur between sub-problems solved independently. Therefore, it cannot be applied directly on distributed real-time power system simulations where at least one time step delay should be considered. In [13], partitioning was included by optimizing the CPU load computation and communication between CPUs, and the optimal number of portions was found in the context of parallel computing applications. From the perspective of the methodology, this method is graph-based partitioning as in [14].

Graph-based system partitioning methods are widely used for the parallel computing of system solutions. However, they cannot be directly used for distributed real-time simulations without modifications needed to include the delays occurring between two simulation units. The analysis and method developed in this paper are based on modal analysis. Circuit latency and modal analysis have already been exploited to determine network partitions. However, in multi-rate power system simulation applications, it is applied to finding decoupling points and subsystems suitable for decoupling small and large simulation time steps [7,15].

In [16], a delay-based method for the decoupling of system equations was proposed for offline system simulation. The idea in the paper is similar to the one used in our work since the paper searched for variables that can be delayed one time step and enable the matrix sparsity needed for parallel processing of the system solution, but did not address, in the original form, distributed system simulations (delayed interface quantities between subsystems are the combination of different system states and inputs and contribute to different elements in a Jacobian system matrix).

A detailed explanation of multi-core simulations of power system transients and methods for system partitioning can be found in [17]; however, none of them address the distributed simulation partitioning issue. Coupling point analysis and motivation for methods for decoupling point selection in distributed real-time simulations can be found in [18], where it is shown how the location of the coupling point can affect the fidelity of the distributed simulation.

3. Motivation

3.1. Distributed Real-Time Power System Simulations

A real-time simulation is a distributed real-time power system simulation if we divide a monolithic model of the system. In the literature, this is also called a naturally coupled system (NCS), on subsystems simulated on different simulation units. The simulation decoupling point is where the monolithic system partitioning is made, and delays occur when interface variables are exchanged. This point is known in the literature as the point of connection (POC).

In this paper, the ideal transformer model (ITM) [19] is assumed to be used as an interface algorithm because of its simplicity and accuracy, as represented in Figure 1, and all further analysis is done assuming ITM as the interface algorithm between subsystems. Though a highly accurate interface algorithm, ITM is proven to have stability issues when used in distributed simulation applications [19]. The model partitioning methodology in this paper aims to determine appropriate decoupling points to preserve the simulation stability while using the ITM interface and acting non-intrusive.

Subsystems in the Figure 1 visualizing partitioning problem can represent simulators or real devices based on the application—distributed simulation or PHiL. The main focus of the paper is the first application. Digital real-time power system simulators use a fixed-step solver with a common simulation time step in electromagnetic transient programs (EMTPs) of 50 $\mathsf{\mu }$s. The term time step is used to describe the time between two consecutive instantaneous outputs of the solver.

The time step is related to the sampling frequency of the simulation and determines the frequency bandwidth of signals that can be accurately reproduced. With this time step up to 2–3 kHz can be simulated based on a common rule of thumb that suggests a time step size of a 10-times smaller value of the period of the fastest frequency following disturbance [20]. The delay between two subsystems $t_1=t_2=k·T_s$ in this work is always deterministic and a multiple of the simulation time step.

A variable that is transferred from one subsystem to another is referred to as an interface variable. The interface variable is sampled and delayed before being applied to the receiving subsystem. The sampling period is the time difference between two consecutive sending samples of interface variables from one subsystem to another. In this paper, only delays occurring between two subsystems are considered, and the sampling period is assumed to be equal to the simulation time step of the subsystems. Both subsystems are simulated with the same time step $T_s=50\mathsf{\mu }$s.

3.2. Simulation Examples Explaining Paper Motivation

Let us observe the example system (simple circuit) in Figure 2 with two possible points in which the system can be decoupled: A and B.

Let us assume our circuit has the following parameters $L_1=1$ mH, $C_1=1$ mF, $L_2=10$ mH, $C_2=10$ mF, $L_3=10$ mH, $C_3=10$ mF, $R_1=1\Omega$, $R_2=1\Omega$, $R_3=1\Omega$ and $R_4=1\Omega$. This system was partitioned in two decoupling points making a distributed system simulation. The system was simulated with a defined $d=k·T_s$ fixed delay between two subsystems, where $T_s=50\mathsf{\mu }$s is the simulation time step that is the same for both subsystems. Decoupling the system in decoupling point A leads to stability issues for the distributed system simulation for $k=10$ or delay $0.5$ ms. For decoupling point B, the simulation stays stable up to the last simulated delay of $k=50$, thus, $2.5$ ms. Let us observe the example system two in Figure 3, with two possible points in which the system can be decoupled: A and B.

Let us assume our circuit has the following parameters $L_1=8$ mH, $C_1=0.2$ mF, $L_2=80$ mH, $C_2=2$ mF, $R_1=1\Omega$, $R_2=1\Omega$ and $R_3=1\Omega$ and that distributed versions are made as in the previous example. Decoupling the system in decoupling point A leads to stability issues for the distributed system simulation, even for $k=2$ or delay $0.1$ ms. For decoupling point B, the simulation stays stable up to the last simulated delay of $k=75$ or $18.7$ ms. These motivation examples show that distributed simulations have stability issues in specific decoupling points. In contrast, another decoupling point would not have a stability problem for the predefined delay and would, therefore, be a better decoupling point candidate in terms of the distributed simulation stability.

4. Mathematical Background of Analysis and Methodology

4.1. State-Space Representation of Partitioned System Model

In the case of distributed real-time simulations, both digital simulator units solve the system in the discrete time-domain. It is, thus, a straightforward approach to perform analysis of the distributed system model in the discrete time-domain. To perform modal analysis on the distributed system simulation model, the time-invariant state-space model of the two connected subsystems with the inclusion of delays is derived.

The implicit integration method typically used in real-time power system simulators is the trapezoidal method, which is numerically A-stable. If this method is applied to the set of stable linear system equations, the time step of the simulation influences only the accuracy of the system simulation and not the numerical stability. In the modelling of the two coupled subsystems in this section, trapezoidal integration was applied to discretize the state-space subsystem models. Let us first assume that we have two state-space subsystem models represented in Figure 4.

State-space models of the subsystems one and two in the discrete time-domain can be represented by Equations (1) and (2) respectively:

$$\begin{array}{c}\hfill {\mathit{x}}_{\mathbf{1}}(\mathit{k}+\mathbf{1})=A_1·{\mathit{x}}_{\mathbf{1}}\left(\mathit{k}\right)+B_1·{\mathit{u}}_{\mathbf{1}}\left(\mathit{k}\right)+B_1^{SS2}·{\mathit{u}}_{\mathbf{1}}^{\mathit{S}\mathit{S}2}\left(\mathit{k}\right)\\ \hfill {\mathit{y}}_{\mathbf{1}}\left(\mathit{k}\right)=C_1·{\mathit{x}}_{\mathbf{1}}\left(\mathit{k}\right)+D_1·{\mathit{u}}_{\mathbf{1}}\left(\mathit{k}\right)+D_1^{SS2}·{\mathit{u}}_{\mathbf{1}}^{\mathit{S}\mathit{S}\mathbf{2}}\left(\mathit{k}\right)\\ \hfill {\mathit{y}}_{\mathbf{1}}^{\mathit{ext}}\left(\mathit{k}\right)=C_1^{ext}{\mathit{x}}_{\mathbf{1}}\left(\mathit{k}\right)+D_1^{ext}{\mathit{u}}_{\mathbf{1}}\left(\mathit{k}\right)+D_1^{SS2,ext}·{\mathit{u}}_{\mathbf{1}}^{\mathit{S}\mathit{S}\mathbf{2}}\left(\mathit{k}\right)\end{array}$$

(1)

$$\begin{array}{c}\hfill {\mathit{x}}_{\mathbf{2}}(\mathit{k}+\mathbf{1})=A_2·{\mathit{x}}_{\mathbf{2}}\left(\mathit{k}\right)+B_2·{\mathit{u}}_{\mathbf{2}}\left(\mathit{k}\right)+B_2^{SS1}·{\mathit{u}}_{\mathbf{2}}^{\mathit{S}\mathit{S}\mathbf{1}}\left(\mathit{k}\right)\\ \hfill {\mathit{y}}_{\mathbf{2}}\left(\mathit{k}\right)=C_2·{\mathit{x}}_{\mathbf{2}}\left(\mathit{k}\right)+D_2·{\mathit{u}}_{\mathbf{2}}\left(\mathit{k}\right)+D_2^{SS1}·{\mathit{u}}_{\mathbf{2}}^{\mathit{S}\mathit{S}\mathbf{1}}\left(\mathit{k}\right)\\ \hfill {\mathit{y}}_{\mathbf{2}}^{\mathit{ext}}\left(\mathit{k}\right)=C_2^{ext}{\mathit{x}}_{\mathbf{2}}\left(\mathit{k}\right)+D_2^{ext}{\mathit{u}}_{\mathbf{2}}\left(\mathit{k}\right)+D_2^{SS1,ext}{\mathit{u}}_{\mathbf{2}}^{\mathit{S}\mathit{S}\mathbf{1}}\left(\mathit{k}\right)\end{array}$$

(2)

where k is the current time step, $k+1$ is the future time step, and ${\mathit{u}}_{\mathbf{1}}^{\mathit{S}\mathit{S}\mathbf{2}}\left(\mathit{k}\right)={\mathit{y}}_{\mathbf{2}}\left(\mathit{k}\right)$, ${\mathit{u}}_{\mathbf{2}}^{\mathit{S}\mathit{S}\mathbf{1}}\left(\mathit{k}\right)={\mathit{y}}_{\mathbf{1}}\left(\mathit{k}\right)$. With simple manipulation over the matrices, the combined state-space model of those two subsystem models can be described by Equation (3), where $N={(I-D_1^{SS2}·D_2^{SS1})}^{-1}$ and I is the unity matrix of the required size.

$$\begin{array}{c}\hfill \left[\begin{array}{c}{\mathit{x}}_{\mathbf{1}}(\mathit{k}+\mathbf{1})\\ {\mathit{x}}_{\mathbf{2}}(\mathit{k}+\mathbf{1})\end{array}\right]=\left[\begin{array}{cc}{A}_1+B_1^{SS2}·D_2^{SS1}·N·C_1& B_1^{SS2}·(I+D_2^{SS1}·N·D_1^{SS2})·C_2\\ B_2^{SS1}·N·C_1& A_2+B_2^{SS1}·N·D_1^{SS2}·C_2\end{array}\right]·\left[\begin{array}{c}{\mathit{x}}_{\mathbf{1}}\left(\mathit{k}\right)\\ {\mathit{x}}_{\mathbf{2}}\left(\mathit{k}\right)\end{array}\right]+\\ \hfill \left[\begin{array}{cc}{B}_1+B_1^{SS2}·D_2^{SS1}·N·D_1& B_1^{SS2}·(I+D_2^{SS1}·N·D_1^{SS2})·D_2\\ B_2^{SS1}·N·D_1& B_2+B_2^{SS1}·N·D_1^{SS2}·D_2\end{array}\right]·\left[\begin{array}{c}{\mathit{u}}_{\mathbf{1}}\left(\mathit{k}\right)\\ {\mathit{u}}_{\mathbf{2}}\left(\mathit{k}\right)\end{array}\right]\\ \hfill \left[\begin{array}{c}{\mathit{y}}_{\mathbf{1}}^{\mathit{ext}}\\ {\mathit{y}}_{\mathbf{2}}^{\mathit{ext}}\end{array}\right]=\left[\begin{array}{cc}{C}_1^{ext}+D_1^{SS2,ext}·D_2^{SS1}·N·C_1& D_1^{SS2,ext}·(I+D_2^{SS1}·N·D_1^{SS2})·C_2\\ D_2^{SS1,ext}·N·C_1& C_2^{ext}+D_2^{SS1,ext}·N·D_1^{SS2}·C_2\end{array}\right]·\left[\begin{array}{c}{\mathit{x}}_{\mathbf{1}}\left(\mathit{k}\right)\\ {\mathit{x}}_{\mathbf{2}}\left(\mathit{k}\right)\end{array}\right]+\\ \hfill \left[\begin{array}{cc}{D}_1^{ext}+D_1^{SS2,ext}·D_2^{SS1}·N·D_1& D_1^{SS2,ext}·(I+D_2^{SS1}·N·D_1^{SS2})·D_2\\ D_2^{SS1,ext}·N·D_1& D_2^{ext}+D_1^{SS2,ext}·N·D_1^{SS2}·D_2\end{array}\right]·\left[\begin{array}{c}{\mathit{u}}_{\mathbf{1}}\left(\mathit{k}\right)\\ {\mathit{u}}_{\mathbf{2}}\left(\mathit{k}\right)\end{array}\right]\end{array}$$

(3)

If we assume that delayed outputs of the state-space subsystem models are inputs of the other state-space subsystem model, this means that ${\mathit{u}}_{\mathbf{1}}^{\mathit{S}\mathit{S}\mathbf{2}}\left(\mathit{k}\right)={\mathit{y}}_{\mathbf{2}}(\mathit{k}-{\mathit{k}}_{\mathbf{2}})$, ${\mathit{u}}_{\mathbf{2}}^{\mathit{S}\mathit{S}\mathbf{1}}\left(\mathit{k}\right)={\mathit{y}}_{\mathbf{1}}(\mathit{k}-{\mathit{k}}_{\mathbf{1}})$, and their connection is visualized in Figure 5.

We assume that time delays between two subsystems are deterministic and multiples of the simulation time step of the subsystems $t_1=k_1·T_s$ and $t_2=k_2·T_s$, where $T_s$ is the simulation time step.

If we augment the states and input vectors up to the output delays as in Equation (4) for the subsystem 1 model, a new augmented state-space model of subsystem 1 can be developed, as in Equations (5) and (6). Here, n is the number of states in subsystem 1, p is the number of external inputs in subsystem 1 (size of vector ${\mathit{u}}_{\mathbf{1}}$), s is the number of inputs coming from another subsystem (size of vector ${\mathit{u}}_{\mathbf{1}}^{\mathit{S}\mathit{S}\mathbf{2}}$), m is the number of outputs going to another subsystem (size of vector ${\mathit{y}}_{\mathbf{1}}$), $I_{n\times n}$ is a unity matrix of size $n\times n$, and $0_{n\times n}$ is a zero matrix of size $n\times n$.

$$\begin{array}{c}\hfill {\tilde{\mathit{x}}}_{\mathbf{1}}\left(\mathit{k}\right)=\left[\begin{array}{c}{\mathit{x}}_{\mathbf{1}}\left(\mathit{k}\right)\\ {\mathit{x}}_{\mathbf{1}}(\mathit{k}-\mathbf{1})\\ ⋮\\ {\mathit{x}}_{\mathbf{1}}(\mathit{k}-\mathit{k}\mathbf{1})\end{array}\right],{\tilde{\mathit{u}}}_{\mathbf{1}}\left(\mathit{k}\right)=\left[\begin{array}{c}{\mathit{u}}_{\mathbf{1}}\left(\mathit{k}\right)\\ {\mathit{u}}_{\mathbf{1}}(\mathit{k}-\mathit{k}\mathbf{1})\\ ⋮\\ {\mathit{u}}_{\mathbf{1}}(\mathit{k}-\mathit{k}\mathbf{1})\end{array}\right],{\tilde{\mathit{u}}}_{\mathbf{1}}^{\mathit{S}\mathit{S}\mathbf{2}}\left(\mathit{k}\right)=\left[\begin{array}{c}{\mathit{u}}_{\mathbf{1}}^{\mathit{S}\mathit{S}\mathbf{2}}\left(\mathit{k}\right)\\ {\mathit{u}}_{\mathbf{1}}^{\mathit{S}\mathit{S}\mathbf{2}}(\mathit{k}-\mathbf{1})\\ ⋮\\ {\mathit{u}}_{\mathit{1}}^{\mathit{S}\mathit{S}\mathbf{2}}(\mathit{k}-\mathit{k}\mathbf{1})\end{array}\right]\end{array}$$

(4)

$$\begin{array}{cc}\hfill & {\tilde{\mathit{x}}}_{\mathbf{1}}(\mathit{k}+\mathbf{1})={\tilde{A}}_1·{\tilde{\mathit{x}}}_{\mathbf{1}}\left(\mathit{k}\right)+{\tilde{B}}_1·{\tilde{\mathit{u}}}_{\mathbf{1}}\left(\mathit{k}\right)+{\tilde{B}}_1^{SS2}·{\tilde{\mathit{u}}}_{\mathbf{1}}^{\mathit{S}\mathit{S}\mathbf{2}}\left(\mathit{k}\right)\hfill \\ \hfill {\tilde{\mathit{y}}}_{\mathbf{1}}\left(\mathit{k}\right)& ={\mathit{y}}_{\mathbf{1}}(\mathit{k}-\mathit{k}\mathbf{1})={\mathit{u}}_{\mathbf{2}}^{\mathit{S}\mathit{S}\mathbf{1}}\left(\mathit{k}\right)={\tilde{C}}_1·{\tilde{\mathit{x}}}_{\mathbf{1}}\left(\mathit{k}\right)+{\tilde{D}}_1·{\tilde{\mathit{u}}}_{\mathbf{1}}\left(\mathit{k}\right)+{\tilde{D}}_1^{SS2}·{\tilde{\mathit{u}}}_{\mathbf{1}}^{\mathit{S}\mathit{S}\mathbf{2}}\left(\mathit{k}\right)\hfill \end{array}$$

(5)

$$\begin{array}{c}\hfill {\tilde{A}}_1=\left[\begin{array}{ccccc}{A}_1& 0_{n\times n}& \dots & 0_{n\times n}& 0_{n\times n}\\ I_{n\times n}& 0_{n\times n}& \dots & 0_{n\times n}& 0_{n\times n}\\ 0_{n\times n}& I_{n\times n}& 0_{n\times n}& \dots & ⋮\\ ⋮& 0_{n\times n}& 0_{n\times n}& \dots & 0_{n\times n}\\ 0_{n\times n}& \dots & 0_{n\times n}& I_{n\times n}& 0_{n\times n}\end{array}\right]{\tilde{B}}_1=\left[\begin{array}{cccc}{B}_1& 0_{n\times p}& \dots & 0_{n\times p}\\ 0_{n\times p}& 0_{n\times p}& \dots & 0_{n\times p}\\ ⋮& 0_{n\times p}& \dots & 0_{n\times p}\\ 0_{n\times p}& 0_{n\times p}& \dots & 0_{n\times p}\end{array}\right]\\ \hfill {\tilde{B}}_1^{SS2}=\left[\begin{array}{cccc}{B}_1^{SS2}& 0_{n\times s}& \dots & 0_{n\times s}\\ 0_{n\times s}& 0_{n\times s}& \dots & 0_{n\times s}\\ ⋮& 0_{n\times s}& \dots & 0_{n\times s}\\ 0_{n\times s}& 0_{n\times s}& \dots & 0_{n\times s}\end{array}\right]{\tilde{C}}_1={\left[\begin{array}{c}{0}_{n\times m}\\ ⋮\\ 0_{n\times m}\\ C_1\end{array}\right]}^T{\tilde{D}}_1={\left[\begin{array}{c}{0}_{p\times m}\\ ⋮\\ 0_{p\times m}\\ D_1\end{array}\right]}^T{\tilde{D}}_1^{SS2}={\left[\begin{array}{c}{0}_{s\times m}\\ ⋮\\ 0_{s\times m}\\ D_1^{SS2}\end{array}\right]}^T\end{array}$$

(6)

In the same way, the augmented subsystem model can be derived for subsystem 2. Those two augmented state-space representations of subsystems 1 and 2 can be connected in the same way as the subsystems in Figure 4 using Equation (3).

4.2. Modal Analysis

Modal analysis has been applied to various power system problems, especially in the domain of inter-area oscillations [21] and Wide-Area Measurement System (WAMS) applications [22,23,24,25]. The modal analysis represents small-signal stability analysis applied to linearized system models. This analyses the stability of the power system around its equilibrium point before or after the disturbances and gives information about system oscillations: how many modes the system has and their frequencies and dampings.

A systematic procedure for assigning state variables and finding the dynamical equations of the system can be found in [26] if a dynamic system is a linear time-invariant (LTI) electric network. An electrical grid is the interconnection of different electrical elements. With some assumptions, all dynamic components in power systems (generators, dynamic loads, lines, regulation and control) can be described by a set of differential and algebraic equations:

$$\begin{array}{cc}\hfill & \dot{\mathit{x}}=f(\mathit{x},\mathit{z},\mathit{u})\hfill \\ \hfill & {\mathbf{0}}_{\mathit{m}}=g(\mathit{x},\mathit{z},\mathit{u})\hfill \end{array}$$

(7)

where $\mathit{x}$ is state vector of size n, $\mathit{u}$ represents the system inputs, $\mathit{z}$ contains algebraic variables of the system, f is a set of differential equations, and g is set of algebraic (without derivatives) equations based on Kirchhoff’s law. The state-space model of the system is a linear representation of Equation (7) at some equilibrium point. If $({\mathit{x}}_{\mathbf{0}},{\mathit{z}}_{\mathbf{0}},{\mathit{u}}_{\mathbf{0}}$) is an equilibrium point of the system (given for example by the load flow calculations):

$$\begin{array}{cc}\hfill & \dot{{\mathit{x}}_{\mathbf{0}}}=f({\mathit{x}}_{\mathbf{0}},{\mathit{z}}_{\mathbf{0}},{\mathit{u}}_{\mathbf{0}})={\mathbf{0}}_{\mathit{n}}\hfill \\ \hfill & {\mathbf{0}}_{\mathit{m}}=g({\mathit{x}}_{\mathbf{0}},{\mathit{z}}_{\mathbf{0}},{\mathit{u}}_{\mathbf{0}})\hfill \end{array}$$

(8)

using the first order Taylor approximation around $({\mathit{x}}_{\mathbf{0}},{\mathit{z}}_{\mathbf{0}},{\mathit{u}}_{\mathbf{0}})$ leads to:

$$\dot{\mathit{x}}=f({\mathbf{x}}_{\mathbf{0}},{\mathit{z}}_{\mathbf{0}},{\mathit{u}}_{\mathbf{0}})+\frac{\partial f}{\partial \mathit{x}}({\mathit{x}}_{\mathbf{0}},{\mathit{z}}_{\mathbf{0}},{\mathit{u}}_{\mathbf{0}})(\mathit{x}-{\mathit{x}}_{\mathbf{0}})+\frac{\partial f}{\partial \mathit{z}}({\mathit{x}}_{\mathbf{0}},{\mathit{z}}_{\mathbf{0}},{\mathit{u}}_{\mathbf{0}})(\mathit{z}-{\mathit{z}}_{\mathbf{0}})+\frac{\partial f}{\partial \mathit{u}}({\mathit{x}}_{\mathbf{0}},{\mathit{z}}_{\mathbf{0}},{\mathit{u}}_{\mathbf{0}})(\mathit{u}-{\mathit{u}}_{\mathbf{0}})$$

(9)

$${\mathbf{0}}_{\mathit{m}}=\frac{\partial g}{\partial \mathit{x}}({\mathit{x}}_{\mathbf{0}},{\mathit{z}}_{\mathbf{0}},{\mathit{u}}_{\mathbf{0}})(\mathit{x}-{\mathit{x}}_{\mathbf{0}})+\frac{\partial g}{\partial \mathit{z}}({\mathit{x}}_{\mathbf{0}},{\mathit{z}}_{\mathbf{0}},{\mathit{u}}_{\mathbf{0}})(\mathit{z}-{\mathit{z}}_{\mathbf{0}})+\frac{\partial g}{\partial u}({\mathit{x}}_{\mathbf{0}},{\mathit{z}}_{\mathbf{0}},{\mathit{u}}_{\mathbf{0}})(\mathit{u}-{\mathit{u}}_{\mathbf{0}})$$

(10)

In order to simplify Equations (9) and (10), $\Delta \mathit{x}=\mathit{x}-{\mathit{x}}_{\mathbf{0}}$, $\Delta \mathit{z}=\mathit{z}-{\mathit{z}}_{\mathbf{0}}$, $\Delta \mathit{u}=\mathit{u}-{\mathit{u}}_{\mathbf{0}}$, $f_x=\frac{\partial f}{\partial \mathit{x}}({\mathit{x}}_{\mathbf{0}},{\mathit{z}}_{\mathbf{0}},{\mathit{u}}_{\mathbf{0}})$, $f_z=\frac{\partial f}{\partial \mathit{z}}({\mathit{x}}_{\mathbf{0}},{\mathit{z}}_{\mathbf{0}},{\mathit{u}}_{\mathbf{0}})$, $f_u=\frac{\partial f}{\partial \mathit{u}}({\mathit{x}}_{\mathbf{0}},{\mathit{z}}_{\mathbf{0}},{\mathit{u}}_{\mathbf{0}})$ are introduced. In the same manner, $g_x$, $g_z$ and $g_u$ are partial derivatives of $g(\mathit{x},\mathit{z},\mathit{u})$ in $({\mathit{x}}_{\mathbf{0}},{\mathit{z}}_{\mathbf{0}},{\mathit{u}}_{\mathbf{0}})$. It can be easily derived that:

$$\Delta \dot{\mathit{x}}=A·\Delta \mathit{x}+B·\Delta \mathit{u}$$

(11)

where $A=f_x-f_z{g}_z^{-1}{g}_x$, $B=f_u-f_z{g}_z^{-1}{g}_u$ with the assumption that $g_z$ is not singular. If we assume that observable outputs of the system $\mathit{y}$ are represented via the function $\mathit{y}=h(\mathit{x},\mathit{z},\mathit{u})$, and, using the same notation, it can be derived that:

$$\Delta \mathit{y}=C·\Delta \mathit{x}+D·\Delta \mathit{u}$$

(12)

where $C=h_x-h_z{g}_z^{-1}{g}_x$ and $D=h_u-h_z{g}_z^{-1}{g}_u$. In the state-space representation of the system, matrix A is specific for the system for a given equilibrium point, whereas B, C and D depend on the chosen inputs and outputs of the system. The system is characterized by the state matrix A and, more precisely, the eigenvalues of matrix A.

The poles of the system are the eigenvalues of the matrix A. These poles can be real and complex if matrix A is real, and the real part of all system eigenvalues needs to be negative, to ensure the stability of the system. The real eigenvalue of A is the system mode, and complex eigenvalues appear in conjugate pairs representing one system mode. The eigenvalue of matrix A is a scalar parameter ${\lambda }_i$ that fulfils the equation:

$$A·\mathit{\varphi }={\lambda }_i·\mathit{\varphi }$$

(13)

were $\mathit{\varphi }$ is n by 1 vector, and n is the size of the square matrix A. To find the eigenvalues, we can rewrite Equation (13) in $(A-{\lambda }_i·I)·\mathit{\varphi }=0$ form. For a non-trivial solution, that means:

$$det(A-{\lambda }_i·I)=0$$

(14)

Equation (14) is a characteristic equation with n solutions that are eigenvalues of A. For any eigenvalue ${\lambda }_i$, the n-column vector ${\mathit{\varphi }}_i$ that satisfies

$$A·{\mathit{\varphi }}_{\mathit{i}}={\lambda }_i·{\mathit{\varphi }}_{\mathit{i}}$$

(15)

is the (right) eigenvector of A associated with ${\lambda }_i$. Similarly, n-row vector ${\mathit{\psi }}_i$ that satisfies:

$${\mathit{\psi }}_{\mathit{i}}·A={\lambda }_i·{\mathit{\psi }}_{\mathit{i}}$$

(16)

is the left eigenvector associated with ${\lambda }_i$. In order to express the eigenproperties of A, modal matrices (in the literature also called transformation matrices) are introduced:

$$\begin{array}{cc}\hfill \Psi & ={\left[\begin{array}{c}{\mathit{\psi }}_{\mathbf{1}}^{\mathit{T}}{\mathit{\psi }}_{\mathbf{2}}^{\mathit{T}}\dots {\mathit{\psi }}_{\mathit{n}}^{\mathit{T}}\end{array}\right]}^T\hfill \\ \hfill \Phi & =\left[\begin{array}{c}{\mathit{\varphi }}_{\mathbf{1}}{\mathit{\varphi }}_{\mathbf{2}}\dots {\mathit{\varphi }}_{\mathit{n}}\end{array}\right]\hfill \end{array}$$

(17)

It can be easily derived that matrices $\Psi$ and $\Phi$ are orthogonal [27] and satisfy $A\Phi =\Lambda \Phi$, $\Psi A=\Lambda \Psi$, where $\Lambda =diag\left({\lambda }_i\right)$. If ${\mathit{\varphi }}_i$ is the right eigenvector, then $k·{\mathit{\varphi }}_{\mathit{i}}$ is as well, and the same counts for ${\mathit{\psi }}_{\mathit{i}}$ and $k·{\mathit{\psi }}_{\mathit{i}}$, where k is scalar. Therefore, $\Psi$ and $\Phi$ can be normalized so that they satisfy $\Psi ={\Phi }^{-1}$ and $\Psi A\Phi =\Lambda$.

Using the transformation matrix $\Phi$, we can transform the system in the new base ($\xi$) where the modes are decoupled ($\Delta \mathit{x}=\Phi \xi$) and show that the right eigenvector ${\mathit{\varphi }}_{\mathit{i}}$ represents the relative activity of the state variables when a particular mode is excited and ${\varphi }_{ij}$ represents how state variable $x_i$ will be impacted by the excitation of the mode j where magnitudes of the elements in ${\mathit{\varphi }}_i$ represent a degree of activities of n state variables in the ith mode, and the angles of elements show the phase shifting of the state variables with regard to the mode.

Thus, a larger ${\varphi }_{ij}$ means that state variable $x_i$ is more impacted by mode j. The overall variation of the mode $\Delta {\mathit{x}}_{\mathit{i}}$ is the sum of the impacts of all system modes. The left eigenvector ${\psi }_i$ gives the combination of state variables displayed in the ith mode. Similarly to the case of the right eigenvectors, if ${\psi }_{ij}$ is the jth element of the ${\mathit{\psi }}_{\mathit{i}}^{\mathit{T}}$, the larger the ${\psi }_{ij}$ value is, the larger is the influence of the state $x_j$ on mode i.

Analysing the connection of system modes and system states looking individually at right and left eigenvectors can cause a problem due to scaling and the units of state variables (the elements of the right and left eigenvectors depend on the units and scaling of the state variables). To overcome this issue, a participation matrix [27,28] is introduced:

$$P=[{\mathit{p}}_{\mathbf{1}}{\mathit{p}}_{\mathbf{2}}{\mathit{p}}_{\mathbf{3}}\dots {\mathit{p}}_{\mathit{n}}]$$

(18)

with

$${\mathit{p}}_{\mathit{i}}=\left[\begin{array}{c}{p}_{1i}\\ p_{2i}\\ ⋮\\ p_{ni}\end{array}\right]=\left[\begin{array}{c}{\mathit{\varphi }}_{1i}·{\mathit{\psi }}_{i1}\\ {\mathit{\varphi }}_{2i}·{\mathit{\psi }}_{i2}\\ ⋮\\ {\mathit{\varphi }}_{ni}·{\mathit{\psi }}_{in}\end{array}\right]$$

(19)

where ${\varphi }_{ki}$ is the kth element of the right eigenvector ${\mathit{\varphi }}_{\mathit{i}}$ and ${\psi }_{ik}$ is the kth element of the left eigenvector ${\mathit{\psi }}_{\mathit{i}}$. The element $p_{ki}={\varphi }_{ki}·{\psi }_{ik}$ is called the participation factor. The influence of the specific eigenvalues (poles or modes) on the states of the system can be measured using a participation matrix.

When $\Psi$ and $\Phi$ are normalized and orthogonal, we have $p_{ij}\le 1$ and ${\sum }_{j=1}^n{p}_{ij}={\sum }_{i=1}^n{p}_{ij}=1$. For this paper, a relative participation matrix is used as introduced in [29], normalizing the state influence on the modes. The relative participation of the kth state variable on the ith mode is $p_{re{l}_{ki}}$ as in Equation (20), representing this influence as a percentage.

$$p_{re{l}_{ki}}=\frac{|{\varphi }_{ik}|·|{\psi }_{ki}|}{{\sum }_{k=1}^{k=n}|{\varphi }_{ik}|·|{\psi }_{ki}|}·100\%$$

(20)

We assume that the ith mode has influence on the kth state if $p_{re{l}_{ki}}>5\%$ as in [30].

5. Analysis and Non-Intrusive Delay-Based System Partitioning

5.1. Analysis

As we saw in Section 3.2, certain decoupling points are more challenging than others. Our methodology focuses on finding how challenging a specific decoupling point is in terms of simulation stability and, if possible, suggesting other decoupling points that would lead to stability of the simulation for the specified delay with preserved fidelity. This was done with the application of modal analysis explained in Section 4.2 and verified on the examples from Section 3.2 modelled using distributed the state-space modelling approach from Section 4.1.

Considering only the simulation of stable monolithic systems, the real parts of all eigenvalues should be negative (${\sigma }_i<0$). Real eigenvalue ${\lambda }_i={\sigma }_i$ corresponds to the non-oscillatory mode. Each complex conjugate pair of eigenvalues is associated with one complex conjugate mode. For complex pairs of eigenvalues ${\lambda }_i={\sigma }_i\pm i·{\omega }_i$, the natural frequency of oscillation is determined by ${\omega }_{n,i}$, and the damping ratio is ${\zeta }_i$, where ${\omega }_i={\omega }_{n,i}·\sqrt{1-{\zeta }_i^2}$, and ${\sigma }_i=-{\zeta }_i·{\omega }_{n,i}$.

The damping ratio determines the rate of transient response decay of the amplitude of oscillation of complex conjugate mode, and the frequency of oscillations is determined by ${\omega }_{n,i}$ as $\frac{{\omega }_{n,i}}{2\pi }$. Following the rule of thumb, the simulation time step needed to capture oscillations of the complex conjugate mode is 10-times smaller than the period of oscillation of the mode [20], which results in $\frac{\pi }{5{\omega }_{n,i}}$, and this value we define as the critical time constant of the complex conjugated mode $T_{i_{cr}}$.

The dynamic properties of the system modes determined by damping ${\zeta }_i$ and the frequency of the oscillations determine the dynamic properties of the system, transient response of the system and, therefore, the overall system dynamics. System modes that influence the states in both subsystems, assuming we have two subsystems, are interaction modes. Local modes are all the modes that are not interaction modes for a specified decoupling point. Modes of the system were defined in the same manner but for another application in [30].

Modes that are interaction modes are the modes impacted by partitioning the system, while local modes are not impacted by partitioning, and therefore delays occur between partitions. Modes of different distributed simulation models were observed in the z-domain to determine which modes moved in the z-domain, thereby, changing the system response and leading to simulation instability.

We observed that local modes of the distributed simulations do not change the value and move in the z-domain significantly, even with significantly increased delays between subsystems. Interaction modes move more, even with only one time step delay if the dynamic of the mode is comparable with the delay and, therefore, change the distributed simulation system response as will be shown in Section 5.2.1.

5.2. Methodology

Avoiding decoupling points in which critical modes are interaction modes can allow greater delays while preserving the stability of the real-time simulation, and since the modes are not significantly moving, thus, preserving simulation fidelity. The method developed in this paper focuses on selecting model decoupling points in which the critical modes of the monolithic model are not significantly modified and proposes possible time delays to preserve the simulation stability.

We define critical modes of the system as complex conjugate modes of the system that have comparable dynamic properties with the time delays we expect between two subsystems and, therefore, are sensitive to delays between subsystems in the case that they are interaction modes. Critical modes of the system are all the complex conjugate ${\lambda }_i$ modes for which Equation (21) is fulfilled, where $d=k·T_s$ is the expected time delay between two subsystems.

$$roun{d}_{T_s}(T_{i_{cr}}·{\zeta }_i)\le d=k·T_s$$

(21)

Value $T_{i_{cr}}·{\zeta }_i$ should be rounded down to the first discrete $m·T_s$, where m is an integer. The stronger the coupling between two subsystems from the perspective of critical mode is, the larger is the influence of decoupling on critical system mode and, therefore, the distributed system simulation. If ${\rm Y}$ is a set of the subsystem states in subsystem w, the relative influence of system mode i on this subsystem can be defined as $P_{re{l}_{S{S}_w}}={\sum }_{k\in {\rm Y}}{p}_{re{l}_{ki}}$. Here it is important to define the coupling index between two subsystems m and s with respect to the system mode i as in Equation (22) where m is the subsystem with the smaller relative influence of the mode i, and s opposite.

$$C{O}_i=\frac{P_{re{l}_{S{S}_m}}}{P_{re{l}_{S{S}_s}}}$$

(22)

The maximal coupling index of two subsystems for the system mode i is $C{O}_i=1$, and if the system mode is the local mode, $C{O}_i=0$. The partitioning methodology of this paper is assumed to be performed on a monolithic system model in the continuous time domain. Since eigenvectors of the system are the same in the continuous and discrete time domain, the participation matrix is also [31].

Therefore, performing participation matrix analysis on the monolithic system model in the continuous time domain is possible without losing correctness. Eigenvalues of the continuous and discrete time domain are connected by bilinear transformation as in Equation (23), where $z_i$ is a discrete eigenvalue, and differential equations are discretized with the trapezoidal rule. In the case simulation, time step $T_s$ is small enough compared to the time constants of the system, discretization does not influence the response of the modes, and continuous eigenvalues can be considered without losing correctness.

$${\lambda }_i=\frac{2}{T_s}·\frac{z_i-1}{z_i+1}$$

(23)

The delay margin for preserving distributed simulation stability cannot be found as a precise value observing only critical mode i properties because $T_{i_{cr}}$ is determined relying on the simulation rule of thumb. Though, empirically on different distributed system simulations, a delay margin for the stability of the distributed system is assured from the perspective of delays occurring between subsystems is found and looks as in Equation (24).

$$d_{stabilit{y}_{assure{d}_i}}=T_{i_{cr}}·{\zeta }_i·\frac{1}{C{O}_i}$$

(24)

Here, i is a critical system mode, $T_{i_{cr}}$ is critical time constant of that mode, ${\zeta }_i$ is the damping of that mode, and $C{O}_i$ is a coupling index of the critical system mode i for the decoupling point. As expected, with a decreasing coupling index between two subsystems, $d_{stabilit{y}_{assure{d}_i}}$ increases. In the case of several critical coupling modes, the smallest of all $d_{stabilit{y}_{assure{d}_i}}$ should be taken.

If the simulation is stable for greater delays, it cannot be answered using this methodology. However, it eliminates the possibility that the simulation is unstable because of the critical mode that disturbances can excite. If the delay d that we expect between partitions is larger than $d_{stabilit{y}_{assure{d}_i}}$, we suggest decreasing the delay until Equation (25) is fulfilled or changing the decoupling point in order to preserve the simulation stability.

$$d<d_{stabilit{y}_{assure{d}_i}}$$

(25)

The non-intrusive delay-based model partitioning method is explained step-by-step and applied on examples from Section 3.2, and the methodology diagram is shown in Figure 6.

The method developed in this paper can be applied in PHiL applications determining how suitable and challenging the point is in which we want to connect the device from the real-time simulation perspective if a suitable model of the device can be found.

5.2.1. Method and Analysis Exemplification

Let us first analyse example system one in Figure 2. Following the methodology, one should find a monolithic system model, find the modes of the system, and check if there are some critical modes with the predefined delay of $k=10$ using Equation (21). As the first step, the monolithic system model is derived with $u=V_1$ as the system input and all system states as outputs. Matrix A of the system is:

$$A=\left[\begin{array}{cccccc}-\frac{R_1}{L_1}& -\frac{1}{L_1}& 0& 0& 0& 0\\ \frac{1}{C_1}& 0& -\frac{1}{C_1}& 0& 0& 0\\ 0& \frac{1}{L_2}& -\frac{R_2}{L_2}& -\frac{1}{L_2}& 0& 0\\ 0& 0& \frac{1}{C_2}& 0& -\frac{1}{C_2}& 0\\ 0& 0& 0& \frac{1}{L_3}& -\frac{R_3}{L_3}& -\frac{1}{L_3}\\ 0& 0& 0& 0& \frac{1}{C_3}& -\frac{1}{C_3·R_4}\end{array}\right]$$

with system states as in Figure 2, ${\left[\begin{array}{c}{x}_1{x}_2{x}_3{x}_4{x}_5{x}_6\end{array}\right]}^T={\left[\begin{array}{c}{I}_{L1}{V}_{C1}{I}_{L2}{V}_{C2}{I}_{L3}{V}_{C3}\end{array}\right]}^T$. From this matrix and the parameters given in Section 3.2, we find that system has four modes with the characteristics presented in

Table 1. Example system 1 modes.

Mode i	Value	${\mathit{\zeta }}_{\mathit{i}}$	${\mathit{\omega }}_{\mathit{n},\mathit{i}}$	${{T}}_{{\mathit{i}}_{\mathit{cr}}\left[\mathbf{ms}\right]}$
$mod{e}_1$	−451.63 ± 904.73 · i	0.44663	1011.2	0.62137
$mod{e}_2$	−77.33 ± 146.50 · i	0.46683	165.65	3.7931
$mod{e}_3$	−140.91	1	140.91	-
$mod{e}_4$	−101.18	1	101.18	-

.

If the expected delay between subsystems can go up to $k=10$, for $d=k·T_s$, where $T_s=50\mathsf{\mu }$s, using Equation (21), $mod{e}_1$ can be classified as a critical mode of the system. The next step in our distributed system analysis is finding the relative participation matrix of the system using Equation (20):

$$P_{rel}=\left[\begin{array}{cccc}44.625& 0.967& 7.094& 3.869\\ 49.947& 0.844& 5.236& 3.126\\ 5.374& 8.107& 54.803& 31.975\\ 0.053& 30.530& 17.702& 30.517\\ 0.001& 40.926& 2.174& 0.004\\ 0& 18.626& 12.991& 30.508\end{array}\right]\%$$

From the relative participation matrix, it follows that $mod{e}_1$ influences state $x_1$ of the system with $44.625\%$, state $x_2$ with $49.947\%$ and state $x_3$ with $5.374\%$ and has a negligible impact on states $x_4$, $x_5$ and $x_6$. In the case of decoupling point A, this mode is interaction mode. For decoupling point A, the influence of critical $mod{e}_1$ on the states is divided between the subsystems on $44.625\%$ and $55.375\%$. Therefore, for decoupling point A, the coupling index between subsystem with respect to critical $mod{e}_1$ is $C{O}_1=\frac{44.625\%}{55.375\%}=0.806$ In the case of decoupling point A, critical mode is interaction mode, and one can say that the stability of distributed system simulation can be assured for delays smaller than $d_{stabilit{y}_{assure{d}_1}}=0.62137·0.44663·\frac{1}{0.806}=0.3443$ ms. Since this delay $d_{stabilit{y}_{assure{d}_1}}=0.3443$ ms is smaller than the $d=k·T_s=10·50\mathsf{\mu }$s = 0.5 ms, we suggest to decrease the delay in order to preserve the stability of the simulation or change the decoupling point, and use decoupling point B for example.

In the case of decoupling point B, critical $mod{e}_1$ is not the interaction mode. Therefore, following the methodology, it is expected that critical mode will not be sensitive to delays occurring between subsystems, since it is localized in one of the subsystems, and a decoupled system should be stable with respect to the delay. Distributed simulations of this system were done up to the delay of $d=k·T_s$, $k=50$, and $T_s=50\mathsf{\mu }$s, and decoupling points A and B. A distributed system (partitioned system) is simulated with $V_1=5$ V and transience is made in $t_s=1$ s with input change to $V_1=4$ V.

Distributed simulations showed that, in the case of decoupling point A, the system becomes unstable for $k=10$, while, for decoupling point B, the system remained stable up to the last simulated delay of $k=50$. In order to analyse the delay impact on the distributed system modes, the distributed system model was derived as in Section 4.1, and in

Table 2. Example system 1, mode 1, for decoupling point A.

	Mode Value	${\mathit{\zeta }}_{{i}}$	${\mathit{\omega }}_{\mathit{n},1}$
Monolithic model-continuous	−451.63 + 904.73 ·i	0.447	1.01 $\times {10}^3$
Monolithic model-discrete	−451.27 + 904.51 ·i	0.446	1.01 $\times {10}^3$
Decoupled model	−426.01 + 919.75 ·i	0.420	1.01 $\times {10}^3$
$\mathit{d}=\mathbf{50}\mathsf{\mu }{s}$	−373.91 + 941.87 ·i	0.369	1.01 $\times {10}^3$
$\mathit{d}=\mathbf{100}\mathsf{\mu }{s}$	−318.14 + 957.58 ·i	0.315	1.01 $\times {10}^3$
$\mathit{d}=\mathbf{150}\mathsf{\mu }{s}$	−262.99 + 963.08 ·i	0.263	9.98 $\times {10}^2$
$\mathit{d}=\mathbf{250}\mathsf{\mu }{s}$	−162.27 + 952.21 ·i	0.168	9.66 $\times 0^2$
$\mathit{d}=\mathbf{500}\mathsf{\mu }{s}$	184.81 + 859.60 · i	−0.21	8.79 $\times {10}^2$

, we can observe the value of critical system mode $mod{e}_1$ with respect to delays between partitions.

Since the model of the distributed system is made in the discrete time domain mode, values in the continuous time domain in the table are made using Equation (23). In the further analysis, we use the term ’decoupled system’, for a system that is made, as in Section 4.1, without delays occurring between subsystems (Figure 4), but with the separate discretization of the two subsystems (partitions). This model is made to see how the separate discretization of the subsystems influenced the system modes before applying delays between subsystems.

The critical system mode that is the interaction mode in the case of decoupling point A, is highly impacted via decoupling in this point and leads to the distributed system instability. It is interesting to observe that the damping ${\zeta }_i$ and ${\omega }_{n,1}$ of the coupling mode $mod{e}_1$ decrease with the increased delay. For decoupling point B, critical mode $mod{e}_1$ is localized with coupling index $C{O}_1=0$. In the case of decoupling point B, $mod{e}_1=-451.63+904.73·i$ for the monolithic model in the continuous and discrete domain, decoupled system model and distributed system model for delays up to $k=50$.

This example shows that the critical system mode leads to instability of the distributed system simulation for the decoupling point, in which this mode was the interaction mode. In contrast, the same critical mode does not change its value with the increased delay between subsystems when it is the local mode for some other decoupling point. Therefore, in avoiding the critical modes as interaction modes, one can preserve the distributed system stability, and indirectly, since the critical mode is not moving and changing the distributed system response the simulation fidelity as well. The system is simulated for both decoupling points A and B and delay $k=5$ to show simulation fidelity. In Figure 7 is presented $x_1$ response for both decoupling points after transient at $t_s=1$ s, showing that not only the stability of the simulation is preserved in the case of decoupling point B. This point shows greater fidelity of the simulation as well.

The fidelities of the output variables are calculated using Equation (26) for each variable i and every simulation point k.

$$FIDELIT{Y}_i=\frac{y_{simulation(i,k)}-y_{monolithic(i,k)}}{y_{monolithic(i,k)}}·100\%$$

(26)

Since the critical interaction mode is the one changing its value with decoupling in decoupling point A, the fidelity of the partitioned system simulation in decoupling point B is better even for $d=1·T_s$ as can be seen in Figure 8. From Figure 8, it can be concluded that, for all simulation points, decoupling point B shows better simulation fidelity compared to decoupling point A, and the monolithic system response is preserved. In the case that we have highly underdamped or an extremely fast critical system mode acting as the coupling mode, this can lead to instability of the simulation even after relatively small delays of $k=1$ and $k=2$, and this is observed on example system two from Figure 3.

Assuming a delay between subsystems up to $k=1$, $d=k·T_s$, using Equation (21) critical system mode $mod{e}_1=57.465\pm 826.83·i$, with $T_{1_{cr}}=0.758$ ms, and ${\zeta }_1=0.06933$ can be found. In the case of decoupling point A, the critical system mode is coupling system mode with $C{O}_1=\frac{45.38\%}{54.62\%}=0.8308$. Using Equation (24), one can say that the stability of distributed system simulation in decoupling point A is assured for delays smaller than $d_{stabilit{y}_{assure{d}_1}}=0.758·0.06933·\frac{1}{0.8308}=0.0633$ ms; thus, $k=1$.

The distributed system was simulated with $V_1=5$ V, and transient is made in $t_s=1$ s with input change to $V_1=4$ V. Indeed, in the case of decoupling point A, the distributed system simulation was stable for one time step delay; therefore, $k=1$, and for $k=2$, the distributed system simulation was unstable. In the case of decoupling point B, critical mode $mod{e}_1$ is localized with $C{O}_1=0$, and the distributed system simulation in decoupling point B showed extreme resiliency towards the delays (the last simulated delay was $d=75·T_s$).

6. Testing of the Method on the Realistic Power System Model

The previously analysed non-intrusive delay-based model partitioning method is verified in power system applications on the part of the CIGRE low voltage European grid benchmark [32], presented in Figure 9. This grid is a representative distribution system benchmark used as the benchmark system for Hardware-in-the-Loop testing of distributed energy resources [1].

The lines of the benchmark are shorter than 15 km; therefore, natural decoupling into subsystems is not possible for the simulation time step of $T_s=50\mathsf{\mu }$s. Part of the grid until node R11 is taken for this use-case, and the inverter is placed in node R11 representing a renewable energy source or battery system. The grid is modelled as a single phase balanced network with L-R lines using parameters from the report [32].

Loads are modelled as constant impedance loads represented as L-R elements, while the inverter is modelled with its L-C filter. Therefore, this work focuses on the physical properties and modes introduced with filter dynamics. In

Table 3. Inverter parameters.

Parameter	Value	Parameter	Value
$f_s$	40 kHz	$S_{rated}$	8.5 kVA
$L_{inv}$	0.535 mH	$V_{LL}$	400 V
$V_{DC}$	$700V$	$C_{filter}$	9.3846 $\mathsf{\mu }$F

are the parameters of the inverter where $S_{rated}$ is the rated power of the inverter, $f_s$ is the switching frequency of the inverter, $C_{filter}$ is the inverter filter capacitance, and $L_{inv}$ is the inverter inductance.

The parameters of the inverter are calculated using Equations (27), where $\Delta I_L$ is $10\%$ current ripple, $P_{rated}$ is the rated active power for a power factor of $0.9$, and adopting a $5\%$ capacitor is assumed.

$$L_{inv}=\frac{V_{DC}}{24·\Delta I_L}$$

(27)

$$C_{filter}=0.05·\frac{P_{rated}}{2·f·\pi ·V_{LL}^2}$$

(28)

In the literature, system models are usually developed focusing on individual system components, such as inverters, lines and transformers [33,34] or merged micro-grid models [35], which are inapplicable for the modelling of a distribution system. In this paper, the overall distribution system model in the state-space domain is made using the approach from [36]. Following the partitioning methodology, one should find if the system has critical system modes using Equation (21). If we determine that the expected delay between partitions can be up to delay $d=1·T_s$, this system has one critical mode—that is $mod{e}_1=-14,650.946\pm 21,439.5972·i$ with ${\zeta }_1=0.5642$, and $T_{1_{cr}}=0.0242$ ms.

Observing the relative participation matrix, the critical mode is purely influencing the inverter filter dynamics and line R1–R11 connecting that inverter. In the case of decoupling point R11, that mode is the coupling mode with a coupling factor between subsystems $C{O}_1=0.53$ Using Equation (24), the stability of the simulation with respect to the delay is assured for $d<d_{stabilit{y}_{assure{d}_1}}=0.03$ ms. Therefore, in the case of $d=1·T_s$, it is suggested to change the decoupling point, since the delay cannot be decreased.

The partitioning methodology would suggest that one should choose a decoupling point for which the inverter and line connecting the inverter are in one subsystem, localizing the critical system mode. The power system model described here is simulated in decoupling point R11 and for delays $d=k·T_s$ Indeed, in the case of decoupling point R11, the distributed system simulation never reaches a stable operating point after a disturbance for one time step delay.

In contrast, choosing decoupling point R1 leads to a stable and accurate distributed system simulation even for the larger delays, and thus it is shown that the partitioning methodology developed in this paper suggests a better decoupling point. A comparison of time responses of the inductance $L_{inv}$ current for decoupling point R1 and a monolithic system simulation for $d=1·T_s$ and in one of the sinusoidal peaks can be seen in Figure 10.

The behaviour that inverter modes are not only influencing the inverter but also the line connecting that inverter was noticed in applying modal analysis and looking at the participation matrix on different power system models. This can be a helpful finding for researchers who want to extend or use this analysis for PHiL and multi-rate simulations applications.

7. Conclusions

Partitioning a system for distributed real-time simulation applications poses challenges in terms of the stability and accuracy of the simulation, and these issues were addressed in this paper. In this paper, we developed and analysed analytical models of distributed real-time simulations. We found that interaction system modes were more impacted by the partitioning of the systems and delays occurring between partitions. With localizing critical system modes, one can preserve the distributed system simulation stability.

The paper showed that the stability of the distributed simulations can be analysed by observing the critical modes of the system, and stable partitioning with regard to delays can be made using the non-intrusive delay-based method developed in the paper. The method acts non-intrusively on the monolithic system model and preserves the simulation fidelity. Verification results of the method on the representative grid are useful for developing automatized partitioning solutions for benchmark distribution system models.