Data-Centric Communication Framework for Multicast IEC 61850 Routable GOOSE Messages over the WAN in Modern Power Systems

: In this paper, a data-centric communication framework is proposed for multicast routable generic object-oriented substation event (GOOSE) messages (MRGM) over the wide area network (WAN) for e ﬀ ective substation-to-substation (SS2SS) and substation to control center (SS2CC) communications. In this structure, the IEC 61850 GOOSE message is transmitted over the WAN using the data distribution service (DDS) as a fast, reliable, and secure data-centric communication middleware. The main feature of this framework is its multicast capability, where several authorized subscribers can receive a published message simultaneously. This can signiﬁcantly improve the system monitoring and control of the protection systems in modern smart grids, where intelligent schemes can be applied. The e ﬀ ectiveness of the proposed platform, in terms of total end-to-end delay between participants, is evaluated through experimental results obtained from the actual hardware-based test setup developed at the Florida International University (FIU) smart grid testbed. The results demonstrate that the latency between sending and receiving a GOOSE message among participants is within its maximum time span deﬁned by the IEC 61850-90-5 working group for communications over the WAN.


•
Introducing a DDS-based communication framework for multicast R-GOOSE messages; • Describing unique features of this framework in terms of scalability, reliability, and multicasting; • Addressing the network traffic issues and minimize the end-to-end time delay; • Verifying the proposed approach through an experimental setup by measuring the real-time end-to-end delay of R-GOOSE message communication over the network layer.
In other words, we are going to address this challenge for IEC 61850 routable communications by introducing a data-centric communication framework for R-GOOSE messages using the data distribution service standard. The proposed solution is a fast and reliable framework that covers all the IEC 61850 routable communication requirements along with the feature of multicasting, which could enable us to develop advanced protection schemes for modern power systems. Furthermore, we develop an experimental setup to measure the real-time end-to-end delay of R-GOOSE message communication over the network layer to verify the effectiveness of the proposed framework.
The rest of this paper is as follow: In Section 2, the message centric and data-centric communication approaches are described and compared. Section 3 is devoted to present the data distribution service as a data-centric communication standard. In Section 4, the proposed multicast routable GOOSE messages (MRGM) communication framework is presented in detail. The proposed test framework along with hardware setup and experimental results are reported in Section 5, and finally, the paper is concluded in Section 6.

Message-Centric vs. Data-Centric Communication Approaches
The message-centric communication approach is used by traditional communication middleware, where a set of messages and data formats are predefined in participants' nodes. In this approach, the communication middleware only delivers the message through the network layer and does not know the message contents and data types. Messages are being appraised within each node of the system (at the application layer) to check the correctness, integrity, data type, and filter data. It means that each node should locally track the state of data, which significantly increases the processing tasks on the application layer. Furthermore, the system expansion is very complicated because major changes are required on the application layer due to any change in message formats or data types [17,18].
Nowadays, modern communication middleware uses the data-centric communication approach. In this data approach, the communication middleware builds the message and updates the system state, therefore, it is aware of the message content and data model [19]. Moreover, instead of processing the messages locally in application, as is done in a message-centric approach; in a data-centric approach the messages are being processed within the middleware layer to assess the correctness of data types delivered to all nodes. It simplifies the application development and system expansion, enhances the system reliability, optimizes the use of network bandwidth, and gives us more opportunities to assign different Quality of Service (QoS) profiles, priorities, and security measures to the data types as comparing with a message-centric approach [18]. In this paper, the DDS is used as a data-centric communication middleware to address the challenges of routing GOOSE message over the WAN. In the following section, DDS is described, and its unique features are mentioned in detail.

Data Distribution Service
The data distribution service (DDS) is a data-centric communication standard which was initially released by the Object Management Group (OMG) in 2004 [20]. The DDS deploys a data-centric publish-subscribe (DCPS) protocol within a virtual global data space to facilitate the communication among participants [21][22][23]. The DCPS protocol is based on defining topics with specific data types and formats through IDL (interface definition language) files over a DDS domain. Afterwards, participants with data writer can publish into these topics and subscribers can use data reader to receive data. It is worth mentioning that several subscribers are able to receive data simultaneously once data is written or updated on a topic, therefore, it is a multicast communication framework which facilitates the data exchange among participants with the same domain ID [15]. This communication platform is fast, expandable, and reliable enough and can be utilized in critical real-time applications. The automatic discovery feature of the DDS makes it possible for dynamic participants to be added/deleted to/from the system without any interruption in the system operation [24][25][26]. The DDS is also equipped with an extraordinary governance and management tool, called Quality of Service (QoS), for flexible communication and controlling the system behavior in terms of data latency budget, priority order, lifespan, durability, etc.
The single point of failure feature in most of the message centric communication middleware directly affect the reliability of system. However, the DDS does not need a message broker for peer-to-peer communication, and therefore it is an excellent and highly reliable tool for distributed applications. In addition to high reliability of the DDS, it is easily scalable due to the real-time publish-subscribe (RTPS) protocol, automatic discovery (dynamic participation of network nodes), and routing service for distributed applications with participants in different networks and diverse transport protocols (e.g., IPv4, IPv6, or shared memory) [27][28][29]. In fact, tracking millions of data values and self-forming of autonomous systems are possible because of scalability and automatic discovery features, respectively. The DDS is optimized to provide a reliable, scalable, and secure integration bus to connect existing and new assets. In dynamic networks, the applicants are plug-and-play if DDS is deployed, and therefore there is no need to deploy configuration because of automatic discovery ability of this middleware. Although DDS could be expanded significantly, its performance is still highly efficient. For example, increasing the number of subscribers from one to 888 only has an impact of 10% on throughput. In addition, the DDS can be implemented for ultra-large-scale systems (ULS) using DDS-wire protocol which is known as DDSI for scalability on ULS deplanements over WAN [28]. The DDS routing service is also achievable through the QoS profile, where the IP address of participants along with proper network protocol (e.g., TCP or UDP) is defined. It results in routing the published data from the first network (where the publisher is located) to the predefined domain ID and topic name at the subscriber's network.
The security of the DDS is another unique feature of this standard. It allows participant to be authenticated before participating in the communication procedure to avoid system spoofing by unauthorized agents, provides an encryption/decryption mechanism through a key management mechanism for secure data exchange among participants to avid data spoofing and bad data injection, and defines a redundant security measure called permission access control, which determines the accessibility for each participant to each domain, topic, and data along with the right to write or read data from that topic. This permission accesses are evaluated through a permission certificate authority, which is initially responsible for signing the certificate for each participant. Figure 1 shows the proposed DDS-based communication framework for multicast R-GOOSE message (MRGM) over the WAN. It also contains two algorisms for GOOSE message transition from the first substation (Substation A) to the second one (Substation B) which are assumed to be in two different local area networks (LAN). As can be seen in this figure, the DDS is selected as a data-centric communication middleware for GOOSE data exchange between two participants in two substations. The IEC 61850 defines a real-time publish-subscribe (RTPS) protocol for GOOSE data exchange within the wall of a substation. In this protocol, the GOOSE message is published by the GOOSE publisher to the local network with specific format including sequence number (SqNum), status number (StNUm), GOOSE dataField, and an APPID (It is the ID of the published message which is checked by subscribers). Once the message is published, subscribers with the same APPID instantly subscribe to the message and praise the message to take an appropriate action. The rate of sending GOOSE messages varies due to the system condition. In normal operating mode, where there is no need for protection relays' actions, GOOSE messages are sent periodically to test the communication system. It usually happens every one second. where the IP address of participants along with proper network protocol (e.g., TCP or UDP) is defined. It results in routing the published data from the first network (where the publisher is located) to the predefined domain ID and topic name at the subscriber's network. The security of the DDS is another unique feature of this standard. It allows participant to be authenticated before participating in the communication procedure to avoid system spoofing by unauthorized agents, provides an encryption/decryption mechanism through a key management mechanism for secure data exchange among participants to avid data spoofing and bad data injection, and defines a redundant security measure called permission access control, which determines the Proposed data-centric communication framework for multicast routable generic object-oriented substation event messages (MRGM).

Proposed MRGM Framework
The high-frequency cascading messages are sent due to protection action requirement because of fault detection in the system or control commands. In this situation, as shown in Figure 2, a train of GOOSE messages are sent to the network starting with a very high frequency and continue to lower frequency rates to make sure that the messages are received by appropriate subscribers to cover the possibility of missing samples. The standard has defined a 4 ms time span for GOOSE message delivery within the data link (Layer 2 of the OSI model). However, the published GOOSE messages could be needed to be subscribed by other IEDs in other substations or by the control/monitoring center. For this purpose, GOOSE messages should be transferred from the data layer (Layer 2 of OSI model) to the network layer (Layer 3 of OSI model) and routed to their destinations in the system over the internet network. The standard has defined a 10 ms time span for routable GOOSE (R-GOOSE) messages.  As mentioned before, specific network settings are required to address this fast delivery over the WAN, therefore, in this paper we propose to deploy the DDS standard for R-GOOSE messages as it covers all the required communication needs. Furthermore, it provides the multicast feature of R-GOOSE messages at the network layer, the same as GOOSE messages at the data layer. This means that several subscribers could receive a published GOOSE message simultaneously and take proper actions immediately. This feature is very helpful in advanced protection systems. In [30], authors have presented an advanced protection scheme for a modern power system, which are experiencing a high level of fault current beyond the breaking capacity of circuit breakers. The proposed advanced protection scheme is mainly based on R-GOOSE messages over the WAN and assumes that a multicast R-GOOSE message framework could deliver the messages to its destination in less than 10 ms. This paper is addressing this framework and demonstrating its capabilities through experimental results. The proposed MRGM algorithm includes three main steps starting with GOOSE publisher in the first network, routing the message over the WAN, and subscribing it within the second network.

Conversion: GOOSE Data Model to DDS Data Object
As shown in the right side of Figure 2 (Substation A), the R-GOOSE communication is begun similar to the conventional GOOSE publishing in data layer. Since GOOSE data model has been defined in the IED platform, to change and add or modify the feature of message we need to develop agents with access to the same network, where GOOSE message is published. This agent is equipped with a GOOSE subscriber with the same APPID of the GOOSE message that we need to send it outside the wall of this substation. Once the GOOSE message is subscribed by this agent, the algorithm is activated by praising the received message, extracting its parameters, and activating a DDS publisher. As mentioned before, a GOOSE data model contains important parameters, i.e., StNum, SqNum, and DataField. For broadcasting these parameters by a DDS publisher, a proper IDL file should be defined by suitable data format for each parameter. The StNum and SqNUm contain integer values while DataField has the Boolean type. Therefore, these data types are defined on a DDS topic (e.g., Topic1) within a DDS domain (e.g., Domain n) by three variables (A, B, and C). The activated DDS publisher writes the received parameters' updates on this topic.

Message Encapsulating and Routing Service
To encapsulate this message and route it over the WAN, the DDS routing service is activated and simply routes this message over the network router and delivers it to a predefined topic within a DDS domain in another network, where DDS subscribes are waiting to read data. This is possible just by defining the IP address of both DDS publisher and subscriber through the DDS routing service profile.
As mentioned before, to meet the maximum end-to-end delay for R-GOOSE communication, As mentioned before, specific network settings are required to address this fast delivery over the WAN, therefore, in this paper we propose to deploy the DDS standard for R-GOOSE messages as it covers all the required communication needs. Furthermore, it provides the multicast feature of R-GOOSE messages at the network layer, the same as GOOSE messages at the data layer. This means that several subscribers could receive a published GOOSE message simultaneously and take proper actions immediately. This feature is very helpful in advanced protection systems. In [30], authors have presented an advanced protection scheme for a modern power system, which are experiencing a high level of fault current beyond the breaking capacity of circuit breakers. The proposed advanced protection scheme is mainly based on R-GOOSE messages over the WAN and assumes that a multicast R-GOOSE message framework could deliver the messages to its destination in less than 10 ms. This paper is addressing this framework and demonstrating its capabilities through experimental results. The proposed MRGM algorithm includes three main steps starting with GOOSE publisher in the first network, routing the message over the WAN, and subscribing it within the second network.

Conversion: GOOSE Data Model to DDS Data Object
As shown in the right side of Figure 2 (Substation A), the R-GOOSE communication is begun similar to the conventional GOOSE publishing in data layer. Since GOOSE data model has been defined in the IED platform, to change and add or modify the feature of message we need to develop agents with access to the same network, where GOOSE message is published. This agent is equipped with a GOOSE subscriber with the same APPID of the GOOSE message that we need to send it outside the wall of this substation. Once the GOOSE message is subscribed by this agent, the algorithm is activated by praising the received message, extracting its parameters, and activating a DDS publisher. As mentioned before, a GOOSE data model contains important parameters, i.e., StNum, SqNum, and DataField. For broadcasting these parameters by a DDS publisher, a proper IDL file should be defined by suitable data format for each parameter. The StNum and SqNUm contain integer values while DataField has the Boolean type. Therefore, these data types are defined on a DDS topic (e.g., Topic1) within a DDS domain (e.g., Domain n) by three variables (A, B, and C). The activated DDS publisher writes the received parameters' updates on this topic.

Message Encapsulating and Routing Service
To encapsulate this message and route it over the WAN, the DDS routing service is activated and simply routes this message over the network router and delivers it to a predefined topic within a DDS domain in another network, where DDS subscribes are waiting to read data. This is possible just by defining the IP address of both DDS publisher and subscriber through the DDS routing service profile.
As mentioned before, to meet the maximum end-to-end delay for R-GOOSE communication, specific network settings are needed. The DDS QoS profile contains control mechanism and policies for data transfer over the network. For example, it could control availability of data, life span, latency budget, etc. [31]. The message delivery time is defined by the latency budget, i.e., the index which shows the maximum allowed time span for the message delivery. To control this latency, message priority indices could be attached to time-sensitive and critical messages, therefore, the network switches transfer the received messages based on the priority policies, regardless of data queue. This means that in limited bandwidth networks with high message deliver latencies, which usually is due to the first come first serve policy, defining the message priority profiles could significantly enhance the critical message communications over the internet network [32]. The DDS supports three priority policies including round ronin (RR), earliest deadline first (EDF), and high priority first (HPF). The RR is known as the simplest priority scheduling scheme which processes the messages in the order they have been received. It prioritizes the messages considering their latency budget scheduled in EDF policy, where the messages with less remaining time span are processed first. It is worth mentioning that the EDF is the default policy scheduled by the DDS. The combination of the RR and EDF could be scheduled by HPF policy, where the messages with the same latency budget are processed by the RR policy [31].

Conversion: DDS Data Object to GOOSE Data Model
By delivering the message to its destination in the Substation B' network, a predefined topic (e.g., Topic2) within a domain (e.g., Domain k), a DDS subscriber deploys its data reader to receive data, as shown in the Figure 2, Substation B' algorithm. Afterward, these data are tuned within a GOOSE publisher and published to the local network over the data link layer. Finally, actual IEDs with the same APPID subscribe the message, verify, and react to it.
The same algorithm is run with all other substations or monitoring centers which are supposed to receive data from this GOOSE publisher. The multicast feature and simultaneous subscribing of the R-GOOSE message enable us to implement real-time, reliable, and advanced protection, control, or remedial actions in modern smart power systems [30].

V. Test Framework
To evaluate the performance of this framework in terms of data delivery and delay time, we proposed a test framework, as shown in Figure 3. The purpose is to measure the latency of the algorithm for both GOOSE publishing and subscribing in the data link layer and the DDS time delay over the network layer. This framework contains four Linux-based embedded agents shown by numbers (1) to (4). Agent (1), receives the analog input signal, converts it to digital mode (0/1), sets the GOOSE DataField according to this value, and publishes it to the network by the APPID # 1000. Agent (2) has an active GOOSE subscriber with the same APPID (#1000) and subscribes to this message. The GOOSE DataField along with StNum and SqNum are extracted and are used by DDS data writer to publish these data over the DDS topic. In Agent (3), the DDS message is subscribed and being published by the GOOSE publisher with another APPID (# 2000). Agent (4) receives this message, converts its digital DataField value to an analog signal, and delivers it to output ports of this agent, where this signal could be compared with input to determine the time delay due to communication platform.

Case (1) GOOSE Publish-Subscribe over Layer Two
This experiment is executed using two agents, as shown in Figure 5. To evaluate the latency of the GOOSE delivery, we needed to examine a large number of samples (around 1k) because of various time delays between input and output signals. It is because of this fact that the processors deal with each sample differently, therefore, we had calculate the average latency of GOOSE communication over the data link layer. The minimum and maximum latency recorded in our measurements were around 300 and 1300 µs, respectively, and the average value was around 0.5 ms (TG = 500 µs) with a standard deviation (S.D.) of 50 µs. The low value of the standard deviation indicates that the average value is a good index to demonstrate the time latency of GOOSE communication at data link layer.

Case (2) DDS Publish-Subscribe over the Network Layer
To measure the DDS delay, we had to measure the total end-to-end delay (Tt) by the proposed test framework, as shown in Figure 6. As mentioned before, this communication occurred by two GOOSE publish-subscribe actions plus a DDS data exchange. Therefore, by measuring the total end-to-end delay, the DDS delay time (TD) is calculated by (1).
The average value of the total end-to-end delay (Tt) was obtained by recording a large number of samples (around 1k) as we did in Case 1. The latency of this case is changing between 1000 and 2700 µs. The average and standard deviation are 2100 and 160 µs, respectively. As a result, and as shown by (2), the DDS delay time is 1100 µs. Table 1 summarizes the results for these two study cases. The test framework does not consider the latency of the DDS routing service and network traffic. However, experimental results report an average of around 70 µs for the DDS routing service action (TR). Hence, the average latency due to network traffic (TN) should not be more than 7830 µs, as calculated by (3).  Section Latency (μs) 500 1100 70 7830

Conclusions
In this paper, a multicast communication framework was proposed for R-GOOSE messages over In this equation, T_Max is the maximum allowed time delay for R-GOOSE messages, which is 10 ms, based on IEC 61850 standard. The average time delay for each section of proposed communication framework is reported in Table 2. These calculations mean that the proposed framework is fast enough and only needs around 2170 µs to encapsulate a GOOSE message and transfer it by the DDS and give us 7830 µs for communication over the WAN, which is feasible by defining a tunnel between two communicating networks along with a proper message priority policy. Generally, the network latency is highly dependent on the size of the message, network bandwidth, and performance (unicast/multicast features). As reported in [31], for a 32 bytes message size with the rate of 1 k message per second, the best effort of the DDS QoS profile resulted in 269 µs latency for the communication system. It means that using a proper QoS profile and tunneling technique, the R-GOOSE communication would be feasible over the maximum time span (7830 µs) available when the proposed framework is utilized.