Utilization of SDN Technology for Flexible EtherCAT Networks Applications

At the beginning of the current century, Ethernet-based communication networks began to be implemented in industrial applications. Some previously used protocols were migrated to Ethernet networks, while many others were strictly developed for this communication medium. Numerous industrial Ethernet protocols do not deliver all the capabilities provided by the Ethernet. For example, limitations may arise associated with wireless communication, use of dedicated switching devices, or operation solely for certain topologies. On the other hand, new technologies are now available, such as software defined networks (SDN), that add new features to Ethernet-based communication systems. In this paper, an EtherCAT network in combination with SDN is analyzed. EtherCAT network may only consist of devices with an implemented EtherCAT protocol stack. Therefore, regular Ethernet switches cannot typically be used in this network and, hence, special network infrastructure may be required to create topologies other than standard line topology. It is shown, however, that this limitation can be overcome by the application of SDN. In addition, a definition of datagram forwarding rules (called SDN flows here) is given, and we demonstrate that EtherCAT datagrams can be sent through routes that are required for proper EtherCAT network operation.


Introduction
Efficient communication systems for industry are indispensable nowadays. New industrial systems are most often distributed, and they consist of many devices connected by networks. For many years, data exchange between nodes in industrial communication systems was based on fieldbuses. However, in the early 2000s, an Ethernet standard began to be extensively implemented for this purpose. Many protocols used for fieldbuses migrated to, after some adaptations, modern Ethernet communication links, e.g., Modbus TCP [1]. In addition, other protocols were developed strictly for the Ethernet, these include: Soon afterwards these protocols, alongside some others, became part of the international standard IEC 61158/IEC 61784-2 [2] and, today, they are among the most recognizable in the domain of industrial automation. It is noteworthy that another important protocol, which is not defined in this standard, is known as OPC UA; its first version was released in 2006 [3,4].
Even though numerous protocols were developed solely for application as an Ethernet standard, they cannot take advantage of all the characteristic features of an Ethernet network. The limitations start from the temporal requirements, typical in industrial systems not send separate datagrams that exchanges data with different slave stations using the request-response approach. EtherCAT datagrams contain processing data that propagates through all the EtherCAT slaves, which are called EtherCAT Slave Controllers ESC. This means that only one datagram is required for communicating with many ESC, which is then processed by the latter without the need for data buffering and storing. Therefore, minimal latency for the propagation of the datagram is introduced [20] and the exchange process for the input and output data becomes highly effective. In detail, when a datagram arrives at an input Ethernet port of a given ESC it is efficiently forwarded almost concurrently to its output port. The output dataset is extracted from the datagram "on-the-fly"; the input state is written similarly to the datagram.
The processing of the datagram is implemented by computer hardware in the form of a Fieldbus Memory Management Unit (FMMU) of the ESC. Datagrams traverse through FMMUs with a latency as low as 1 µs for slave stations equipped with at least one Ethernet interface (100BaseTX or 100BaseFX) and 300 ns for slaves with only low voltage differential signal (LVDS) ports (this is also called an E-Bus interface, in devices by Beckhoff (Verl, Germany), which is used as a backplane connection in terminal modules such as input and output modules [27].) EtherCAT datagrams are transferred by just using Ethernet frames with broadcast MAC addresses. In other words, in reference to the ISO/OSI model, only the physical and data link layers (defined in Ethernet standard) are implemented; protocols from the upper layers are not used. In particular, no IP addressing is required since EtherCAT is only concerned with the application layers.
Apart from EDP, which is briefly described above, there is also the possibility of using EAP. This protocol involves communication between the master stations and it is based on the producer-consumer principle. However, the details on EAP are not described in this paper since we solely focus on EDP (for simplicity, this is just called EtherCAT later); the presented solution can, however, be applied to both. In EAP, "typical" Ethernet network devices and "typical" datagram processing can be used (but "on-the-fly" processing does not apply).
EtherCAT Device Protocol operates in a logical ring. Every datagram traverses from one ESC to another in a consecutive fashion. By taking advantage of full-duplex communication, the final topological ESC returns the EtherCAT datagrams to the master station, as shown in Figure 1. As stated earlier, it is not possible to include regular Ethernet switches in an EtherCAT network; one of the reasons is the need for broadcast MAC addressing. When a tree or star topology is built, a module called an EtherCAT junction should be used (also displayed in Figure 1).

EtherCAT Protocol
Communication using EtherCAT Device Protocol, which is typically simply called the EtherCAT protocol, is based upon a master-slave principle. However, in contrast to other master-slave communication networks (for example, the well-known Modbus communication network or Profibus, in which the master-slave principle is applied alongside the token passing mechanism between the master stations), the EtherCAT master station does not send separate datagrams that exchanges data with different slave stations using the request-response approach. EtherCAT datagrams contain processing data that propagates through all the EtherCAT slaves, which are called EtherCAT Slave Controllers ESC. This means that only one datagram is required for communicating with many ESC, which is then processed by the latter without the need for data buffering and storing. Therefore, minimal latency for the propagation of the datagram is introduced [20] and the exchange process for the input and output data becomes highly effective. In detail, when a datagram arrives at an input Ethernet port of a given ESC it is efficiently forwarded almost concurrently to its output port. The output dataset is extracted from the datagram "on-the-fly"; the input state is written similarly to the datagram.
The processing of the datagram is implemented by computer hardware in the form of a Fieldbus Memory Management Unit (FMMU) of the ESC. Datagrams traverse through FMMUs with a latency as low as 1 µs for slave stations equipped with at least one Ethernet interface (100BaseTX or 100BaseFX) and 300 ns for slaves with only low voltage differential signal (LVDS) ports (this is also called an E-Bus interface, in devices by Beckhoff (Verl, Germany), which is used as a backplane connection in terminal modules such as input and output modules [27].) EtherCAT datagrams are transferred by just using Ethernet frames with broadcast MAC addresses. In other words, in reference to the ISO/OSI model, only the physical and data link layers (defined in Ethernet standard) are implemented; protocols from the upper layers are not used. In particular, no IP addressing is required since EtherCAT is only concerned with the application layers.
Apart from EDP, which is briefly described above, there is also the possibility of using EAP. This protocol involves communication between the master stations and it is based on the producer-consumer principle. However, the details on EAP are not described in this paper since we solely focus on EDP (for simplicity, this is just called Ether-CAT later); the presented solution can, however, be applied to both. In EAP, "typical" Ethernet network devices and "typical" datagram processing can be used (but "on-thefly" processing does not apply).
EtherCAT Device Protocol operates in a logical ring. Every datagram traverses from one ESC to another in a consecutive fashion. By taking advantage of full-duplex communication, the final topological ESC returns the EtherCAT datagrams to the master station, as shown in Figure 1. As stated earlier, it is not possible to include regular Ethernet switches in an EtherCAT network; one of the reasons is the need for broadcast MAC addressing. When a tree or star topology is built, a module called an EtherCAT junction should be used (also displayed in Figure 1).  For the above reasons, common IT infrastructure cannot typically be utilized in Ether-CAT communication, although its application could be made possible by using the Time Sensitive Network concept [28]. Since 2021, an IEEE task group has been attempting to standardize the real-time and safety of networks [29], and apply a critical enhancement for the Ethernet [30]. Unfortunately, we authors have not found any information on the

Software Defined Networks
In traditional Ethernet networks, the decisions on the routing of the datagrams are made by the switching devices. In SDN, an additional autonomous device is used that controls the data traffic-this is called the SDN controller. In SDNs, the control plane is separated from the data plane and the central SDN controller coordinates the data flow within the network. SDN only switches forward the datagrams according to the SDN controller instructions, which is known as the flows [31]. Great advantages may arise by decoupling the control and the data plane alongside a control of the centralized network. Above all, in such a scenario, decisions made by the SDN controller are based on a global view of the network. Therefore, all the decisions are more relevant to the current condition of the whole network [32].
SDN may be applied in various areas: Internet of Things [33] and Wireless Sensor Networks [34][35][36], industrial systems [37], or on the factory floor [38]. They can be implemented to improve the management and maintenance of networks [39], as well as increasing the security by use of Intrusion Detection Systems [26], protecting against eavesdropping attacks [40,41], creating resilient networks [42], and enhancing network reliability [43]; related papers also concern communication determinism [27]. In further reference to SDN, the Future Industrial Network Architecture (FIND) defines a guidance on the industryrelated requirements in communication systems development [28]. For our purposes, SDN is applied so that datagrams forward to different ports of the SDN switching devices (called OVS or vSwitches). Moreover, every datagram is forwarded to the hosts of the networks, consecutively, in a predefined order. As a result, the topology resembles a line topology that corresponds to datagrams routing within EtherCAT networks.

EtherCAT over SDN
As stated earlier, a regular Ethernet infrastructure cannot be used within EtherCAT networks because a significant part of the datagrams in the EtherCAT is sent as broadcasts. Moreover, the broadcasted datagrams must reach every slave device in consecutive order. While typical Ethernet switches cannot cope with this type of datagram forwarding, the OVS within SDN are capable of this task. Despite the various addressing possibilities for the datagrams (i.e., either broadcasted, multicasted, or unicasted), forwarding by OVS is possible for any required scenario in a consecutive manner that accords to the needs of every slave device. By defining the appropriate flows in the SDN network, datagrams routing can be freely chosen according to, for example, the type, length, or payload of the datagrams.

Experimental Research
The concept of EtherCAT over SDN (also known as EOS) was verified during the experimental research. The testbed consisted of one Beckhoff controller (a EtherCAT master) and four ESC devices: EK1100 with some I/O modules. The EtherCAT network components were connected in two ways: • A regular EtherCAT network in a line topology, which uses the switches that are integrated into the devices (see Figure 2). • An implementation of the EOS concept, so that the ESC is connected to the master station via OVS (see Figure 3). In Figure 2, a typical EtherCAT network is presented that contains one master station and four ESC devices connected in a line topology. This is the most common topology in EtherCAT networks, since every ESC device contains an Ethernet switch built into the network interface. Other topologies are also possible, but they require the use of additional modules such as the EtherCAT extension modules (e.g., EK1110 by Beckhoff) or the EtherCAT junction modules (e.g., EK1122 by Beckhoff) that are mentioned in the previous section (see Figure 1). An identical set of EtherCAT devices, which are connected to an SDN network, representing the EtherCAT over SDN (EOS) solution is shown in Figure 3. For this scenario, any physical topology is possible; here, a star topology was chosen (as depicted in the figure) in order to present a topology that is significantly changed from the original line topology.
Using the EOS solution, a logical line topology that is typical for EtherCAT networks is realized by using an appropriate definition for the SDN flows. According to such flows, every datagram that is sent by the EtherCAT master is received on port P1.3 of the S1 OVS and then forwarded to port P1.4, which is where the ESC1 (remote I/O station) is connected. Every datagram that derives from the ESC1 slave on port P1.5 is forwarded to port P1.6, which is connected to the ESC2 slave, and so on. The datagrams eventually reach ESC4 on port P2.3, from which the ESC4 datagrams return to the master station via the same route (the opposite direction required for this is not marked on Figure 3). Of course, while the datagrams travel through every ESC device, the IO data associated with the EtherCAT master station In Figure 2, a typical EtherCAT network is presented that contains one master station and four ESC devices connected in a line topology. This is the most common topology in EtherCAT networks, since every ESC device contains an Ethernet switch built into the network interface. Other topologies are also possible, but they require the use of additional modules such as the EtherCAT extension modules (e.g., EK1110 by Beckhoff) or the EtherCAT junction modules (e.g., EK1122 by Beckhoff) that are mentioned in the previous section (see Figure 1). An identical set of EtherCAT devices, which are connected to an SDN network, representing the EtherCAT over SDN (EOS) solution is shown in Figure 3. For this scenario, any physical topology is possible; here, a star topology was chosen (as depicted in the figure) in order to present a topology that is significantly changed from the original line topology.
Using the EOS solution, a logical line topology that is typical for EtherCAT networks is realized by using an appropriate definition for the SDN flows. According to such flows, every datagram that is sent by the EtherCAT master is received on port P1.3 of the S1 OVS and then forwarded to port P1.4, which is where the ESC1 (remote I/O station) is connected. Every datagram that derives from the ESC1 slave on port P1.5 is forwarded to port P1.6, which is connected to the ESC2 slave, and so on. The datagrams eventually reach ESC4 on port P2.3, from which the ESC4 datagrams return to the master station via the same route (the opposite direction required for this is not marked on Figure 3). Of course, while the datagrams travel through every ESC device, the IO data associated with the  Figure 3. Depiction of the EtherCAT over SDN (EOS) network solution in a star topology. The encircled digits "1" to "4" denote the ET2000 probe locations (measurements points no. 1 to 4).
In Figure 2, a typical EtherCAT network is presented that contains one master station and four ESC devices connected in a line topology. This is the most common topology in EtherCAT networks, since every ESC device contains an Ethernet switch built into the network interface. Other topologies are also possible, but they require the use of additional modules such as the EtherCAT extension modules (e.g., EK1110 by Beckhoff) or the EtherCAT junction modules (e.g., EK1122 by Beckhoff) that are mentioned in the previous section (see Figure 1). An identical set of EtherCAT devices, which are connected to an SDN network, representing the EtherCAT over SDN (EOS) solution is shown in Figure 3. For this scenario, any physical topology is possible; here, a star topology was chosen (as depicted in the figure) in order to present a topology that is significantly changed from the original line topology.
Using the EOS solution, a logical line topology that is typical for EtherCAT networks is realized by using an appropriate definition for the SDN flows. According to such flows, every datagram that is sent by the EtherCAT master is received on port P1.3 of the S1 OVS and then forwarded to port P1.4, which is where the ESC1 (remote I/O station) is connected. Every datagram that derives from the ESC1 slave on port P1.5 is forwarded to port P1.6, which is connected to the ESC2 slave, and so on. The datagrams eventually reach ESC4 on port P2.3, from which the ESC4 datagrams return to the master station via the same route (the opposite direction required for this is not marked on Figure 3). Of course, while the datagrams travel through every ESC device, the IO data associated with the given device are processed, i.e., the output and input data are read from and written to the datagrams, respectively. This type of data processing is based on a "on-the-fly" approach, which means that the IO data can be processed even before the whole datagram is received from the network. The presented concept was verified during the experimental research; this is described in the next section. In greater depth, the main goal of the experimental research was to verify if EtherCAT communication can be coordinated in a SDN network, which was shown to be possible.
It is also important to determine the temporal costs that the OVS introduce to the EtherCAT communication network. Measurements on the network traffic were performed to obtain this necessity. For this purpose, an ET2000 multi-channel probe by Beckhoff was used. This enabled the capture of Ethernet datagrams with precise timestamping at the four monitoring points of the network; these are marked by encircled digits from "1" to "4" in Figure 3 reffered latter as measurment points. The datagrams were registered using a regular PC with Wireshark software. Analysis of the timestamps enabled the measure of the network latencies, which are introduced by the devices located between the monitoring points. The timestamps are added, by the ET2000 probe, at the end of every Ethernet datagram. Therefore, the Ethernet traffic analysis is not dependent on the device used for capturing the datagrams. The timestamps had the resolution of 1 ns; however, the actual accuracy of the timestamping according to the vendor is 40 ns [44], whereas the delay in communication that the probe introduces to the network is lower than 1 µs.
For the testbed illustrated in Figure 3, the following components were used: • The two switches by Mikrotik (Riga, Litvia) that were used had their firmware changed to the OpenWRT and OpenvSwitch package. In addition, the L2 functionality of the switches was disabled. Therefore, without the defined SDN flows, the switches cannot provide any forwarding for the datagrams. Configuring the SDN flows required OpenFlow Manager, which was used to define the flows as described above. These flows did not contain any filtering, so that every datagram received on port P1.3 was forwarded to port P1.4, and so on.

Results
The Ethernet network traffic was captured for the two system topologies, i.e., with and without application of the EOS solution, as depicted by Figures 2 and 3. The results of which are presented in terms of the histograms of Figure 4 (for a system without SDN) and Figure 5 (with SDN and the EOS solution applied). In the experiment, 100,000 datagrams were captured and analyzed. In the former case, without the SDN, the x-axis represents time in the units of microseconds, while the y-axis is the number of measurements at a given network cycle, which was configured to be 10 ms. Most of the results reside at 10.1 ms, although signficant numbers are seen at a slightly longer time. In general, the communcation data remain within the configured cycle time with a 2% jitter. This shows that the latencies introduced by the ESC are minimal; the specifications of which are described in Section 3. Datagrams traverse through the FMMUs of the ESCs with latencies as low as 1 µs for the slaves that contain at least one Ethernet interface (100BaseTX or 100BaseFX) and 300 ns for the slaves with only LVDS ports. The delay introduced by the ESC2-ESC4 and the module EK1100 of ESC1 equates to 1 µs. While every I/O module of the ESC1 creates a latency of 300 ns. A total latency as little as 1.2 µs, for the four modules, is thus highly plausible.
The system with the implemented EOS solution was more thoroughly tested. Measurements at the four different points were taken, which enbled a calculation of the Ether-CAT datagrams flow in terms of latency time.    The system with the implemented EOS solution was more thoroughly tested. Measurements at the four different points were taken, which enbled a calculation of the Ether-CAT datagrams flow in terms of latency time.    The system with the implemented EOS solution was more thoroughly tested. Measurements at the four different points were taken, which enbled a calculation of the EtherCAT datagrams flow in terms of latency time. Figure 5a represents the variation in latency for the EtherCAT datagrams with the EOS solution implemented. Every measurement corresponds to the latency of a datagram that is transferred through the network. For example, over 16,000 datagrams each required a latency time of 460 µs to be transferred from the PLC to the final ESC device and returned. Here, in contrast to Figure 4, only the latencies are presented with no reference to the network cycle time (which was again set at 10 ms). For the EOS solution, compared to the system without SDN, the latency values are significantly greater and the distribution of them is much more spread out. The most likely measurement is 460 µs and over 90% of the measurements reside in a range from 360 to 680 µs-the jitter is, thus, much greater when the EOS solution is applied. This dataset is represented in the figure by ESC1 RX , which refers to the time period that begins when the datagrams are sent from the PLC (through the measurement point no. 1) and ends when the returned datagram is received from ESC1. In other words, the time required for the datagram to travel through the ESC1, ESC2, ESC3, and ESC4 devices and back. In an EtherCAT network, this involves one datagram that is consecutively processed by all the ESC.
The latency of ESC1 RX is calculated based on the datagrams captured with the ET2000 device at the measurement point no. 1. For every measurement point, the two Ethernet interfaces with the ET2000 probe are related. Datagrams that are received on one port are then forwarded to another port (which applies to every measurement port) and every pair of ports creates a monitoring channel. Additionally, the incoming datagrams are duplicated and sent to the monitoring port (in detail, the uplink-nineth Ethernet port of the ET2000 probe is used, while the ports of the monitoring points operate in the 100 Mbs mode and the uplink, i.e., the port with the duplicated datagrams, is a 1 Gbs port). The monitoring port is connected to the capturing device-namely, a PC with Wireshark software. The duplicated datagrams are extended by a status field (called the EtherCAT switch link), which consists of the port number that corresponds to when the datagram was received and a timestamp. For the measurement point no. 1, the calculations for the delay between receiving the datagrams from the PLC and the returned datagrams from ESC1 are determined. Other descriptions, similar to ESC1 RX , are displayed in Figure 5b. They refer to the following measurements: A summary of the mean and median of the measured results is presented in Table 1. The difference between the ESC1 TX and ESC2 TX values is very small, because it refers to the latency introduced by the ESC2 device only. According to its specifications it should be 1 µs, which is consistent with the presented results. The variation between ESC2 TX and ESC3 TX corresponds to the latency of the OVS-which is around 85 µs. However, the difference between ESC3 TX and ESC4 RX includes two passages through the OVS switches and one through the ESC4; here, the difference is around 100 µs. Moreover, the value of ESC1 TX includes a double routing through the OVS S1 switch, which shows some discrepancy since it has an average value around 147 µs. However, this inconsistency may result from not using a very powerful device, which is made from a regular OpenWRT switch, in the role of the OVS. It is expected that the results will be much more consistent when a dedicated OVS is used. Nevertheless, the experiment shows that the EOS solution can operate as required, although its temporal characteristics should be fully considered when applied to real-time systems. It is noteworthy that a similar OVS switch could be implemented on a Raspberry Pi device, which is equipped with Ethernet interfaces connected to the USB ports. However, the latency introduced by this type of switch could be much more significant [45].

Discussion
Our experimental research demonstrated that, by the implementation of SDN technology, it is possible for an appropriate routing of EtherCAT datagrams to create communication in EtherCAT networks via SDN. The definition of flows for the EtherCAT datagrams using OVS enable the ESC devices to operate seemingly in a line topology, while the actual (physical) architecture may not have this topological characteristic at all. As a result, the EtherCAT traffic can be routed through regular IT infrastructure, which enables efficient real-time interconnects between different areas of a production plant or factory floors without a need for elaborate and expensive devices.
Importantly, inclusion of additional devices in the EtherCAT network (i.e., the vSwitches/ OVS) will introduce some extra latency to the datagram transfer. In the testbed experiment used, this was measured to be around 85 µs. On a comparison of this value with the latency of the EtherCAT slaves, which are as low as 1 µs or even 300 ns (depending on the type of slave device used), the OVS latency may seem to be overly large. Nevertheless, in most industrial applications, the network cycle is expressed in milliseconds rather than microseconds. Therefore, even with these types of delays, the presented solution can be effectively applied to real scenarios. Nevertheless, of course, it cannot be employed when high-precise communication is required, for example, in the synchronous operation of servo drives; in most cases, however, it should be applicable. In addition, the measured latency applies to the general-purpose switches (the Mikrotik) that were used in the testbed experiments, where the firmware is changed to OpenWRT. It is expected, therefore, that more advanced solutions that significantly minimize the latency in such cases should be possible.
Another drawback of the presented solution is that double Ethernet cables are necessary for the connection of every ESC to the OVS. Importantly, these additional cables are only required between an OVS and the first ESC, since the other ESC can be connected directly to the first ESC in a line topology. In future work, there are plans to eliminate this requirement, so that the operation of the OVS is similar to the EtherCAT junction module shown in Figure 1, in which only one Ethernet cable for every ESC is needed.

Conclusions and Future Work
In this study, we demonstrated that SDN technology can be implemented in the region of industrial Ethernet applications and it may overcome some of the limitations associated with EtherCAT networks. Typically, for example, EtherCAT networks are required to have a line topology when no EtherCAT junction modules are employed and regular Ethernet switches cannot usually be applied. We have demonstrated that both of these limitations can be removed by the use of SDN. Application of common switches, such as OVS (or SND switches), may deliver flexible network infrastructure even when industrial communication protocols such as EtherCAT are used. Moreover, common IT network infrastructure become useable in such networks if it supports SDN. Another advantage is that changes to the structure of the network is possible by a modification of the program, so that physical rewiring of the network can be avoided. To enable system architecture to be even more flexible, future works plan to implement wireless communication in EtherCAT networks, since it should be possible to define SDN flows that enable the routing of EtherCAT datagrams in a Wi-Fi network. The general intention, for this future research, is to interconnect more than two devices with one wireless network.
The presented solution is applicable not only to EtherCAT networks but also systems based on other protocols. For example, in EPL networks, only one distributor node may be used, called the Managing Node (MN), which creates one real-time domain. When additional MNs are present, separate Ethernet infrastructure should, in general, be used. Moreover, different EPL real-time subsystems may coexist on the same infrastructure, which has applications to managed Ethernet switches and the logical separation of the domains by a VLANs configuration. However, such domain separation could be based on SDN in future. It would increase the security of the EPL by not forwarding datagrams that arrive from other ports of the OVS (SDN vSwitches), which is where the controlled nodes (CN) are connected; for example, those from outside the processing network.