A microgrid (MG, Figure 1
) is a low voltage distributed network formed by various distributed energy resources (DERs) consisting of a variety of loads, microsources (MS), energy storages systems (SS), and other incipient elements like electric vehicles (EVs) [1
]. Microgrids have emerged as a powerful, resilient and sustainable power grid concept that can integrate renewable energy systems for power generation [4
], and manage in real time a large amount of distributed energy resources [6
], Microgrids can operate in grid-connected mode and islanded mode disconnected from the point of common coupling (PCC) with the main grid in case of faults [8
]. In general, there are several kinds of faults that can originate the disconnection of a certain area in an electrical grid, as would be the case of short-circuits, line overloads, faults in substations, and so on. Once the fault has disappeared, they can be reconnected to the grid. In addition, a microgrid must have its own control architecture to ensure the correct operation and coordination of the different DERs.
In recent years, the introduction of Information and Communication Technology (ICT) in microgrid energy systems has led to the term “Smart Microgrid” (SMG) [11
]. This concept has emerged to describe the way ICT can transform microgrids’ operation and entails several applications like intelligent monitoring and control of distributed energy resources, automation, data networking and other tasks that can improve the efficiency of the system [12
]. To realize these capabilities, distributed control systems have been proposed. Particularly, recent studies [13
] suggest Peer-to-Peer (P2P) overlay communication networks as a solution with great potential to provide high performance to decentralized microgrid control systems, including scalability, fault-tolerance and robustness.
Peer-to-Peer (P2P) overlay networks are virtual networks where the connectivity among the nodes is carried out through a physical IP network, while the network topology is created in a virtual network, called overlay, which is built on top of the physical one. Overlays increase flexibility, extensibility and adaptive reconfiguration. This implies that each node communicates with each other to create self-organizing overlay structures on top of the subjacent physical networks [18
Distributed Hash Tables (DHTs) allow developing structured P2P systems due to their routing capabilities. DHTs support exact-match queries [20
] i.e., given a query for a specific key, DHTs can efficiently locate the node which holds the keyword key. However, DHTs do not implement any mechanism for retrieval data application [1
] and locality is not considered [21
]. These facts are a serious drawback to implement communications in microgrids by means of DHTs due to, on the one hand, the fact a data retrieval mechanism is essential to obtain and/or update data in this application [23
]. On the other hand, locality allows creating a group of peers for a particular task [21
]. Peers with close interests can create “shortcuts” and use them to locate content. The underlying physical network path could be significantly different from the path on the overlay network if locality in DHTs is not considered. Therefore, the lookup latency in the overlay network could be quite higher and decrease the performance of the application layer [21
The control and the management of microgrids demand high efficiency in terms of network quality requirements (high bandwidth and low latency). However, peer-to-peer networks have been mainly developed for other applications such as file sharing and processor cycle sharing, whose network performance requirements are less of a priority [26
The main contributions of this work are, first, the proposal of a fully functional decentralized communication architecture, which includes a new clustering algorithm based in DHTs that adds locality capabilities for creating node clusters with close interests. The second contribution of the paper is the proposal of a specific application protocol, running on Transmission Control Protocol/ Internet Protocol (TCP/IP), for microgrids monitoring and control.
The paper is organized as follows: Section 2
gives some background on P2P networks and their application to microgrids. In addition, a specific DHT protocol (Chord) is described. Section 3
presents the network requirements in order to deploy efficient communication architectures for smart microgrids. In Section 4
and Section 5
the proposed solutions are described. Concretely, the lookup algorithm is presented in Section 4
, while Section 5
gives details about the communication architectures and a specific protocol for microgrids. Section 6
reflects the results of the tests that have been carried out to evaluate the performance of the proposed solutions. Finally, Section 7
provides some conclusions.
2. Peer-to-Peer Approach for Decentralized MG’s Control
One of the key points for integrating DERs into microgrids is the design of a control architecture that coordinates each one of the DER units. Traditionally, the control tasks are carried out by means of a three level hierarchical structure [27
], i.e., primary, secondary and tertiary control. The Field Level (Primary Control) is implemented in local DER. It is responsible for stable operation conditions in each DER. The Management Level (Secondary Control) is controlled by a Microgrid Central Controller (MGCC), which ensures the synchronization between the MG and the main grid or the quality of the frequency and voltage of MG, among other tasks. Finally, Grid Level (Tertiary Control) carries out the management of MG when it operates in the market providing an economically optimal operation of the microgrid.
Primary, Secondary and Tertiary levels can be combined or not with communication systems. However, to evolve towards the development of the intelligent microgrid, communication interfaces among the different control levels should be created. Thus, a bi-directional communication and channels are stablished in order to allow information transfer between the different controllers [32
] for knowing the state of the whole system.
Today’s microgrid installations use centralized architectures to implement the three level hierarchical scheme and data transfers between entities. These centralized architectures where a central controller take decisions and provides communications between all microgrid resources have been implemented, for many years, by Supervisory Control and Data Acquisition (SCADA) systems [33
]. In addition to the SCADA systems, other alternative technologies also used for control and management of centralized microgrids are Power Line Carrier (PLC) and Wireless Networks (WNs). PLC technologies use the electric power lines as a medium that enables the bidirectional data exchange. This provides a vast coverage and in terms of infrastructure is the most cost-effective technology since the lines already exist. However, PLC technology has negative effects in the communication channel such as a noisy medium, distortion, frequency impedance alterations and the risk of signal attenuation [34
Besides, wireless technologies have emerged as an alternative for centralized wired technologies due to their low cost, low power, flexibility and easy deployment [35
]. Some studies about wireless networks in power systems for monitoring and control of segments in power transmission and delivery have been presented in [35
]. However, the use of wireless technologies in power system environments presents a number of challenges such as reliability concerns, wireless signal disruption due to electromagnetic interference (EMI), faded signals due to large transmission distances or obstacles in the line-of-sight; overloading of bandwidth; quality or latencies degradation, among others [35
In Information and Communication Technology (ICT) terminology, these centralized structures (see Figure 2
a) are based on client-server communications architectures where every device of the network plays one of two roles: server or client. The server is the central point that receives status information from the whole unit and it is able to calculate an optimal global control strategy. Besides, the client shares its resources via direct links to the server. This control structure causes inefficiencies in the communication system of the microgrid, which are provoked by several causes. On one hand, a failure in the centralized control point could lead to several faults or even the shutdown of the entire system. In addition, there are difficulties in managing a wide range of devices in real time [42
], because the system is hardly scalable. As a result, these disadvantages can lead to provide poor services, bottlenecks or under-utilization of the network resources. To address these shortcomings, the communications network should move from the conventional centralized infrastructure to a decentralized one. A decentralized communication infrastructure, removes the central controller as a single point of failure, providing a significant improvement in terms of reliability. Therefore, the hierarchical management scheme of microgrids should be implemented employing a decentralized architecture.
Decentralized architectures for power systems are usually implemented through Multi-Agent Systems (MAS) technology [44
]. In this structure each DER unit is considered as an agent. An agent is a computer system capable of performing tasks autonomously and with the ability to communicate with its neighboring agents to solve problems through cooperation, coordination and negotiation. Most MAS implement hierarchical topologies where some agents (super-agents) have authority over the actions of other agents [48
] to form distributed decentralized systems (see Figure 2
b). However, these agents do not have the capacity to reorganize themselves [49
]. In addition the individual behavior of the agents is easy to know while the whole system is unreachable [50
]. The absence of such functionality might result in a lack of knowledge about the global status of the microgrid, as well as in a sub-optimal allocation and a poor management of the critical resources [49
]. Therefore, a new paradigm based on peer-to-peer (P2P) communications could be a promising solution for improving the use of DERs and the network performance in microgrids [17
P2P systems are formed by peers. A peer is an agent [49
] that can act both as server and client, characteristic that allows agents be proactive. P2P systems avoid a single point of failure and are much scalable because the available resources grow with the number of nodes joining the network. Nodes are capable to cooperate to achieve a common goal and they have self-organization capabilities. The connectivity between nodes, as Figure 2
c depicts, is carried out by creating virtual links, overlays, which are built on top of the IP network. Some advantages of these systems are their greater flexibility, their extensibility and their adaptive reconfiguration. This implies that each node communicates with each other to create self-organizing overlay structures on top of the subjacent physical networks [54
It is worth pointing out that P2P networks have a highly distributed nature, so they offer much large scalability than possible with other networks that are based on centralized or partially distributed systems, due to the fact each peer could act as a server. Therefore, the typical bottlenecks of centralized and partially distributed systems are avoided because the number of servers increases linearly with the one of clients.
P2P architectures can be classified as Structured or as Unstructured according to their logical topology and degree of decentralization. On the one hand, an unstructured P2P network has a random and unstructured mesh network topology. There is not an algorithm for organization. The information and data resources are distributed among peers. A Broadcasting lookup technique is used to locate resources and data retrieval in unstructured P2P, in such a way that each peer propagates a request to its directly connected peers. The propagation remains until the message time to live (TTL) threshold (typically four) has been reached [55
]. This broadcast creates a large amount of signal traffic and uses of a lot of network bandwidth [56
]. These characteristics do not result in a scalable and efficient system.
On the other hand, a structured P2P network has a dedicated network and a well-defined topology where peers are responsible for information and data resources. In structured overlays, a Distributed Hash Table is used for routing in order to locate resources in the network. In this strategy each peer has a local table that is used as a lookup algorithm to route the request data according to node tables [57
]. DHT allows the peers to find the addressed data using flat identifiers (IDs). This kind of P2P system improves the network communication usage, and it is shown in Figure 3
compares the performance of broadcasting and DHT lookup algorithms for distributed peer to peer architectures. The complexity of each of these lookup algorithms can be evaluated by analytical metrics [58
]. Analytical metrics for structured DHTs and Broadcasting-unstructured P2P designs have been proposed in [59
]. The computational behavior of these lookup algorithms is described by means of O, which describes the asymptotic behavior of the algorithms [58
]. In a structured DHT overlay, each node maintains information about the O (logn) value of other nodes and solves lookup via O (logn) messages per lookup. In broadcasting, the metric is O (logn) messages per lookup. It is worth noting that O notation expresses the asymptotic upper bounds, since it bounds the growth of the message exchange for a large enough number of input nodes. This means that the exchange messages grows no faster than a certain constant value (in the case of broadcasting) or a logarithmic one (in the case of DHTs). In fact, it grows slower. From a practical point of view, the value of O(n) agrees with n [58
Message counts are considered as a metric value for communication overhead [60
] and can provide evidence about the network and the bandwidth usage, as well as about the end-to-end latencies. O(logn) and O(n) indicators depict the worst case scenario for n nodes in the network.
As Figure 3
shows, the DHT lookup technique is more efficient than Broadcasting. Thus, it will be used in the proposed solution. In addition with DHT the resources discovery can be satisfied in a bounded number of steps even for large scale distributed systems.
3. Description of the Chosen DHT Algorithm
There are in the literature a large amount of DHT routing infrastructures based on P2P systems, and a possible taxonomy is presented in [54
] Examples of DHT protocols include Chord [61
], CAN [62
], Pastry [63
], and Tapestry [64
], among others. They differ mainly in how peers maintain their routing tables to guarantee an efficient content location [65
Among all of possible DHT routing protocols listed before, the Chord protocol has been chosen as lookup algorithm in this work. The reason is that Chord is the most popular structured routing protocol [54
] and it has several features that distinguish it from the other DHT lookup algorithms. These characteristics are [67
]: (i) Chord is a completely distributed system, meaning that all nodes have the same role in the system; (ii) Chord properly scales for a large number of nodes; (iii) Chord nodes can always be found, even if they are continuously entering and leaving the network.
Although the DHT-Chord protocol has been chosen to build up the proposed lookup algorithm, it is worth noting that the proposed concepts are independent of the chosen DHT protocol and they can be easily extended to other protocols. The Chord DHT-overlay organizes peers on a virtual ring topology ranged from 0 to
a), where m
is the number of bits in the identifiers. In this protocol each node is responsible for a collection of keys between its predecessor and itself for resource lookup (key-space). In this way, the resource ID is stored in the first peer, whose ID ≥ Resource ID (see Figure 4
b). Each node in the ring upholds a routing DHT table, called the finger table, which is used by the lookup algorithm (Figure 4
The lookup algorithm is started by one node in the ring in order to find a particular key in the space-keys or by an external request and follows two steps [54
Firstly it checks if the node that started the search is in charge of that key. If this is true the search is over and the algorithm ends.
Otherwise the node will employ its finger table to localize the closest successor of the target node’s key and request the search of the key to the target node.
Let us consider an example. When peer #6 is searching the resource whose ID is 16, i.e., the target node will be peer #3, which is the node that stores ID = 16 (see Figure 4
). The peer #6 verifies if is in charge of that key. As it is not responsible for this resource, it will check its finger table records. The finger table of peer #6 stablish that the nearest peer to the destination is peer #15 (see the finger table of N6
in Figure 4
c), and it sends the request to this successor. As the peer #15 is also not responsible for this resource, it would also check its own finger table, searching the closest peer of the target’s node. The finger table of peer #15 establishes in this case that the closest node to meet the request is peer#3 (see finger table of N15
in Figure 4
c), thus it forwards the message to this successor. Peer #3 has the resource and it is the destination, so the process ends.
In a P2P system, each peer can join and leave the system at any arbitrary time. Chord adapts efficiently as nodes join and leave the system, and can answer queries even if the system is continuously changing [24
]. This means that the finger table is dynamically built through a stabilization mechanism.
4. Network Requirements for Smart Microgrids
To control and monitor the intelligent microgrid, a Real-Time Measurement Parameters (RTMP) function is required [69
]. To reach this objective, it is mandatory to know what bandwidth (used to signify the data rate in bits/second) and what latency (delay) the transmitted data may have among the microgrid components (DERs) for each microgrid function [71
]. Each microgrid function has its own latency and bandwidth requirements depending upon the kind of system response it is dealing with. The International Electrotechnical Commission's (IEC) 61850 and IEEE 1646 standards give specifications for these requirements as Table 1
The communication system of the microgrid must satisfy these timing requirements because a low bandwidth can lead to bottlenecks, loss of data packets and distortion. Besides, if the communication delay exceeds the required time, the information does not fulfill its purpose and, in the worst case, damage might occur [34
]. In this regard, the underlying communication system needs to be designed with well-defined network performance requirements to meet the needs of time sensitive data streams, bandwidth and latency, among others [77
shows the needed performance requirements as stated by the IEC 61850 and IEEE 1646 standards. Besides these requirements, microgrids are evolving towards distributed and intelligent systems, needed to process a high volume of data. Thus, the management and control of microgrids is becoming a critical challenge. The latest research indicates that current distributed MAS have some limitations for the efficient and optimal management of microgrids [49
]. On the one hand, to get the overall information about the microgrid dynamic network, the agents should be grouped in clusters, following a certain criterion (for instance, the kind of resource). The absence of such functionality might result in a lack of knowledge about the global status of the microgrid and, therefore, in sub-optimal resource allocation. On the other hand, agents have neither dynamic reorganization nor self-healing capabilities by themselves, which would result in better response to local-failures, microgrid blackouts, agents’ crashes or communication failures.
According to the general needs described above, P2P communication systems offer some interesting features to meet the desired requirements. First, P2P protocols have a high degree of flexibility and can be easily adapted to new applications by means of overlays. Second, the communication system must be highly scalable while maintaining the capability to process the growing volume of data. These characteristics are inherent to P2P systems. However, P2P networks were mainly developed for file sharing and processor cycle sharing. In these applications, the network performance requirements are less critical than the ones of microgrids (see Table 1
). Thus, they should be adapted to provide:
An application layer to retrieve data from the system. In conventional DHTs, the nodes store the keys to locate information of the key it is responsible for. Thus, each node should establish its own methods to retrieve this information. To achieve this objective a communication infrastructure based on a stack protocol with an application layer has to be developed according to the goals of the microgrid.
A communication system able to achieve efficient and scalable routing for enhancing network performance, resulting in a highly distributed communication system, with low network traffic and latencies. Conventional DHTs do not consider locality issues. However, locality lets one create clusters with similar interests, reducing the network overhead and latencies.
Therefore, conventional DHTs need to be adapted to the specific case of microgrids. The proposed solutions are broadly explained in Section 5
and Section 6
5. Proposed Locality-Routing Algorithm
A missing feature in conventional DHTs is the ability to perform complex queries, such as a lookup operations containing regular expressions. One solution to the above problem is to perform the broadcasting in all the nodes (Full Flooding) or only in some of them, which are a part of the DHT (Limited Flooding). These solutions are currently used to many peer to peer applications.
In Reference [78
] a structured flooding algorithm is applied to the Chord architecture. In this flooding scheme, Full-Flooding (Full_F), when a node receives a lookup query, it forwards it to all the nodes of its finger table and these nodes recursively retransmit the lookup query to all their respective nodes. The flooding stops after TTL = 4 hops. In this scheme the lookup message will visit each node at least one time to obtain data retrieval. However, in this model locality is not considered.
Typically, different criteria can be adopted to provide locality by creating logical groups that form communities in order to achieve common objectives and goals [83
]. This approach is common in limited flooding solutions. With the objective to compare the performance of the flooding algorithms with the one of the proposed solutions, we have adapted the conventional flooding algorithms, by adding the application layer and some specific features of microgrids.
In the case of microgrids, these logical groups are often categorized based on their nodes functionality (loads, generators, storage system, etc.) so that the exchange of messages outside a certain community is not taken into account and thus doesn’t consume communication resources [19
]. Therefore, the basic idea of the proposed algorithm is to add network locality to DHTs based on the type of DERs connected to the microgrid and also enabling data retrieval. In this way, to provide locality aware in DHTs systems, the nodes’ finger tables of the traditional Chord model have been modified in order to embed DER functionality and keep the neighborhood of locality in the ID space (Chord space-key is not considered). In this scheme, when a node joins the network, it first identifies the kind of DER that has been associated to it, which may be a critical load (CL), a semi-critical load (SCL), a non-critical load (NCL), a distributed generator (DG) or a storage system (SS). After that, Limited Flooding (Limited_F) is applied. In Limited Flooding a node initiates the search for a specific functionality of the DER node by sending the query to all its neighbors that have the searched prefix. When a node receives the query message, it continues the flooding search until the message time to live (TTL) threshold (in this case TTL = 4). Therefore the lookup message will visit each searched kind of node at least one time to achieve data retrieval.
However, with the main idea of reducing network latencies and unnecessary network traffic, besides the proposed locality finger table, a new Locality-Routing Algorithm (Proposed_LRA) has been developed. The locality-routing algorithm helps know the number of devices and their type (the searched prefix depends on the kind of device: CL, SS, DG, etc.) that are connected to the microgrid to provide an efficient management of the microgrid resources.
shows the flowchart of the LRA algorithm. When the lookup process of the peer-to-peer nodes starts, two threads are generated, either for ingoing or outgoing lookups petitions. The outgoing lookup process is started by a node of the network in order to find the searched prefix or kind of device. For that, a lookup message must be created. The searching message has three main fields: (i) IP_node_source that contains the IP of the node which starts the lookup process; (ii) DER prefix that contains the searched prefix nodes and (iii) Match_Node_List, which is generated as the message progresses in the network and contains a list with the matched nodes. To conform the Match_Node_List field the Match_Search_Node Process starts and therefore the neighboring LRA finger table is going scrolled down. When the prefixes of the nodes in the LRA table match with the searched prefix, the node is added to the Node_Match_List and its IP is added to the Nodes_to_Send_List (only if the node number and its IP were not before in these lists). In addition, it is possible that the node does not contact any neighbor containing the searched prefix in its LRA finger table. Therefore, the successor node must be verified to avoid a break in the searching process.
Once all the searched nodes are found, the searching message is encoded and it will be sent to all the matched nodes. In this way, when an ingoing lookup process is started at the receiving node, it will be able to check the node match list and add its own new matched nodes (from the data stored in the received Match_Node_List) before to send a new query, to avoid sending redundant searching messages. The searching process continues until the source’s node is reached.
6. Proposal of a Layered Communication Architecture and an Application Protocol for MG’s
The efficient integration of DERs in microgrids needs the ability to monitor all units and provide real time control. According to that, the P2P control architecture introduced in Section 2
seems to be a good infrastructure for the exchange of data and control actions among the DERs of microgrids. To create an operational peer able to run in the communication network of a smart microgrid, several software layers need to be built [57
]. The proposed communication architecture is based on a structured decentralized peer-to-peer overlay network and it is depicted in Figure 6
This solution applies the P2P overlay principles to the field of energy control and monitoring of DERs. Therefore, the proposed communication architecture should be implemented on each one of the nodes where an electric module (such as inverters, transformers, switches and so on) is connected so each node is interconnected with their electric modules and with the communications network. The development of the different layers for implementing the proposed overlay P2P service is described below.
6.1. Network Layer
The network protocols used for this architecture are based on TCP/IP suite model. To improve the smart capacities of a microgrid, it is mandatory the ability to connect a large number of devices and handle a large volume of data in real time, which must be quickly updated to make decisions in the least possible time. Therefore, for the transfer of data the following are needed: (i) a large bandwidth; (ii) high data rate and (iii) a network layer with fast channel response. To accomplish these requirements, TCP/IP is preferred for interconnecting power systems devices, because it provides high connectivity, high bandwidth and quick response [86
]. The interconnection network ensures reliable and secure communication between components by employing an internet protocol (IP) and data traffic remains secure and safe because the transport communications protocol (TCP) is connection-oriented. In addition the IEC 61850 via Ethernet standard suggests the use of TCP/IP for efficient communications among microgrid components [87
6.2. P2P Layer
The proposed P2P layer defines two levels or software layers for implementing overlay-P2P services (see Figure 6
): On one hand, the DHT-Chord layer provides P2P networking services to achieve the following tasks: (i) resource and network discovery; (ii) session establishment; (iii) routing; (iv) data transfer among nodes and (v) entry/exit management of nodes. To evaluate the network performance, the different algorithms previously developed will be implemented in this level. On the other hand, the second P2P layer is responsible for handling the peer resources information.
In the P2P overlay networking layer, an Application Programing Interface (API) has been developed to run the node functions (i) to (iv) that have been listed in the previous paragraph. In this layer, each peer maintains the connection with their neighbor peers. Moreover, this layer includes neighbor discovery and it has a bootstrap engine for session establishment. For joining and leaving the network there is a join/leave protocol. The overlay networking layer also provides the mechanisms for routing data packets across a large network. In addition, Chord runs a special stabilization algorithm, called stabilize, which is periodically executed by each node every 60 ms to keep the list of predecessors and successors and to confirm that the ring integrity has not been corrupted. Therefore, if a certain node n fails, the nodes whose finger table includes n are responsible for finding the n’s successor. It must be avoided that the failure of n provokes a disruption of the queries that are in process while the system is re-stabilizing. To achieve that, each peer maintains a successor list of its r nearest successor’s node. In this way, when a certain node m notices that its successor has failed, it replaces the failed node by the first alive entry from its successor list. The queries will continue while the algorithm is correcting the finger table entries and the successor list. This allows not significantly increasing the end to end latencies if a node fails.
Besides, the DHT-Data layer establishes how the peers self-organize its resources and also the data management. It is worth noting that the DHT-data layer includes databases and uses the DHT-Chord one for storage of data. The Data layer for each node is defined as described below. On the one hand, peers should be able to react by sending messages in certain situations, for instance, in faulty conditions, when the operative power limits are reached, etc. On the other hand, peers should be continuously providing information about its actual status and also the one of its neighbors. Thus, the data base of each node is composed by two parts, named static and dynamic sides. The static side provides the description of the node itself (kind of DER: CL, SCL, NCL, DG…), the IP address, the Global Unique Identifier (GUID) and the definition of its power limits. The dynamic side periodically collects, from itself and from its neighboring peers, the main power parameters for knowing the global status of the microgrid.
6.3. Application Layer
The application layer is responsible for the processing and data analysis, as well as event management and supervision of the microgrid status. In addition, a specific microgrid communication protocol has been developed and implemented at the application layer for transmitting data among peers. Figure 7
shows an overview of the implemented Microgrid communication packet structure. The message header contains the type of message (ToM), the id device address, and data parameters and values. As Figure 7
shows, there are five ToMs that each peer of the network can send to the others. The function and the size of each ToM are also described.
shows four different architectures that can be used to implement the proposed algorithms. Depending on the used architecture, the performance of the network varies. The four evaluated architectures are: (i) Full-flooding-Architecture (Full_F*); (ii) Limited-flooding-Architecture (Limited_F*); (iii) Simple Query-Response (simple QR) and (iv) Embedded Query-Response (Embedded QR).
In the approaches shown in Figure 8
a–c, any peer of the network can launch the primitive QUERY about a specific kind of device to know their operational parameters. The difference between a normal QUERY and a primitive QUERY is that the second one is directly and only sent to the matched nodes. These nodes reply to the source node without transmitting the query through the ring. When the query is launched, the lookup process starts (according to the algorithm executed) and find the nodes of the network that are of the queried type. Then, the resources management is performed. Once the node receives the responses from the different peers, the data exchanges using periodic request-responses queries are executed. These queries occur between queried nodes through the application layer. In Figure 8
d the application layer is embedded into the P2P layer, so that both data exchanges and lookup process are launched simultaneously.
To better understand the differences between simple and embedded query response architectures, Figure 9
illustrates the sequence diagram of both architectures. Flooding architectures have not been considered because they are not based on the chosen DHT protocol.
As it can be seen in Figure 9
a, a peer sends out a lookup message to the LRA finger table neighbors that match the searched prefix. Thus the neighbor peers propagate the lookup message among the matched nodes that are listed in their LRA finger tables. The lookup message is passed around the ring following this procedure. When the lookup message reaches the source node, it has the information about all the matched nodes. Once it has this information, it is able to send the primitive query directly to the matched nodes without passing through other peers. Figure 9
b, shows that the primitive query message is embedded in the lookup message. As a result, both the number of messages that need to be sent and the latency of the system are significantly reduced. Figure 10
shows the used pseudo-code to embed the application layer into the DHT-P2P one.
shows how the application layer has been implemented in the P2P layer. The Manager
module manages the overall operations of each node in the P2P network. The peer is initialized by providing a port to listen for incoming connections (serverport), a host address (serverhost) and a node identifier (node). The handlers
is a class that contains the different peers’ functions. Using threads the mainloop
of a peer begins up a socket that listens for incoming connections from other peers. When an incoming connection is accepted, the server will have a new socket object used to send and receive data on the connection. Then, the main loop calls a separate method to handle communication with this connection in a new thread. When the peer wants to lookup matched nodes, it set the parameters for query (params
) and conform the message through invoking the application layer
module. The message is passing by reference to the lookup
function which allow for embedding the application layer into the DHT layer. The lookup
function method uses the LRA routing function to decide where to send the message.
It is worth noting that the proposed communication architecture should be implemented in each one of the nodes where an electric module (such as inverters, transformers, switches and so on) is connected. Therefore, each node is interconnected with the electric modules and also with the communication infrastructure through embedded single board computers (SBC). Moreover, the proposed communication system is adaptable to any kind of microgrid with Ethernet connectivity. In this case, the experimental evaluation by means of virtual machines (VM) has been performed by taking into account that the proposed solutions will be applied in the future to a hybrid AC-DC microgrid [89
] that is currently under deployment in our laboratory, as Figure 11
shows. The nomenclature used in this figure has been detailed in Table 2
7. Experimental Results
An experimental setup has been built to evaluate both the performance and behavior of the different lookup algorithms and the communication architecture for control purposes. To achieve this objective, a server with a hypervisor (VMware ESXi) (VMware Inc, Palo Alto, CA, USA) has been used to run 25 virtual machines. Each VM runs Ubuntu 14.04. The VMs are connected to the LAN through a virtual switch limited to 100 Mbps.
The resources of each VM have been limited to 512 MB of RAM and 200 MHz of CPU. This configuration agrees with the one of the embedded systems that we have initially chosen to migrate from virtual to physical machines, so the experimental results that are presented are realistic and valuable to preliminarily validate the proposed solutions. The physical platform that we will finally use is based on the UDOO X86 (UDOO, Burlington, MA, USA), whose computational features are higher than the initially pre-defined for the virtual machines (2 GB of RAM and CPU working at 2 GHz). Therefore, a superior performance can be expected by means of this platform.
7.1. Algorithms Comparison
The performance of the different lookup algorithms has been studied by measuring the number of lookup messages and the lookup latency. Figure 12
presents the total number of lookup messages that have been sent from a source node (which initiates the search) and retransmitted by the intermediate nodes. It may be seen that the number of messages that are required to perform the query process increases as the number of nodes does. However the rate of increase with the flooding algorithms is more abrupt than with LRA. The reason is that flooding algorithms do not use a method to know if the queried peer has been visited before. The proposed Locality-Routing Algorithm uses a routing algorithm that allows knowing what target peers have been previously visited and avoid to send redundant messages, so that the number of messages required is significantly reduced.
shows the evolution of the lookup latency as the number of peers increases. It has been verified that flooding algorithms need much more time to perform lookups than the proposed algorithm. Two reasons justify this result; on one hand, only the target’s peers are visited with the proposed algorithm, so that the number of peers to perform the lookup process is lower. On the other hand, each peer is visited only one time, avoiding redundant visits.
7.2. Communication Architectures Comparison
The performance of the proposed communication architectures has been studied. The evaluated scenario reproduces data exchanges between matched peers using request-response at 100% of the communications load, i.e., the transmitted messages have the Maximum TCP Segment Size (1500 bytes). The criteria for performance evaluation are the following:
Total number of messages exchange: this refers to the total number of messages that have to be exchanged among all peers both for lookup of the match peers and for sharing information among them.
End-to-end Latency: it contains the information obtained from the total nodes in the network regarding the lookup latency and the processing latency.
In both cases, the number of matched nodes has been taken to be equal to 60% of the total nodes in the network, i.e., for 25 peers, the number of matched nodes will be 15. Figure 14
shows that the number of total messages exchanged with the embedded QR approach is lower than with simple QR even if the same locality-routing algorithm is used. It is worthwhile to point out that simple QR needs an additional application layer to recollect data from the match nodes, while embedded QR doesn’t need this feature because the nodes respond by sending their operational parameters in the same lookup process.
shows that end-to-end delay in embedded QR architecture is lower than the query-response approach because the total delay is mainly produced by the processing delay. The lookup delay can be considered insignificant when the application layer is embedded in the P2P routing layer. The total end to end delay for 25 nodes with embedded QR is around 500 ms while for simple QR is around 650 ms because its lookup latency is around 200 ms (see Figure 13
). Thus, the average processing delay per node in embedded QR architecture can be estimated around 20 ms. This means that the average latency to process a TCP packet with maximum segment size in an embedded query-response design (round-trip time delay) is around 20 ms, which meet the network microgrid requirements that were outlined in Table 1
7.3. Network Performance Analysis
In the previous sections it has been demonstrated that the embedded query-respond is the most efficient communication architecture for control and monitoring of smart microgrids, as well as the latency network requirements are fulfilled. In this section, the network usage and the bandwidth requirements are evaluated. With this goal, periodic query-response queries tests have been carried out, where information is transmitted from each peer at a frequency of 1 Hz and 100% of TCP load.
and Figure 17
show the network usage generated by peers in the network and the bandwidth utilization (per peer), respectively. The total traffic generated by 25 peers in Figure 16
is around 400 kBps. The traffic is much lower than the bandwidth provided by the Ethernet installation (Ethernet’s 100 BaseT twisted pair cable (UTP)), which would possibility the execution of a large number of nodes. Otherwise, the average bandwidth per peer (Figure 17
) is around 1200 bytes/s inbound and 1600 bytes/s outbound. These results show that both network usage and average bandwidth meets the distributed control microgrid requirements (Table 1