An intelligent multiagent system for autonomous microgrid operation

A microgrid is an eco-friendly power system because renewable sources such as solar and wind power are used as the main power sources. For this reason, much research, development, and demonstration projects have recently taken place in many countries. Operation is one of the important research topics for microgrids. For efficient and economical microgrid operation, a human operator is required as in other power systems, but it is difficult because there are some restrictions related to operation costs and privacy issues. To overcome the restriction, autonomous operation for microgrids is required. Recently, an intelligent agent system for autonomous microgrid operation has been studied as a potential solution. This paper proposes a multiagent system for autonomous microgrid operation. To build the multiagent system, the functionalities of agents, interactions among agents, and an effective agent protocol have been designed. The proposed system has been implemented by using an ADIPS/DASH framework as an agent platform. The intelligent multiagent system for microgrid operation based on the proposed scheme is tested to show the functionality and feasibility on a distributed environment through the Internet.


Introduction
Microgrid operation is an especially important research topic.An important function of microgrid operation is to maintain a constant frequency, such as 50 Hz or 60 Hz, as a critical requirement.It relates to the balance between power supply and power demand.A microgrid is operated in two modes: the grid-connected mode and the islanded mode.In the grid-connected mode, a microgrid can trade power with the upstream power grid to solve the power imbalance.On the other hand, when the power imbalance occurs in the islanded mode, the decrease of total output of distributed generators (DGs), or load-shedding, which is intentional load reduction, can be used to solve the power imbalance.In this paper, we focus on the grid-connected mode.
For efficient and economical microgrid operation, a human operator is required as in existing power systems.However, there are some restrictions relating to operation costs and privacy issues, because distributed devices or systems in the microgrid are too small-scale for hiring human operators, and microgrids can be located at residential homes.In order to address the restrictions, autonomous operation for microgrids is required.
In order to implement autonomous systems or functions, intelligent agent technologies can be a good solution.An intelligent agent (abbreviated as an agent) can sense external environmental changes, effectively make decisions based on designed purposes against the changes, and act autonomously according to the decision.A multiagent system is an autonomous system or society composed of multiple agents.In the multiagent system, agents can communicate using an agent communication language (ACL) and share knowledge for their cooperation [1][2][3].For these reasons, interests in multiagent systems for microgrid applications as a potential solution have been growing.
In this paper, a multiagent system is modeled and designed for autonomous microgrid operation in the grid-connected mode.The proposed multiagent system for autonomous microgrid operation is tested to show its feasibility and functionality.
The composition of this paper is as follows: related work is described in Section 2. Section 3 describes the concept of microgrid operation and adopted operation rules.Section 4 explains the configuration of the proposed multiagent system, interactions among agents, decision-making strategies for agents, and implementation.Section 5 shows the results of the feasibility and functionality test of the proposed multiagent system in a distributed environment.Finally, we summarize the outcomes in Section 6.

Related Work
As studies related to microgrid control, in [4,5], the hierarchical multiagent system based control strategy is proposed.Besides, a multiagent system based on the Zeus platform and the microgrid system based on Matlab are implemented to show the feasibility of the proposed control strategy in the grid-connected and the islanded modes.In [6], a multiagent control strategy for the islanded mode is proposed.In this work, a coordination of multiagent platform architecture including the coordination level agent, multiple area agents, and various component agents is used and the coordinated control strategy is adopted.The system is implemented to show the feasibility of the control strategy using the Java language.In [7], a decentralized control architecture based on a multiagent system for the autonomous operation of a microgrid with power electronic interfaces is presented.The proposed system is demonstrated on a system comprising four microsources using Matlab/Simulink.In [8], a hierarchical control scheme using a multiagent system for black start operation of a microgrid with power electronic interfaces is proposed.For this, five types of agents, such as the grid agent, the central agent, the generation agent, the load agent, and the breaker agent, are designed and their cooperated control for black start operation is demonstrated using Matlab/Simulink.In [9], the conceptual approach of a multiagent system is introduced for distributed energy resources (DERs) control in the microgrid.In this study, three types of agents, namely, the regional agent (RA), the local agent (LA), and the service agent (SA) are defined.Besides, two-layer control strategies are proposed to achieve local autonomy and global optimization respectively, operating in both the grid-connected mode and the island mode.In [10], a multi-agent based control architecture for microgrids are presented.The architecture includes a cooperative method to achieve user-defined objectives.In [11], a novel multi-microgrids distributed control oriented hierarchical and distributed multiagent system architecture is conceptually constructed and designed.This multiagent system is composed of the multi-microgrids management agent, the microgrid control agent, and the local agent.It uses CORBA technology as communication mode in the whole system.
As other topics, in [12], a general framework for microgrids control based on the multiagent system technology is proposed and a multiagent reinforcement learning algorithm to operate autonomously in the island mode is dealt with.In [13], a fully distributed multiagent based load restoration algorithm is proposed.In this work, a distributed algorithm for coefficient setting is proposed.In [14], a multiagent system for service restoration of the microgrid is proposed on the basis of the hierarchical control strategy.Accordingly, the special functions and control strategies of each agent are discussed.In [15], a hardware implementation of a completely decentralized multiagent system for capacity discovery in microgrids is presented.Performance in terms of time-to-convergence of the decentralized system using wireless communications is reported.To implement the algorithm, an Arduino microcontroller and ZigBee radio are used.
As similar studies related to microgrid operation, in [16], a scalable multiagent system for optimal operation of the islanded mode is presented.The multiagent system is designed such that it maximizes the power production of local distributed generators and minimizes the operational cost of microgrid subject to system constraints and units constraints.In [17], a multiagent based model for the hierarchical and distributed energy management of a microgrid is presented.Agents under three different levels are classified and a simulation platform for the energy management system of the microgrid (EMS-MG) is designed in terms of Client-Server and implemented under C++ Builder environment.
Through the literature survey, we can see that studies on multiagent system applications to microgrids have been dealt with in several areas and multiagent systems for microgrid control have been studied most.The operation problem deals with relatively longer periods than the control problem, such as a day.To deal with the operation problem, microgrid operation rules should be required.Since currently microgrid technology is not in the commercialization stage and it can be applied to various power market environments, general microgrid operation rules are not easy to find.Therefore, microgrid operation rules are assumed considering conventional operation rules for power grids in this paper.A multiagent system is designed based on the operation rules.To design the multiagent system, an agent set and interactions among agents are defined and the protocols, messages, and each agent are designed.

Microgrid
A microgrid is a small-scale power system composed of DGs such as solar power, wind power, or combined heat power (CHP) systems, distributed storage systems (DSs) such as a battery system or a flywheel system, and loads such as residential buildings, commercial buildings, public offices, or industrial compounds.A microgrid is connected to a utility grid (or an upstream power grid) though the point of common coupling (PCC).
Microgrid operation is assumed in this paper as follows.A microgrid is operated in two steps: planning and implementation, like in conventional power systems as shown in Figure 1.The Microgrid Operation and Control Center (MGOCC) establishes an operation plan for the next interval and implements the established operation plan during the next interval.The interval period depends on applied operation rules [18][19][20].According to the characteristics of competition in power markets, two kinds of power market environments for microgrid can be considered.One is a power market related to a power grid.The other is a power market for the inner-microgrid (abbreviated as the inner-market).In this paper, we consider both power market environments, but we focus more on the latter environment because it is closely related to microgrid operation.In the former environment, the power imbalance is solved briefly by power trading with an upstream power grid.

Operational Rules
We assume the following operation rules based on conventional operation rules for power grids:  From the upstream power grid, the MGOCC receives a price for selling power to the power grid (P S ) and a price for buying power from the power grid (P B ) at the beginning of an interval.P B is higher than P S like general market places. The MGOCC announces the prices to participants in the microgrid. DGs bid their supply amounts and prices to the MGOCC. Loads inform their load amounts to the MGOCC.
 DSs can select one among three actions: no action, charge, and discharge.In the case of charge, DSs inform load amounts to the MGOCC.In the case of discharge, DSs bid supply amounts and prices to the MGOCC. All bidding prices should be less than or equal to P B . The MGOCC selects final suppliers considering total power demand.The final suppliers are selected by the descending order of bidding prices. Final suppliers should supply agreed amounts during the next interval. In the case of supply shortage, the microgrid should buy power from the upstream power grid at P B . In the case of supply surplus, suppliers not selected as final suppliers or final suppliers having additional supply power can sell their power to the power grid directly at P S .
Figure 2 shows the above-mentioned operation rules.From the operation rules based on the power market environment, the electricity price of a microgrid is decided between P S and P B .This point gives an advantage to suppliers and consumers of the microgrid because suppliers can supply their power at a price greater than or equal to P S and consumers can use power at a price less than or equal to P B .In the case of DSs, they should forecast the power balance of the microgrid for effective decision-making for the DSs between charge and discharge, but this is difficult in practice.For simplicity, it is considered that the MGOCC computes a power balance after gathering the information of power supply/power demand from DGs/loads, and then informs the DSs of the power balance.

Configuration of the Multiagent System
To construct a multiagent system for microgrid operation, the following agent set (Ag) is defined: where Ag MGOCC is the MGOCC agent; AG L is a set of load agents (Ag L ); AG DG is a set of DG agents (Ag DG ); and AG DS is a set of DS agents (Ag DS ).In the system, Ag MGOCC interacts with all participants, i.e., Ag DG s, Ag DS s, and Ag L s as a manager for microgrid operation.

Design of Interaction among Agents
A communication protocol for the multiagent system has been designed based on the Contract Net Protocol (CNP).The CNP is a high-level protocol for communication and control in a distributed system [21].One salient feature of the CNP is a simple framework based on announcing a new task, bidding, and awarding a contract.The basic steps of the CNP are similar to contract steps in human society.It has been modified for effective applications in many areas [22][23][24].In order to implement the proposed multiagent system, we employ a modified CNP (MCNP).Figure 3 shows the workflow based on the MCNP designed for microgrid operation.We define messages for interactions among agents, such as call for proposal (cfp), propose-load, refuse, propose-supply, reject-proposal, accept-proposal, and report.Table 1 shows the details of the messages.Three tasks are also defined such as T1 (a task between Ag MGOCC and Ag L s), T2 (a task between Ag MGOCC and Ag DG s), and T3 (a task between Ag MGOCC and Ag DS s).Additionally, there are four deadlines:  D1: deadline for submitting proposal of Ag L s and Ag DG s in interval i (T1 & T2). D2: deadline for submitting proposal of Ag DS s in interval i (T3). D3: deadline for awarding contracts to final suppliers and consumers in interval i (T1 − T3). D4: deadline for submitting the report in interval i + 2 (T1 − T3).

Design of Agents
In our system, there are four types of agents: Ag MGOCC , Ag DG , Ag L , and Ag DS .First of all, we define the functionalities of Ag MGOCC as a managing agent of microgrid operation as follows:  Announcing a new task with trade prices with the power grid to Ag DG s and Ag L s.  Gathering information of supply and demand from Ag DG s and Ag L s.  Checking the power balance. Announcing a new task to Ag DS s with information of the power balance and the trade prices. Selecting final suppliers by the descending order of bidding prices. Announcing information of supply and demand to Ag DG s as the final suppliers and Ag L s.  Distributing operation results. Receiving reports from the final suppliers.
Figure 4 shows the pseudo codes to select final suppliers for Ag MGOCC .Once Ag MGOCC gathers information from AgDGs, AgLs, and AgDSs, it compares total amount of power supply (in the figure , we call it supply_sum) with total amount of load demanded (in the figure, we call it load_sum).There are three cases; when supply_sum is greater than load_sum, when supply_sum is less than load_sum, and when supply_sum is the same with load_sum.For the first case, Ag MGOCC selects the final suppliers among suppliers by their bidding prices in lines 11-21 of the figure.For the second case, Ag MGOCC buys the power from the power grid as much as supply shortage in lines 22-28.For the last case, Ag MGOCC selects all suppliers as final suppliers in lines 29-34.Next, the functionalities of Ag DG is defined as follows:  Proposing the amount of power supply and a bidding price to Ag MGOCC . Selling power to the upstream power grid directly in the case of a supplier who is not selected as a final supplier or who has additional power. Sending a report to Ag MGOCC after finishing the agreed contract.
The Ag DG takes charge of a DG or a group of DGs located in the same place and uses a bid function (F bid ) and a trade function (F trade ) for decision making on its bid to Ag MGOCC and trading with a power grid, respectively: where P cost is the production cost of a DG.
The Ag L takes charge of a consumption device or a group of consumption devices located in the same place.It is assumed that Ag L has the ability to forecast the amount of load for the next interval.It has been designed to implement the following tasks:  Waiting for a new task from AgMGOCC at the beginning of an interval. Informing AgMGOCC of the load amount.
Finally, Ag DS has two roles: supplier and consumer.As a supplier, Ag DS use F bid in Equation ( 3) for bidding on an amount of supply.Its bidding price is updated by using the arithmetical mean considering the charged amount and the electricity charge.As a consumer, Ag DS has been designed to charge its available amount fully, i.e. the difference between maximal capacity and the state of charge (SOC).

Implementation
In this paper, we implement the proposed multiagent system by using the Agent-based Architecture of Distributed Information Processing Systems (ADIPS)/Distributed Agent System based on Hybrid Architecture (DASH) framework as an agent platform [25][26][27].The main features of the ADIPS/DASH framework are summarized as follows:  The ADIPS/DASH is a repository-based agent framework which consists of the distributed agent workplace (abbreviated as the workplace) and the agent repository (abbreviated as the repository). An agent is designed and implemented to describe the agent's behavior knowledge for solving a cooperative problem with the agent's meta-knowledge for managing the agent in the repository. The behavior knowledge is represented as a set of rules using the rule-type knowledge representation format (in contrast to the meta-knowledge, which is described using the frame-type knowledge representation format). The ADIPS/DASH framework provides a wrapping mechanism to utilize external software modules such as Java programs as the procedural knowledge of the agent. Agents can communicate with different types of agents such as FIPA-compliant JADE agents by using ACL messages of the DASH agent.

Test Environment
In order to test the functionality and feasibility of the proposed multiagent system, the system is composed of Ag MGOCC , three Ag DG s, an Ag DS , and two Ag L s. Figure 5   Ag DG s and Ag DS use their production costs as bidding prices for simplicity.Table 2 shows power trade prices with the power grid (P S and P B ) and loads (L1 and L2) in 24 intervals.Figure 6 shows the assumed power trade prices in 24 intervals.

Results
Figure 7 shows the load amounts of L1, L2, and DS as the result of multiagent-based microgrid operation, where DS plays the role of a consumer by its demand for charge in intervals 1, 13, and 23. Figure 8 shows the power supplied by DG1, DG2, DG3, and DS as the result of microgrid operation, where DS plays the role of a supplier by discharge action in intervals 7, 8, and 18. Figure 9 shows power trades with the power grid, where selling power to the grid is conducted by DG3.

Evaluation
The experimental results shown in Figures 7-9 are summarized in Tables 3 and 4. Table 3 shows the power balance between power supply and power demand.By charging DS, the power supply of the DGs increases in intervals 1, 13, and 23.In addition, the discharge of DS plays a potential role in lowering the power price of the microgrid in intervals 7, 8, and 18.Table 4 shows the behavior of DG3 having the highest production cost among the DGs.Because of its highest production cost, the power amount selected from Ag MGOCC using the merit order is different according to the power balance of each interval.In the case of supply surplus, the amount of power it supplies to the microgrid is decreased.However, DG3 can sell its extra power to the power grid whenever the selling price of the power market is higher than its production cost.
From the results, we can find that DG3, having the highest production cost, can be restricted to supply power to the microgrid, because DS has lower production cost.For example, in interval 7, DG3 saves its generation cost by reducing the power generation and sells its extra power to the power grid, but that is not guaranteed.The situation is caused by the merit order of Ag MGOCC .This can be solved by giving higher priority than the DSs' bidding prices in practical application.The problem deciding the priority can be considered as a design parameter of the proposed multiagent system.

Conclusions
In this paper, a multiagent system for microgrid operation has been proposed.We have defined the functionalities of agents in the microgrid and interactions among agents.For effective interactions among agents, the MCNP has been designed.We have implemented the proposed system using the ADIPS/DASH framework.The implemented system has been tested on a distributed environment using the Internet to evaluate the functionality and feasibility for microgrid operation.
In this paper, we considered scenarios that selling prices are always less than buying prices.However, in some other scenarios, the selling prices are not necessarily always less than buying prices.Thus, as a future work, we plan to consider the various scenarios to provide the flexible microgrid operation.In addition, we plan to integrate control functions of the microgrid in the proposed system into a hardware-in-the loop simulation (HILS) and in a small-scale microgrid pilot in order to consider the whole microgrid operation.

Figure 5 .
Figure 5. Distributed environment based on four PCs for test.

Figure 7 .
Figure 7. Demand operated by the system.

Figure 8 .
Figure 8. Supply operated by the system.

Figure 9 .
Figure 9. Power trade operated by the system.

Table 1 .
Details of the defined messages.

Table 2 .
Power trade prices with the power grid and loads in 24 intervals.

Table 3 .
Power balance of microgrid in 24 intervals.