Performance of Communication Network for Monitoring Utility Scale Photovoltaic Power Plants

: The grid integration of large scale photovoltaic (PV) power plants represents many challenging tasks for system stability, reliability and power quality due to the intermittent nature of solar radiation and the site accessibility issues where most PV power plants are located over a wide area. In order to enable real-time monitoring and control of large scale PV power plants, reliable two-way communications with low latency are required which provide accurate information for the electrical and environmental parameters as well as enabling the system operator to evaluate the overall performance and identify any abnormal conditions and faults. This work aims to design a communication network architecture for the remote monitoring of large-scale PV power plants based on the IEC 61850 Standard. The proposed architecture consists of three layers: the PV power system layer, the communication network layer, and the application layer. The PV power system layer consists of solar arrays, inverters, feeders, buses, a substation, and a control center. Monitoring parameters are classiﬁed into di ﬀ erent categories including electrical measurements, status information, and meteorological data. This work considers the future plan of PV power plants in Saudi Arabia. In order to evaluate the performance of the communication network for local and remote monitoring, the OPNET Modeler is used for network modeling and simulation, and critical parameters such as network topology, link capacity, and latency are investigated and discussed. This work contributes to the design of reliable monitoring and communication of large-scale PV power plants.


Introduction
Saudi Arabia is moving forward toward increasing the grid integration of large-scale photovoltaic (PV) power plants, supported by great opportunities of having long sunshine hours and high solar radiation intensity [1]. Different projects have been scheduled to be constructed, such as Qurrayat (200 MW), Alfaisalia (600 MW), Rabigh (300 MW), Jeddah (300 MW), Saad (300 MW), etc. [2]. Considering the previous research work related to the monitoring system of PV systems, as given in Table 1, there is a knowledge gap regarding the performance of the communication infrastructures associated with monitoring large-scale PV power plants with respect to network topology, network traffic and application definition, which have not received much attention in the literature. This work aims to study the performance of the communication infrastructure associated with remote monitoring and control of large-scale PV power plants. The major elements of a PV power plant are PV arrays, PV inverters, feeders, buses, meteorological masts, substations, and the control center. The communication infrastructure comprises measurement devices (sensor nodes, meters and intelligent electronic devices) and networking components (network switches, cables, and servers). Monitoring parameters are classified into different types: environmental information, status information and electric information based on IEC 61724 standard. In order to design the communication network, many factors should be determined, including the number and types of sensor nodes and the generated amount of data. Network topology, link capacity, and latency are critical parameters investigated and discussed. The main contributions of this research are as follows: • Design a communication network architecture for remote monitoring of large scale PV power plants based on the IEC 61850 Standard. The proposed architecture consists of three layers: the PV power system layer, the communication network layer, and the application layer. • Classify monitoring parameters into different types: environmental information, status information and electric information based on IEC 61724 standard. • Propose a communication model for the PV power system and define data types, data size and the number of sensor nodes. • Build communication network models for the PV system using OPNET Modeler and evaluate the network performance with respect to end-to-end delay for a single PV power plant and a group of PV power plants from different regions in Saudi Arabia. Ref.

Sensors Data Transmission
Storage and Analysis [3] No Yes No RS-485 → Between inverter and data logger Ethernet, WiFi, ZigBee → local monitoring RS-232 and USB → On-site configuration Overview of communication systems for grid integration of PV power plants [6] No No Yes Application of machine learning and computer vision for improving the reliability of PV arrays [7] Yes Yes No RF data transmission between sensor nodes (for electric data and meteorological data) and the gateway Low cost wireless monitoring system for PV module [8] Yes

PV Power System Layer
The basic components of a typical large-scale PV power plant are PV panels, PV inverters, transformers and protection devices [17]. The PV panels (a number of solar cells connected in series and encapsulated with a special frame) convert solar energy into electricity. PV panels are arranged in groups and are connected together to form module strings. If there is a fault in any single module or shading, the output voltage of PV panels will be affected, which may degrade the system output [18]. The outputs from the PV panels are connected to PV inverters. The PV inverters are electronic devices used to allow the conversion from DC to AC. This can be done through one stage (DC/AC) or two stages (DC/DC, DC/AC). Transformers are used to step up the output voltage from the PV inverters and to connect the PV power plant to the grid. Figure 1 shows different configurations for the interconnection between PV panels and inverters such as central, string and multi-string [4,5]. The selection among different configurations is based on the cost, reliability, and power quality. In the central configuration, all the PV panels are connected to one inverter. The PV panels are clustered into PV arrays. Each PV array consists of many PV strings connected in parallel and each string consists of many PV panels connected in series. In the string configuration, each string is connected to one inverter. In the case of multi-string configuration, the PV strings are connected to a DC/DC converter and each group of DC/DC converters are connected one DC/AC inverter.  The internal topology of a PV power plant represents the electrical connections among PV panels, PV inverters and transformers. Different topologies and configurations could be considered based on output power, cost, and reliability such as central topology, string topology and multi-string topology. There are different configurations for connecting the internal DC collection grid and the AC collection grid, such as radial, star, and ring [4,5].

•
Radial topology considers a group of PV generators connected to one feeder. This topology is simple and cheap but has low reliability. In the case of a fault in the first generator connected to the feeder, all the string will be disconnected.

•
Ring topology is based on the radial topology configuration but adds an extra feeder to the string from the other side. This enables connection to the PV generators even in the case of a PV generator fault. However, this configuration brings more complexity and cost for the installations.

•
Star topology offers higher reliability compared with radial and ring. Each PV generator is connected to the main controller through a feeder connection. The connections between PV generators and feeders increase the total cost of the star topology configuration.

Data Communication of PV Power Plants
In PV power plants, supervisory control and data acquisition (SCADA) systems play an important role in the remote monitoring and control of field devices (sensors, smart meters, remote terminal units (RTUs), intelligent electric devices (IEDs), etc.) [19,20]. A typical SCADA system consists of data acquisition units, RTUs, communication networks, and system servers. Data acquisition units measure and collect monitoring parameters such as voltage, current, temperature, irradiance, etc., and then transmit it through the communication infrastructure to the control center. As the control center represents the core of the SCADA system, there are several application servers, such as front-end servers, historian servers, web servers and human-machine interfaces, where monitoring data are analyzed and visualized. The front-end servers are responsible for collecting data from the field devices, while the historian servers are used for data storage. Figure 2 shows the communication network for a PV monitoring system. Each local control center is dedicated to the monitoring and control of a PV power plant. All control centers are connected to a wide area network via routers. The internal topology of a PV power plant represents the electrical connections among PV panels, PV inverters and transformers. Different topologies and configurations could be considered based on output power, cost, and reliability such as central topology, string topology and multi-string topology. There are different configurations for connecting the internal DC collection grid and the AC collection grid, such as radial, star, and ring [4,5].

•
Radial topology considers a group of PV generators connected to one feeder. This topology is simple and cheap but has low reliability. In the case of a fault in the first generator connected to the feeder, all the string will be disconnected.

•
Ring topology is based on the radial topology configuration but adds an extra feeder to the string from the other side. This enables connection to the PV generators even in the case of a PV generator fault. However, this configuration brings more complexity and cost for the installations.

•
Star topology offers higher reliability compared with radial and ring. Each PV generator is connected to the main controller through a feeder connection. The connections between PV generators and feeders increase the total cost of the star topology configuration.

Data Communication of PV Power Plants
In PV power plants, supervisory control and data acquisition (SCADA) systems play an important role in the remote monitoring and control of field devices (sensors, smart meters, remote terminal units (RTUs), intelligent electric devices (IEDs), etc.) [19,20]. A typical SCADA system consists of data acquisition units, RTUs, communication networks, and system servers. Data acquisition units measure and collect monitoring parameters such as voltage, current, temperature, irradiance, etc., and then transmit it through the communication infrastructure to the control center. As the control center represents the core of the SCADA system, there are several application servers, such as front-end servers, historian servers, web servers and human-machine interfaces, where monitoring data are analyzed and visualized. The front-end servers are responsible for collecting data from the field devices, while the historian servers are used for data storage. Figure 2 shows the communication network for a PV monitoring system. Each local control center is dedicated to the monitoring and control of a PV power plant. All control centers are connected to a wide area network via routers.

Communication Network Architecture for Large Scale Photovoltaic Power Plants
The proposed communication network architecture for large-scale PV power plant consists of three layers: the PV power system layer, the communication network layer and the application layer, as shown in Figure 3.

PV Power System Layer
The PV power system layer consists of all physical devices including PV modules, junction boxes, circuit breakers, protection devices, PV inverters, power cables, grid connection, transformers, the substation, etc. PV modules are connected in series to form a module string. Groups of strings are connected together in a string combined box. The string combiner boxes are connected to a power

Communication Network Architecture for Large Scale Photovoltaic Power Plants
The proposed communication network architecture for large-scale PV power plant consists of three layers: the PV power system layer, the communication network layer and the application layer, as shown in Figure 3.

Communication Network Architecture for Large Scale Photovoltaic Power Plants
The proposed communication network architecture for large-scale PV power plant consists of three layers: the PV power system layer, the communication network layer and the application layer, as shown in Figure 3.

PV Power System Layer
The PV power system layer consists of all physical devices including PV modules, junction boxes, circuit breakers, protection devices, PV inverters, power cables, grid connection, transformers, the substation, etc. PV modules are connected in series to form a module string. Groups of strings are connected together in a string combined box. The string combiner boxes are connected to a power

PV Power System Layer
The PV power system layer consists of all physical devices including PV modules, junction boxes, circuit breakers, protection devices, PV inverters, power cables, grid connection, transformers, the substation, etc. PV modules are connected in series to form a module string. Groups of strings are connected together in a string combined box. The string combiner boxes are connected to a power Energies 2020, 13, 5527 7 of 17 condition unit (PCU), which plays an important role in the PV power plant as it inverts the output DC voltage from PV panels to AC and is connected to a transformer based on PV power plant topology.

Communication Network Layer
The communication network layer enables two-way communications between PV power plant subsystems and the local control center. It consists of various communication devices connected with communication links. There are different types of sensor nodes and measurement devices interfacing with the power system layer. For example, pyranometers are used for measuring solar radiation, while anemometers are used for measuring wind speed and wind direction. Other measurements include modules temperature, air humidity, ambient temperature, atmospheric pressure and rain. Monitoring data from the PV subsystem and meteorological parameters are transmitted to the control center via the communication network layer. The communication network layer in PV power plant can be divided into subnetworks: the PV array area network (AAN), PV farm area network (FAN) and control area network (CAN). Each sub-network has different requirements in view of network configuration, coverage range and data rate. The main applications are given below: • SCADA system: Used for monitoring the performance of the PV power plant for documentation and analysis. This includes monitoring DC and AC parameters (voltage, current and power), PV modules, total generation from panels.

•
Meteorological system: Used for collecting meteorological information such as solar irradiance, ambient temperature, humidity and wind speed. • Protection and control system: Used for PV power plant protection. It interconnects among all protection and control IEDs in a PV power plant, which are responsible for safety and protection of the utility's hardware components.

•
Metering the grid interface system: Used for measuring the electrical parameters at the point of interconnection. This could be done by monitoring real and reactive power, voltage and connection status at the point of interconnection in order to ensure the grid quality assurance.
The IEE 1646 Standard defines different requirements for data transmission for different applications [21]. Table 2 shows the requirements for data delivery time based on IEEE 1646 Standard. The requirements of data delivery for the protection information required strict delay compared with monitoring and control.

Application Layer
The control center is responsible for the monitoring and controlling the operation the PV power plant including PV panels, inverters, transformers, grid connection point, etc. Data collected from sensor nodes, measurement devices are processed and stored in system servers, and appropriate control actions are taken accordingly.

Modeling Photovoltaic System Based on IEC 61850-7-420 Standard
In a typical PV power plant, there are many factors that affect the plant operation, such as faults, cell damage, shading, dust, degradation, etc. [22]. Most of these factors are hidden and difficult to identify in a large-scale PV power plant. Supervisory control and data acquisition (SCADA) system represent an essential element for remote monitoring and control of PV power plant. It ensures a reliable and a safe operation for PV power plant subsystems as it enables the control center operator to analyze the performance, detect operation issues and identify faults during abnormal conditions. The PV power plant can be divided into different levels: the module level, string/array level, inverter level and grid level.

•
The module level monitors the voltages and current in each module. The status of each PV module can be determined and faults in any module can be identified.

•
The string/array level monitors the voltages and current in each string/array. Faults in any string/array can be identified.

•
The inverter level monitors the array output power and the performance of the section connected to that inverter. In the case of a malfunction, both the array and the inverter can be identified.

•
The grid level monitors the total performance of the PV power plant.
The IEC 61724 standard defines the general guideline for monitoring the overall performance of the PV system including in-plane irradiance, array output and energy storage system [23]. The required parameters to be measured are shown in Table 3, where all parameters should be continuously measured. The amount of data generated from sensor nodes are calculated based on the sampling rate, the sample size and the number of channels. The data rate was calculated given the assumption that each data sample is represented with 2 bytes (16 bit) [24]. IEC 61850-7-420 standard defines the general data model for distributed energy resource (DER) systems. The standard describes the information model of the photovoltaic system including PV panels, conversion system, circuit breakers, protection devices, and metering devices. Figure 4 shows the architecture model of the photovoltaic system based on IEC 61850-7-420 standard [25]. A photovoltaic array controller (DPVC) and a PV tracking controller (DTRC) represent the logical nodes of the PV Energies 2020, 13, 5527 9 of 17 panels, as shown in Equation (1). In order to maximize the output power of the PV system, the output voltage and current are adjusted through the maximum power point tracking.
The main operation of the switchgear is to monitor and control the operation of circuit breakers and switches. The logical nodes of the DC switch (SWLN) is represented with a circuit breaker (CSWI) and a switch controller (XSWI) while the circuit breaker (CBLN) is represented with a circuit breaker (CSWI) and a circuit switch (XCBR), as shown in Equations (2) and (3).
The rectifier (ZRCT), inverter (ZINV) and DC metering (MMDC) represent the logical nodes of the inverter (INVLN), as shown Equation (4). The electric measurements are represented with the logical node metering (MMTR). The logical node MMXU represents the metering of the AC current while MMDC represents the DC current. The logical node (MMET) represent the meteorological measurements, which includes monitoring parameters of meteorological information such as solar irradiation and ambient temperature. The list of logical nodes associated with the photovoltaic system is given in Table 4. There are three types of IED: circuit breaker CB-IED, merging unit MU-IED, and protection and control P&C IED. The CB-IED monitors the status of the circuit breaker and sends status information for the P&C IED. The MU-IED measures the current and voltage signals and transmits it to the P&C  (2) and (3).

The main operation of the switchgear is to monitor and control the operation of circuit breakers and switches. The logical nodes of the DC switch (SW LN ) is represented with a circuit breaker (CSWI) and a switch controller (XSWI) while the circuit breaker (CB LN ) is represented with a circuit breaker (CSWI) and a circuit switch (XCBR), as shown in Equations
The rectifier (ZRCT), inverter (ZINV) and DC metering (MMDC) represent the logical nodes of the inverter (INV LN ), as shown Equation (4).
The electric measurements are represented with the logical node metering (MMTR). The logical node MMXU represents the metering of the AC current while MMDC represents the DC current. The logical node (MMET) represent the meteorological measurements, which includes monitoring parameters of meteorological information such as solar irradiation and ambient temperature. The list of logical nodes associated with the photovoltaic system is given in Table 4.
There are three types of IED: circuit breaker CB-IED, merging unit MU-IED, and protection and control P&C IED. The CB-IED monitors the status of the circuit breaker and sends status information for the P&C IED. The MU-IED measures the current and voltage signals and transmits it to the P&C IED. The P&C IED manages and supervises the operation of all IEDs. Based on IEC 61850 standard, the P&C system can be divided into three levels: the process level, bay level and station level.

Single PV Power Plant Scenario
This section explains the simulation models that were developed to evaluate the performance of the communication network of a PV power plant. The simulation models were built using OPNET Modeler based on a real project (Layla solar plant (10 MW), phase 1) in Saudi Arabia. The developed model consists of 80 PV strings, two feeders, two power conditioning units (PCU), one bus, one transformer, two meteorological masts and a local control center, as shown in Figure 5. The main components of the PV power plant and IEDs are summarized in Tables 5 and 6, respectively. Energies 2020, 13, x FOR PEER REVIEW 10 of 17 IED. The P&C IED manages and supervises the operation of all IEDs. Based on IEC 61850 standard, the P&C system can be divided into three levels: the process level, bay level and station level.

Single PV Power Plant Scenario
This section explains the simulation models that were developed to evaluate the performance of the communication network of a PV power plant. The simulation models were built using OPNET Modeler based on a real project (Layla solar plant (10 MW), phase 1) in Saudi Arabia. The developed model consists of 80 PV strings, two feeders, two power conditioning units (PCU), one bus, one transformer, two meteorological masts and a local control center, as shown in Figure 5. The main components of the PV power plant and IEDs are summarized in Tables 5 and 6, respectively.    In a real PV power plant, independent communication networks are used to support different applications (sometimes from different service providers) where all subnetworks are connected together via a common backbone network. In this work, independent communication networks (independent communication links and switches) are considered to support different applications (PV arrays, protection and control and utility grid). Figure 6 shows the OPNET simulation model for the PV power plant. It consists of seven octagons representing four PV arrays, two meteorological masts and one control center. The configuration is based on the actual PV power plant dimensions. Two scenarios are considered: fixed mounted PV arrays (no tracker tilt angle nor tracker azimuth angle) and PV arrays with sun trackers (each PV sun tracking unit is equipped with a pyranometer). Two different link capacities are considered for local area communication (LAN): 10 BaseT and 100 BaseT. The traffic configuration, including the source and destination, is given in Table 7. The performance of the proposed communication network is evaluated for a switched Ethernet architecture in view of bandwidth and end-to-end delay. Tables 8  and 9 show the end-to-end delay for different scenarios of PV arrays with different channel capacities. Each PV array is modeled with an Ethernet LAN of 20 workstations in a switched topology. The results of latency are compared with different numbers of PV arrays. considered for local area communication (LAN): 10 BaseT and 100 BaseT. The traffic configuration, including the source and destination, is given in Table 7. The performance of the proposed communication network is evaluated for a switched Ethernet architecture in view of bandwidth and end-to-end delay. Tables 8 and 9 show the end-to-end delay for different scenarios of PV arrays with different channel capacities. Each PV array is modeled with an Ethernet LAN of 20 workstations in a switched topology. The results of latency are compared with different numbers of PV arrays.    In general, the architecture of the substation automation system consists of a process level, protection and control level, and station level. In this work, the substation automation system of the PV power plant is implemented using different IEDs. These different IEDs include PV generator, collector feeder, collector bus and transformer which are interconnected through a communication network. The model of the transformer IEDs consists of two CB IED, one MU IED and one P&C IED. The models of collector bus and collector feeders consist of one CB IED, one MU IED and one P&C IED. The communication network is configured as a star topology. A PCI server is used to represent the station level, while the bay level includes different protection and control IEDs. The data generated from IEDs are transmitted to the PCI server. Note that the protection and control functions require a fast response time compared with SCADA monitoring. Two scenarios are considered for the substation communication using LAN with 10 Mbps and 100 Mbps. The end-to-end delay for different protection and control IEDs are shown in Table 10.

PV Power Plants Scenario
This work considers the planned PV power plants in Saudi Arabia based on the Renewable Energy Project Development Office (REPDO) [26]. Table 11 shows a list of the planned solar power projects across the kingdom. The projects have been divided into three rounds: round 1 includes one project (Sakaka PV project, 300 MW), round 2 includes six projects (1.47 GW) and round 3 includes four projects (1.2 GW). Al-Faisaliah project is the largest among them, with a generation capacity of about 600 MW. Figure 7 shows the locations of PV power plants. The solar PV power plants are distributed among different regions, where all projects will be connected to the national electricity grid. Table 11. Summary of future PV power projects in Saudi Arabia [20].

Round Status Solar PV Projects Capacity (MW) Region
Round The proposed communication network model was developed for the wide monitoring of PV power plants. It consists of 12 subnetworks (each subnetwork is represented by an octagon) and a central control center. Each PV power plant is configured to transmit the monitoring data of the point of common coupling (PCC) including voltage, current and power to the central control center (CCC), where monitoring data are stored and used by other applications such as an energy management system. As the computer systems are the core part of the control center, the system must be sized to receive, process and store the necessary information. The main elements are front-end units (industrial servers), a historian server, operator consoles, a web server and a video wall [27]. The front-end units are responsible for receiving information from PV power plants. Additional front-end units are added as the number of PV power plants increases. Historical servers are used to store the information received from the front-end units. The decision regarding the type of historical database in the control center is based on the amount of information to be stored, the scan rate and the speed required in data access using reports. The new control centers are moving toward virtual machines with the target of decreasing the amount of equipment, reducing maintenance costs, and reducing energy consumption and wiring, etc.  The proposed communication network model was developed for the wide monitoring of PV power plants. It consists of 12 subnetworks (each subnetwork is represented by an octagon) and a central control center. Each PV power plant is configured to transmit the monitoring data of the point of common coupling (PCC) including voltage, current and power to the central control center (CCC), where monitoring data are stored and used by other applications such as an energy management system. As the computer systems are the core part of the control center, the system must be sized to receive, process and store the necessary information. The main elements are front-end units (industrial servers), a historian server, operator consoles, a web server and a video wall [27]. The front-end units are responsible for receiving information from PV power plants. Additional frontend units are added as the number of PV power plants increases. Historical servers are used to store the information received from the front-end units. The decision regarding the type of historical database in the control center is based on the amount of information to be stored, the scan rate and the speed required in data access using reports. The new control centers are moving toward virtual   Considering the measured parameters at the grid side, the amount of the received traffic for voltage, current and power are 12,288, 12,288 and 10 bytes/s, respectively. The total amounts of traffic received at the control center are 270,336, 135,168 and 220 bytes/s, for current, voltage, and power, respectively. All monitoring data are received successfully. Table 12 shows the end-to-end delay between PV power plants and the central control center using a link capacity of 44.736 Mbps. The packet discard ratio (PDR) of the IP cloud has been configured at 1%, which specifies the percentage of packets dropped (ratio of dropped packets to the total submitted packets to this cloud). In case of no PDR, the end-to-end delay was about 7.79 ms for Qurayyat PV power plant, while about 5.83 ms for Saad PV power plant. With the PDR of 1%, the end-to-end delay was about 7.93 ms for Qurayyat PV power plant, while about 5.90 ms for Saad PV power plant. The difference in end-to-end delay among different projects is due to the location of Considering the measured parameters at the grid side, the amount of the received traffic for voltage, current and power are 12,288, 12,288 and 10 bytes/s, respectively. The total amounts of traffic received at the control center are 270,336, 135,168 and 220 bytes/s, for current, voltage, and power, respectively. All monitoring data are received successfully. Table 12 shows the end-to-end delay between PV power plants and the central control center using a link capacity of 44.736 Mbps. The packet discard ratio (PDR) of the IP cloud has been configured at Energies 2020, 13, 5527 15 of 17 1%, which specifies the percentage of packets dropped (ratio of dropped packets to the total submitted packets to this cloud). In case of no PDR, the end-to-end delay was about 7.79 ms for Qurayyat PV power plant, while about 5.83 ms for Saad PV power plant. With the PDR of 1%, the end-to-end delay was about 7.93 ms for Qurayyat PV power plant, while about 5.90 ms for Saad PV power plant. The difference in end-to-end delay among different projects is due to the location of each PV power plant. Table 12. End-to-end delay between PV power plant and the central control center. PDR: packet discard ratio.

Conclusions
Reliable communication networks play an important role in supporting the grid integration of large scale PV power plants. The underlying communication infrastructures are responsible for real-time monitoring and have a direct impact on the performance of the PV power plants as well as the capability for meeting the target application requirements. This work proposed communication network models for the PV power system in order to evaluate future scenarios of PV power plants. Various scenarios have been considered in order to evaluate the performance of the communication network, including a single PV power plant and a group of PV power plants from different regions in Saudi Arabia. The monitoring measurements are communicated to a local control center in a single PV power plant scenario, and to a central control center in the multiple PV power plants scenario for further assessment. The proposed model considered 11 PV power plants which are geographically distributed in different regions and the proposed communication models have been evaluated with respect to end-to-end delay for different applications using OPNET Modeler. The simulation results in a single PV power plant scenario show that the maximum end-to-end delay for protection data was less than 4 ms considering a channel capacity of 100 Mbps. In the case of multiple PV power plants, the simulation results show that the background traffic and packet discard ratio have a direct impact on the end-to-end delay. Future works aim to evaluate the performance of the proposed communication network using different wireless communication technologies.