Automatic Inspection of Photovoltaic Power Plants Using Aerial Infrared Thermography: A Review

: In recent years, aerial infrared thermography (aIRT), as a cost-efﬁcient inspection method, has been demonstrated to be a reliable technique for failure detection in photovoltaic (PV) systems. This method aims to quickly perform a comprehensive monitoring of PV power plants, from the commissioning phase through its entire operational lifetime. This paper provides a review of reported methods in the literature for automating different tasks of the aIRT framework for PV system inspection. The related studies were reviewed for digital image processing (DIP), classiﬁcation and deep learning techniques. Most of these studies were focused on autonomous fault detection and classiﬁcation of PV plants using visual, IRT and aIRT images with accuracies up to 90%. On the other hand, only a few studies explored the automation of other parts of the procedure of aIRT, such as the optimal path planning, the orthomosaicking of the acquired images and the detection of soiling over the modules. Algorithms for the detection and segmentation of PV modules achieved a maximum F1 score (harmonic mean of precision and recall) of 98.4%. The accuracy, robustness and generalization of the developed algorithms are still the main issues of these studies, especially when dealing with more classes of faults and the inspection of large-scale PV plants. Therefore, the autonomous procedure and classiﬁcation task must still be explored to enhance the performance and applicability of the aIRT method. perform the ﬁnal classiﬁcation of the images, in the same way as a conventional neural network. In the end, the CNN output is the probability that the input image belongs to one of the classes for which the network was trained [40].


Introduction
As the world experiences a continuously growing share of photovoltaics (PVs) in the energy mix, increasing the performance and reliability of PV installations is of utmost importance. In this context, infrared thermography (IRT) has become a well-established and competitive fault detection method for the condition monitoring and maintenance of PV systems [1]. It provides reliability and accuracy in the detection of typical PV module faults such as bypassed or disconnected substrings, microcracks, soldering problems, shunted cells and disconnected modules. Another feature of this technique is the possible large-scale applicability, through the combination of IRT cameras with an unmanned aerial vehicle (UAV), configured for aerial infrared thermography (aIRT) [1,2].
The first description of the potential of using aIRT in the literature was given in 2012 by Denio [3]. This was followed by the publication of the results of an experimental setup that inspected 60 different PV plants of up to 1 MWp, based on a remote-controlled drone [4]. Since then, several publications have demonstrated the technique's capability to detect failures in photovoltaic systems quickly and efficiently from the commissioning phase of the power plant through its expected 25 years of operation [5][6][7][8][9].
To further improve the time and cost efficiency of the method, the automation of the entire procedure of the aIRT technique has been assessed in recent years by several

Method of Review
The literature search for conference and journal papers was carried out in the scientific databases IEEE Xplore and ScienceDirect, and on the web scientific indexing services Web of Science and Google Scholar. The keywords used included PV systems, inspections and thermography synonyms. Keywords related to aerial and UAVs were not included to not limit results and exclude the automatic assessment of ground-based thermographic images, besides reducing false positives related to the development of UAVs powered by PV modules. The string used for the search was "(photovoltaic OR PV OR (PV AND modules)) AND (faults OR detection OR classification) AND (automatic OR (artificial AND intelligence) OR processing) AND (thermography OR thermal OR infrared)" and initially returned 183 papers. The papers were filtered to fit the theme and classified according to application and automation algorithm. Papers that focused on the automatic assessment of visual images were not excluded since, normally, aIRT-aimed UAVs also have a visual camera coupled to them, which is also used in the inspection to better classify faults.
The list of papers was expanded using references of the original papers and the references already known by the authors. When overlapping work was found in multiple publications (e.g., preprint in a conference and full paper in a journal), only the publication deemed most important was included.

Infrared Thermography (IRT)
Infrared thermography (IRT) is a technique that assesses the radiation emitted by the surface of any body in the infrared wavelength spectrum between 1.4 and 15 µm. IRT cameras used in PV inspections normally have the capability of measuring wavelengths in the mid-infrared wavelength range of 7-14 µm. This is a good compromise between costs and product availability, according to measuring conditions in the field [10].
In a PV plant, PV modules that are close to each other receive almost the same amount of irradiance. Some 20% of this irradiance will be converted to electricity; however, most of the photons which are not converted into electricity will produce heat in the cell. This will increase the temperature of these damaged cells and modules, and the faults will appear in the acquired IRT images as temperature differences [11], like a "temperature signature".
IRT has been shown to be a reliable and non-destructive tool for detection of different types of faults in PV cells, modules and strings, as they have an effect on the PV module thermal behavior. Table 1 presents some thermal signatures which have been identified and classified in previous studies [2,12] and are standardized according to the international standard IEC TS 62446-3 Edition 1.0 2017-06 [13].

Unmanned Aerial Vehicles (UAVs)
Unmanned aerial vehicles (UAVs) are aircrafts that are capable of remote or autonomous operation. They were initially developed for military applications, but due to recent developments, they are now available for civil activities and commonly used in applications such as rescuing and disaster relief, energy power line monitoring and environmental and forest control [16]. UAVs are becoming more popular especially in the energy and agriculture sectors, due to their fast large-area coverage, precise imagery, high flexibility, light weight, low cost and ability to operate in hostile environments [17,18]. Several countries have established legal guidelines for the use of UAVs. The rules can include limitations of flight areas, license requirements and insurance [19].
UAVs can carry various cameras and sensors to collect data and can be classified by size, range or endurance [20]. The most appropriate UAV equipment type for thermographic inspections is multicopters, which use rotary blades to generate lift, because of their stability and easy usability. They can be classified by the number of motors (tricopters, quadcopters, hexacopters and octocopters), with quadcopters being the most popular on the market [19]. They can also be classified by autonomy levels, being manually operated, semi-autonomous (i.e., need of a human operator for mission planning and for taking some of the movement decisions) or fully autonomous (i.e., all decisions for a delegated mission are made by the UAV based on sensor observations) [21]. Suspicious conductor strip [13,14] Energies 2022, 15, x FOR PEER REVIEW 3 of 25

Unmanned Aerial Vehicles (UAVs)
Unmanned aerial vehicles (UAVs) are aircrafts that are capable of remote or autonomous operation. They were initially developed for military applications, but due to recent developments, they are now available for civil activities and commonly used in applications such as rescuing and disaster relief, energy power line monitoring and environmental and forest control [16]. UAVs are becoming more popular especially in the energy and agriculture sectors, due to their fast large-area coverage, precise imagery, high flexibility, light weight, low cost and ability to operate in hostile environments [17,18]. Several countries have established legal guidelines for the use of UAVs. The rules can include limitations of flight areas, license requirements and insurance [19].
UAVs can carry various cameras and sensors to collect data and can be classified by size, range or endurance [20]. The most appropriate UAV equipment type for thermographic inspections is multicopters, which use rotary blades to generate lift, because of their stability and easy usability. They can be classified by the number of motors (tricopters, quadcopters, hexacopters and octocopters), with quadcopters being the most popular on the market [19]. They can also be classified by autonomy levels, being manually operated, semi-autonomous (i.e., need of a human operator for mission planning and for taking some of the movement decisions) or fully autonomous (i.e., all decisions for a delegated mission are made by the UAV based on sensor observations) [21]. Disconnected substring [13,15] Energies 2022, 15, x FOR PEER REVIEW 3 of 25

Unmanned Aerial Vehicles (UAVs)
Unmanned aerial vehicles (UAVs) are aircrafts that are capable of remote or autonomous operation. They were initially developed for military applications, but due to recent developments, they are now available for civil activities and commonly used in applications such as rescuing and disaster relief, energy power line monitoring and environmental and forest control [16]. UAVs are becoming more popular especially in the energy and agriculture sectors, due to their fast large-area coverage, precise imagery, high flexibility, light weight, low cost and ability to operate in hostile environments [17,18]. Several countries have established legal guidelines for the use of UAVs. The rules can include limitations of flight areas, license requirements and insurance [19].
UAVs can carry various cameras and sensors to collect data and can be classified by size, range or endurance [20]. The most appropriate UAV equipment type for thermographic inspections is multicopters, which use rotary blades to generate lift, because of their stability and easy usability. They can be classified by the number of motors (tricopters, quadcopters, hexacopters and octocopters), with quadcopters being the most popular on the market [19]. They can also be classified by autonomy levels, being manually operated, semi-autonomous (i.e., need of a human operator for mission planning and for taking some of the movement decisions) or fully autonomous (i.e., all decisions for a delegated mission are made by the UAV based on sensor observations) [21].

Unmanned Aerial Vehicles (UAVs)
Unmanned aerial vehicles (UAVs) are aircrafts that are capable of remote or autonomous operation. They were initially developed for military applications, but due to recent developments, they are now available for civil activities and commonly used in applications such as rescuing and disaster relief, energy power line monitoring and environmental and forest control [16]. UAVs are becoming more popular especially in the energy and agriculture sectors, due to their fast large-area coverage, precise imagery, high flexibility, light weight, low cost and ability to operate in hostile environments [17,18]. Several countries have established legal guidelines for the use of UAVs. The rules can include limitations of flight areas, license requirements and insurance [19].
UAVs can carry various cameras and sensors to collect data and can be classified by size, range or endurance [20]. The most appropriate UAV equipment type for thermographic inspections is multicopters, which use rotary blades to generate lift, because of their stability and easy usability. They can be classified by the number of motors (tricopters, quadcopters, hexacopters and octocopters), with quadcopters being the most popular on the market [19]. They can also be classified by autonomy levels, being manually operated, semi-autonomous (i.e., need of a human operator for mission planning and for taking some of the movement decisions) or fully autonomous (i.e., all decisions for a delegated mission are made by the UAV based on sensor observations) [21].

Unmanned Aerial Vehicles (UAVs)
Unmanned aerial vehicles (UAVs) are aircrafts that are capable of remote or autonomous operation. They were initially developed for military applications, but due to recent developments, they are now available for civil activities and commonly used in applications such as rescuing and disaster relief, energy power line monitoring and environmental and forest control [16]. UAVs are becoming more popular especially in the energy and agriculture sectors, due to their fast large-area coverage, precise imagery, high flexibility, light weight, low cost and ability to operate in hostile environments [17,18]. Several countries have established legal guidelines for the use of UAVs. The rules can include limitations of flight areas, license requirements and insurance [19].
UAVs can carry various cameras and sensors to collect data and can be classified by size, range or endurance [20]. The most appropriate UAV equipment type for thermographic inspections is multicopters, which use rotary blades to generate lift, because of their stability and easy usability. They can be classified by the number of motors (tricopters, quadcopters, hexacopters and octocopters), with quadcopters being the most popular on the market [19]. They can also be classified by autonomy levels, being manually operated, semi-autonomous (i.e., need of a human operator for mission planning and for taking some of the movement decisions) or fully autonomous (i.e., all decisions for a delegated mission are made by the UAV based on sensor observations) [21].
Module with broken front glass [13] Energies 2022, 15, x FOR PEER REVIEW 3 of 25

Unmanned Aerial Vehicles (UAVs)
Unmanned aerial vehicles (UAVs) are aircrafts that are capable of remote or autonomous operation. They were initially developed for military applications, but due to recent developments, they are now available for civil activities and commonly used in applications such as rescuing and disaster relief, energy power line monitoring and environmental and forest control [16]. UAVs are becoming more popular especially in the energy and agriculture sectors, due to their fast large-area coverage, precise imagery, high flexibility, light weight, low cost and ability to operate in hostile environments [17,18]. Several countries have established legal guidelines for the use of UAVs. The rules can include limitations of flight areas, license requirements and insurance [19].
UAVs can carry various cameras and sensors to collect data and can be classified by size, range or endurance [20]. The most appropriate UAV equipment type for thermographic inspections is multicopters, which use rotary blades to generate lift, because of their stability and easy usability. They can be classified by the number of motors (tricopters, quadcopters, hexacopters and octocopters), with quadcopters being the most popular on the market [19]. They can also be classified by autonomy levels, being manually operated, semi-autonomous (i.e., need of a human operator for mission planning and for taking some of the movement decisions) or fully autonomous (i.e., all decisions for a delegated mission are made by the UAV based on sensor observations) [21].

Unmanned Aerial Vehicles (UAVs)
Unmanned aerial vehicles (UAVs) are aircrafts that are capable of remote or autonomous operation. They were initially developed for military applications, but due to recent developments, they are now available for civil activities and commonly used in applications such as rescuing and disaster relief, energy power line monitoring and environmental and forest control [16]. UAVs are becoming more popular especially in the energy and agriculture sectors, due to their fast large-area coverage, precise imagery, high flexibility, light weight, low cost and ability to operate in hostile environments [17,18]. Several countries have established legal guidelines for the use of UAVs. The rules can include limitations of flight areas, license requirements and insurance [19].
UAVs can carry various cameras and sensors to collect data and can be classified by size, range or endurance [20]. The most appropriate UAV equipment type for thermographic inspections is multicopters, which use rotary blades to generate lift, because of their stability and easy usability. They can be classified by the number of motors (tricopters, quadcopters, hexacopters and octocopters), with quadcopters being the most popular on the market [19]. They can also be classified by autonomy levels, being manually operated, semi-autonomous (i.e., need of a human operator for mission planning and for taking some of the movement decisions) or fully autonomous (i.e., all decisions for a delegated mission are made by the UAV based on sensor observations) [21].

Unmanned Aerial Vehicles (UAVs)
Unmanned aerial vehicles (UAVs) are aircrafts that are capable of remote or autonomous operation. They were initially developed for military applications, but due to recent developments, they are now available for civil activities and commonly used in applications such as rescuing and disaster relief, energy power line monitoring and environmental and forest control [16]. UAVs are becoming more popular especially in the energy and agriculture sectors, due to their fast large-area coverage, precise imagery, high flexibility, light weight, low cost and ability to operate in hostile environments [17,18]. Several countries have established legal guidelines for the use of UAVs. The rules can include limitations of flight areas, license requirements and insurance [19].
UAVs can carry various cameras and sensors to collect data and can be classified by size, range or endurance [20]. The most appropriate UAV equipment type for thermographic inspections is multicopters, which use rotary blades to generate lift, because of their stability and easy usability. They can be classified by the number of motors (tricopters, quadcopters, hexacopters and octocopters), with quadcopters being the most popular on the market [19]. They can also be classified by autonomy levels, being manually operated, semi-autonomous (i.e., need of a human operator for mission planning and for taking some of the movement decisions) or fully autonomous (i.e., all decisions for a delegated mission are made by the UAV based on sensor observations) [21].

Unmanned Aerial Vehicles (UAVs)
Unmanned aerial vehicles (UAVs) are aircrafts that are capable of remote or autonomous operation. They were initially developed for military applications, but due to recent developments, they are now available for civil activities and commonly used in applications such as rescuing and disaster relief, energy power line monitoring and environmental and forest control [16]. UAVs are becoming more popular especially in the energy and agriculture sectors, due to their fast large-area coverage, precise imagery, high flexibility, light weight, low cost and ability to operate in hostile environments [17,18]. Several countries have established legal guidelines for the use of UAVs. The rules can include limitations of flight areas, license requirements and insurance [19].
UAVs can carry various cameras and sensors to collect data and can be classified by size, range or endurance [20]. The most appropriate UAV equipment type for thermographic inspections is multicopters, which use rotary blades to generate lift, because of their stability and easy usability. They can be classified by the number of motors (tricopters, quadcopters, hexacopters and octocopters), with quadcopters being the most popular on the market [19]. They can also be classified by autonomy levels, being manually operated, semi-autonomous (i.e., need of a human operator for mission planning and for taking some of the movement decisions) or fully autonomous (i.e., all decisions for a delegated mission are made by the UAV based on sensor observations) [21].

Unmanned Aerial Vehicles (UAVs)
Unmanned aerial vehicles (UAVs) are aircrafts that are capable of remote or autonomous operation. They were initially developed for military applications, but due to recent developments, they are now available for civil activities and commonly used in applications such as rescuing and disaster relief, energy power line monitoring and environmental and forest control [16]. UAVs are becoming more popular especially in the energy and agriculture sectors, due to their fast large-area coverage, precise imagery, high flexibility, light weight, low cost and ability to operate in hostile environments [17,18]. Several countries have established legal guidelines for the use of UAVs. The rules can include limitations of flight areas, license requirements and insurance [19].
UAVs can carry various cameras and sensors to collect data and can be classified by size, range or endurance [20]. The most appropriate UAV equipment type for thermographic inspections is multicopters, which use rotary blades to generate lift, because of their stability and easy usability. They can be classified by the number of motors (tricopters, quadcopters, hexacopters and octocopters), with quadcopters being the most popular on the market [19]. They can also be classified by autonomy levels, being manually operated, semi-autonomous (i.e., need of a human operator for mission planning and for taking some of the movement decisions) or fully autonomous (i.e., all decisions for a delegated mission are made by the UAV based on sensor observations) [21].

Aerial Infrared Thermography
In the past, IRT inspections of PV systems were performed on the ground or on lifting platforms with handheld IRT cameras. This procedure cannot provide fast and accurate coverage of the power plant and is dependent on human labor and competence, besides being very time-consuming and labor-intensive. As a result, the inspection accuracy is susceptible to human error. A possible solution to the problem is to combine the IRT sensor with aerial technologies such as UAVs. This procedure is known as aIRT and increases the cost effectiveness of IRT inspection, allowing the technique to be employed for large-scale PV plants or roof-mounted PV systems with limited access [8,[22][23][24].
This method has shown its potential in recent years, but its use can still be expanded to increase and simplify routine inspections of PV power plants and revolutionize the future of PV plant monitoring procedures [16]. For this method to achieve its full potential, however, it should be combined with automation algorithms and technologies, such as automated route planning and defect identification.
Aerial IRT uses visual and IRT cameras that are mounted on the UAV. The equipment provides real-time and precise imagery, allowing a time-efficient inspection. The procedure can detect faults of different types, such as cell cracks, corrosion spots, broken cells, hot spots, snail trails, discoloration, soiling, disconnected modules or strings and potentialinduced degradation (PID) [18,25]. According to the inspection goal and the PV plant layout, aIRT can be performed from different altitudes and directions to identify specific defects or faults, also depending on the time available for the inspection [26].
The aIRT process is carried out in three stages: acquisition of imagery, assessment of data and remediation actions. For the acquisition step, the UAV must fly a designated route acquiring successive photos or videos over the site to build an imagery database covering all modules in the system. As the equipment has a limited battery capacity, each flight has a duration of around 20 min [6].
During the flight, the weather conditions, wind speed and sunlight reflection must be monitored, as they can affect the measurements and consequently the quality of the aIRT images. Additionally, the velocity of the UAV and the orientation and angle of the IRT sensor must be controlled to avoid self-shading, blurred images and other reflections. aIRT should be performed on cloudless, sunny and clear days, with minimum irradiance of 600 W/m 2 on the plane of the PV array under inspection. The flight path and velocity should be planned beforehand in order to optimize the coverage and to avoid any drift during the flight [25][26][27][28].
The acquired data are then analyzed, and an action report is produced. With a detailed site mapping, it is possible to obtain an accurate location of the system faults, and the remediation can be planned based on full knowledge of the site state. The report is passed to the stakeholders for remediation actions such as assessment of connections or replacement of modules or fuses.

Digital Image Processing
The field of digital image processing (DIP) has been an object of increasing interest because it allows many applications such as: remote sensing, component defect detection, biomedical engineering, geoprocessing, metallography and industrial automation applications. DIP aims to transform and analyze images by extracting relevant information, highlighting and identifying patterns and objects [29,30].
A digital image consists of a set of a finite number of elements (i.e., pixel), each one with a specific location and intensity. DIP techniques apply several operations to these pixels to transform the images, allowing the automatic interpretation of them by systems or machines. These operations include image rotation, thresholding, binary image analysis, brightness and contrast adjustment, filtering, resizing and interpolation [27,29].
The methods of DIP are normally simple but can often solve problems in a more time-and computing-efficient way than DL techniques. Pixel thresholding or algorithms such as the scale-invariant feature transform (SIFT) are normally very general, do not require a large dataset to be trained and can be replicated in other images. DIP can even be combined with DL to take the best from both methods to automate image processing and recognition [31,32].

Deep Learning (DL)
Artificial neural networks (ANNs) are the most popular technique in machine learning (ML) and were first developed based on the structure and operation of the human brain. They are commonly used because they can deal with highly nonlinear systems and allow constant updates in the model [33]. They are composed of a set of simple, connected processors called neurons that produce a sequence of activation calculations. The fundamental property of an ANN is its ability to learn from the environment (read a set of examples), through an iterative process of adjustments applied to synaptic weights and bias levels. Learning a task consists in finding weights that make the ANN exhibit a desired output when processing an input. Depending on the problem and how the neurons are connected, this process may require chains of consecutive computational stages, where each one modifies the aggregate activation of the network. DL is about accurately assigning credit across many of these stages, using convolutional neural networks (CNNs) [34,35].
CNNs were inspired by the human visual system's structure and are the state-of-theart method for image classification [36][37][38][39]. They are easy to train if there is a substantial number of labeled images that represent distinct categories.
They are layered sequences, and each has a specific function in the propagation of the input signal. There are three main types of neural layers: (i) convolutional layers, (ii) pooling layers and (iii) fully connected layers. Figure 1 illustrates the CNN architecture and its three main layers for a task of object detection in an image:

•
Convolutional layers: responsible for extracting attributes from the input volumes.

•
Pooling layers: responsible for reducing the spatial dimensions of the input volume after the convolutional layers, reducing the computational work of the network. • Fully connected layers: perform the final classification of the images, in the same way as a conventional neural network.
In the end, the CNN output is the probability that the input image belongs to one of the classes for which the network was trained [40].
as the scale-invariant feature transform (SIFT) are normally very general, do not requ large dataset to be trained and can be replicated in other images. DIP can even be c bined with DL to take the best from both methods to automate image processing recognition [31,32].

Deep Learning (DL)
Artificial neural networks (ANNs) are the most popular technique in machine le ing (ML) and were first developed based on the structure and operation of the hu brain. They are commonly used because they can deal with highly nonlinear systems allow constant updates in the model [33]. They are composed of a set of simple, conne processors called neurons that produce a sequence of activation calculations. The fu mental property of an ANN is its ability to learn from the environment (read a set o amples), through an iterative process of adjustments applied to synaptic weights and levels. Learning a task consists in finding weights that make the ANN exhibit a des output when processing an input. Depending on the problem and how the neuron connected, this process may require chains of consecutive computational stages, w each one modifies the aggregate activation of the network. DL is about accurately ass ing credit across many of these stages, using convolutional neural networks (CN [34,35].
CNNs were inspired by the human visual system's structure and are the state-of art method for image classification [36][37][38][39]. They are easy to train if there is a substa number of labeled images that represent distinct categories.
They are layered sequences, and each has a specific function in the propagatio the input signal. There are three main types of neural layers: (i) convolutional layers pooling layers and (iii) fully connected layers. Figure 1 illustrates the CNN architec and its three main layers for a task of object detection in an image:

•
Convolutional layers: responsible for extracting attributes from the input volum • Pooling layers: responsible for reducing the spatial dimensions of the input vol after the convolutional layers, reducing the computational work of the network • Fully connected layers: perform the final classification of the images, in the same as a conventional neural network.
In the end, the CNN output is the probability that the input image belongs to on the classes for which the network was trained [40].

Other Machine Learning Techniques
Other ML techniques are also used in the automation of aIRT inspection, espec for classification in combination with other techniques. Among them, support vector chines (SVMs) are supervised ML algorithms that are usually used for classification regression analysis of data, as they are based on a decision function that divides the cla by decision planes. SVMs work relatively well when there is a clear separation betw classes, and they are more effective in high-dimensional spaces [41,42].

Other Machine Learning Techniques
Other ML techniques are also used in the automation of aIRT inspection, especially for classification in combination with other techniques. Among them, support vector machines (SVMs) are supervised ML algorithms that are usually used for classification and regression analysis of data, as they are based on a decision function that divides the classes by decision planes. SVMs work relatively well when there is a clear separation between classes, and they are more effective in high-dimensional spaces [41,42].
Another commonly used classifier is the random forest (RF), which is an algorithm that builds several decision trees on multiple random portions of the data for training. In these trees, different binary classifications concatenated in a tree structure are performed, for different input features, and a combination of the nodes is used to calculate the result. The most common result among the trees is "elected" as the output of the classifier [43,44].
On the other hand, k-nearest neighbors (KNN) is a classification supervised algorithm that does not require training, as the samples are classified by a metric of distance, where the k-nearest points to the test sample are defined as a class [43].
For clustering, the most popular method is k-means clustering, which is an unsupervised ML algorithm. This method consists of dividing data in k clusters that will be grouped by the mean distance between points [45,46].

Algorithm Evaluation Metrics
Performance indices quantify the capacity of an algorithm to identify the events of interest. The accuracy (Ac) of an algorithm is the percentage of correct predictions over the test dataset. The precision index (Pr) provides the proportion of all segments that were identified in one class that in fact belong to this class (results of true positives over all the positives). The recall (Re) measures how well the algorithm can identify each class among the dataset (results of true positives over all images in a dataset that were originally labeled as positives). The F1 score is calculated by the harmonic mean of Pr and Re. A good algorithm should have all of these metrics close to 100% for all the classes [47].
Other less common metrics are also used in the papers covered by this work. The Matthews correlation coefficient (MCC) is a coefficient related to accuracy, which does not penalize classes of different sizes [48]. The Dice coefficient and the Jaccard index are often used for evaluating segmentation tasks. The first represents the overlapping between the predicted result with the ground truth, and the second measures the similarity of the predicted result to the ground truth [37,49]. The Jaccard index is also known as Intersection over Union (IOU). The area under the receiver operating characteristic (AUROC) is an index obtained by integrating the curve of the true positives of a task over the false positives at various decision thresholds [50].
In this paper, when available, the preferred metrics are F1 score and precision indices, in order to provide a comparison between different works, whenever possible.

Automatic Path Planning
In an automatic aIRT mission, the UAV flies over a set of waypoints that cover all modules of the PV plant. Therefore, an optimized path-planning algorithm aiming at an optimal path for time and battery efficiency is essential [51]. Figure 2 depicts the coverage area by a UAV based on the field of view (FoV) and resolution of the camera on board as well as the essential parameters used for path planning [51].
Available market software packages already provide an automatic flight based on a so-called "lawn mower" flight pattern. However, they do not always provide the most efficient flight and do not guarantee a centralized view to the PV arrays, especially in power plants installed over complex topography.
Studies aiming at optimizing path planning include different approaches to the problem. In the study developed by Salahat et al. [52], the traveling salesman shortest path algorithm was used to generate a path that includes a randomly selected set of modules that represent the entire PV plant, allowing an optimization of the battery use. Ding et al. [53] based their method on density clustering, boustrophedon path planning and Bezier curves. Luo et al. [54] also based their algorithm for path planning optimization on Bezier curves in a joint approach with particle swarm optimization (PSO), taking into consideration the flight attitude, gimbal limitation and path length.
Image stitching and DIP techniques were used by Henry et al. [55] to find contours of the power plant and generate a "lawn mower" path over it. A similar approach, using DL, was adopted by Moradi Sizkouhi et al. [51,56], which was also complemented with a dynamic path planning, which deviates the previous flight plan to take closer photos when faults are detected. Pérez-González et al. [57] also used DL to detect the area of the PV plant and then used different algorithms to determine the best flight path, wherein exact cellular decomposition boustrophedon and grid-based wavefront coverage algorithms produced the best results.
Other real-time algorithms have been proposed that calculate the optimized path of the UAV during the flight. In Roggi et al. [58], the UAV corrected the pre-planned "lawn mower" path according to the images that it acquired and the image processing techniques that are applied. A vision-based flight control was also proposed by Xi et al. [59], which performs a real-time direction and velocity correction.
Image stitching and DIP techniques were used by Henry et al. [55] to find contours of the power plant and generate a "lawn mower" path over it. A similar approach, using DL, was adopted by Moradi Sizkouhi et al. [51,56], which was also complemented with a dynamic path planning, which deviates the previous flight plan to take closer photos when faults are detected. Pérez-González et al. [57] also used DL to detect the area of the PV plant and then used different algorithms to determine the best flight path, wherein exact cellular decomposition boustrophedon and grid-based wavefront coverage algorithms produced the best results.
Other real-time algorithms have been proposed that calculate the optimized path of the UAV during the flight. In Roggi et al. [58], the UAV corrected the pre-planned "lawn mower" path according to the images that it acquired and the image processing techniques that are applied. A vision-based flight control was also proposed by Xi et al. [59], which performs a real-time direction and velocity correction.

Detection of PV Systems
Although not always related to aerial inspections, the detection of PV systems in aerial imagery (UAV or satellite imagery) has been proposed by many authors and even used for path planning before the aIRT flight [56,57]. Table 2 presents a summary of methods used in the literature for detecting PV systems in aerial imagery. The table presents the best metrics obtained in each study, the type of image data used as input and the type of detection output obtained with each method. The detection output was classified into three categories: (a) Boxes: the output is given by the coordinates of a box or polygon placed around the PV system; (b) Mask: the output is a binary image where the pixels corresponding to the segment of the PV system are represented by the value 1 and the rest of the image is represented by 0; (c) Binary: for each image, the presence or absence of a PV system is the result of the algorithm (1 or 0). Figure 3 shows two examples of two types of detections, by the coordinates of boxes around the detected PV panels (left) and masks of the segment of the PV system (right). Besides developing an algorithm for the detection of PV arrays, Wu et al. [60] also matched them to their string identifiers.

Detection of PV Systems
Although not always related to aerial inspections, the detection of PV systems in aerial imagery (UAV or satellite imagery) has been proposed by many authors and even used for path planning before the aIRT flight [56,57]. Table 2 presents a summary of methods used in the literature for detecting PV systems in aerial imagery. The table presents the best metrics obtained in each study, the type of image data used as input and the type of detection output obtained with each method. The detection output was classified into three categories: (a) Boxes: the output is given by the coordinates of a box or polygon placed around the PV system; (b) Mask: the output is a binary image where the pixels corresponding to the segment of the PV system are represented by the value 1 and the rest of the image is represented by 0; (c) Binary: for each image, the presence or absence of a PV system is the result of the algorithm (1 or 0). Figure 3 shows two examples of two types of detections, by the coordinates of boxes around the detected PV panels (left) and masks of the segment of the PV system (right). Besides developing an algorithm for the detection of PV arrays, Wu et al. [60] also matched them to their string identifiers.

Detection of PV Modules
The detection of the area of the modules is an important step in the image processing task, which is required for both detecting modules with defects and locating them in the power plant.

Detection of PV Modules
The detection of the area of the modules is an important step in the image processing task, which is required for both detecting modules with defects and locating them in the power plant. The first attempts to automatize the segmentation of the PV modules in IRT images appear to have been made in 2015, with images obtained with a moving cart, using simple DIP methods [76]. Other attempts were described in Menéndez et al. [77], Montanez et al. [78] and Wu et al. [79]. Uma et al. [80] did the same using a k-means clustering algorithm. In 2021, Xie et al. [81] used a Segnet, a CNN used for segmentation. For aerial images, Tribak and Zaz [82], Salamanca et al. [83] and Perez et al. [84] published results on detecting PV modules in aerial visual images, and many studies used different techniques to detect and segment PV module boundaries in aIRT data. Table 3 shows the studies related to DIP and DL algorithms. In this case, the detection result of the algorithm can be given by a box or polygon around the PV module, a mask of the segment of the module or lines that mark the borders of the modules in a PV panel.
The best metric among the studies was obtained with a combination of many algorithms (DIP, SVM and DL) [85]. On the other hand, the worst metrics were obtained with simple DIP filters [86]. It is important to note that not all studies are comparable, because not all have presented metrics for their performance, and they have different dataset sizes, which make the comparison difficult. The studies described by Carletti et al. [87] and Bommes et al. [88], besides detecting the PV module, also developed ways to track the modules in subsequent frames of a video. A tracking system was also developed by Xie et al. [89] using an AlexNet CNN.

Orthomosaicking
The localization of faults within a power plant is a challenging issue that can be addressed by creating an orthomosaic of the PV power plant [103]. Image mosaicking, also known as image stitching, is a computational technique that detects over-lapping key points in spatially subsequent photos and uses them to create a so-called panorama picture [94].
To improve the sight perspective and enable an expanded view of the localization of faults in both visual and thermal images of PV power plants, some researchers used commercially available software packages to create orthomaps with aerial imagery. Lee and Park [104] and Zefri et al. [105] used the software Pix4D capture to process thermal and visual images and create orthographic images with temperature information. Oliveira et al. [103] compared the use of two software packages (DroneDeploy and ContextCapture) to create an orthomosaic of a 1 MW PV power plant. Higuchi and Babasaki [106] used the software OpenDroneMap to generate the orthographic image of a 2 MW PV power plant.
Grimaccia et al. [94], Aghaei et al. [27] and Ismail et al. [107] have proposed methods for the orthomosaicking of visual images of PV power plants using DIP techniques. Tsanakas et al. [108] used the method of aerial triangulation, and Lafkih and Zaz [109] and Zefri [102,110] used the SIFT technique to perform the task. To optimize the mosaicking of visual PV images, Qi et al. [111] used a Faster R-CNN to detect key points in aerial sequence images in the world coordinate system, so it avoids redundant information generated by traditional methods. López-Fernández et al. [92] developed a tool that creates a 5D point cloud of the power plant, where each coordinate point has a temperature and an intensity value associated with it. After segmenting the modules in a dataset, Costa et al. [73] used a sliding window algorithm with overlapping pixels, combining frames side by side to reconstruct orthomosaics of power plants.

Soiling
A common cause of hot spots in PV power plants is soiling and shadow over the modules, which hinders the evaluation of results since they are not considered real defects of the PV modules [6]. Cipriani et al. [112] approached this issue by using a CNN to differentiate hot spots caused by faults from soiling, obtaining an accuracy of up to 98%.
Another solution to the problem is the analysis of the visual images that are normally taken together with the IRT images in the UAV, which enables the operator to discard hot spots caused by soiling. Automation of the task of detecting soiling in individual modules was proposed by Yang et al. [113], Pivem et al. [114] and Qasem et al. [115] using DIP techniques. Similar techniques were employed by Wen et al. [116], and by Karaköse and Firildak [117] to detect shadows over PV systems. Hanafy et al. [43] compared different ML algorithms (KNN, NN, RF and SVM) to classify modules in different categories of cleanliness and obtained an accuracy of over 90% using an SVM algorithm. Mehta et al. [37] proposed a method that uses a weakly supervised CNN-based classification network to predict power loss, detect soiling and categorize it given a PV module image. This method obtained an accuracy of about 87% on a test dataset of about 50 images.

Detection and Classification of Faults
The manual assessment of aIRT imagery is a time-and computing-consuming task; therefore, its automation is the most explored part of the aIRT framework in the literature. This detection can either be processed on board, during the UAV flight, as shown in the example in Figure 4, or in a computer software, after the acquisition of images has been carried out by the UAV (Figure 5). Both Figures 4 and 5 show the procedure of the inspections for the two different approaches, including all tasks being automatized in each case.   In 2002, Pilla et al. [118] used the Sobel and canny edge operators to detect cracks in IRT PV cell images. In 2003, Wang et al. [119] used thresholding and a fuzzy classifier to detect faults in IRT images. After that, in 2011, Vergura and Falcone [120] used DIP techniques to analyze IRT images for faults. Since then, many other studies have used mostly DIP techniques to segment faults in IRT images. Table 4 presents a summary of the main algorithms for the detection and classification of faults in IRT images. In this case, the results can be in the form of a segmentation of the faults (mask), the detection of modules with faults and the classification of these faults in categories. Some methods presented high metrics when used in association with a classification algorithm, such as an SVM [121]. Many papers do not present a metric for the performance evaluation of the algorithms proposed, especially for the case of fault segmentation (masks). It is important to note that a comparison between metrics is not always possible, because the dataset size and number of classes differ among studies.   In 2002, Pilla et al. [118] used the Sobel and canny edge operators to detect cracks in IRT PV cell images. In 2003, Wang et al. [119] used thresholding and a fuzzy classifier to detect faults in IRT images. After that, in 2011, Vergura and Falcone [120] used DIP techniques to analyze IRT images for faults. Since then, many other studies have used mostly DIP techniques to segment faults in IRT images. Table 4 presents a summary of the main algorithms for the detection and classification of faults in IRT images. In this case, the results can be in the form of a segmentation of the faults (mask), the detection of modules with faults and the classification of these faults in categories. Some methods presented high metrics when used in association with a classification algorithm, such as an SVM [121]. Many papers do not present a metric for the performance evaluation of the algorithms proposed, especially for the case of fault segmentation (masks). It is important to note that a comparison between metrics is not always possible, because the dataset size and number of classes differ among studies. In 2002, Pilla et al. [118] used the Sobel and canny edge operators to detect cracks in IRT PV cell images. In 2003, Wang et al. [119] used thresholding and a fuzzy classifier to detect faults in IRT images. After that, in 2011, Vergura and Falcone [120] used DIP techniques to analyze IRT images for faults. Since then, many other studies have used mostly DIP techniques to segment faults in IRT images. Table 4 presents a summary of the main algorithms for the detection and classification of faults in IRT images. In this case, the results can be in the form of a segmentation of the faults (mask), the detection of modules with faults and the classification of these faults in categories. Some methods presented high metrics when used in association with a classification algorithm, such as an SVM [121]. Many papers do not present a metric for the performance evaluation of the algorithms proposed, especially for the case of fault segmentation (masks). It is important to note that a comparison between metrics is not always possible, because the dataset size and number of classes differ among studies.  Table 5 shows the summary of studies of the detection and classification of faults in visual images. The faults detected in this case are mostly related to visible problems, such as delamination, burn marks and glass breakages. An important issue that makes the comparison between studies difficult is the difference between the data resolution used as input for each one of them, as images vary from PV cells [140] to aerial images [141].  Tables 6 and 7 present a summary of the methods for detecting and classifying faults in aIRT images of PV systems, using DIP and DL algorithms, respectively. In general, the algorithms with the highest results are the ones dedicated to the detection of faults or the classification of a few types of faults, with the classification of many classes of faults being a much more complex task. It is noticeable that the DIP algorithms have comparable results to DL techniques, even though most of them use smaller datasets, and therefore their generalization capabilities can be jeopardized.
Some of the challenges for the development of a robust automatic classification of faults include the reflections and shadows from surroundings and the lack of a standardized image database (with standard flight directions and weather conditions). When using DL, the computation requirements, the need for a large dataset with annotated data and the processing time must also be overcome.

Other Applications
Imaging techniques have been employed in some other applications to facilitate the analysis of PV modules. An example is the detection of blurred images that was addressed by Tribak and Zaz [176] with image processing techniques in order to filter frames of videos before employing mosaicking techniques. Similar techniques were used by Shen et al. [177] to correct the angle distortion of IRT images.

Discussion
This review has shown that different automatization algorithms, including DIP, DL and classification techniques, have been employed for automating different tasks of the aIRT procedure for inspecting PV power plants. Among the conclusions, this review showed that only a few among the selected studies have assessed two important aspects of the autonomous inspection procedure, namely, the optimization of the flight path (nine papers), and the detection of soiling (eight papers). These two topics are of great importance to increase time efficiency in aIRT and therefore should be further investigated. The latter goal of detecting soiling over PV modules and differentiating it from actual faults of the modules was investigated by some authors, e.g., Dunderdale et al. [170] and Arosh et al. [159], together with the detection and classification of other faults.
For the task of performing the orthomosaicking of aIRT images to facilitate the localization of the faults in the field, four papers employed existing software to perform the task, while ten studies approached the development of algorithms to create the orthomosaic of the PV plant. However, most of the proposed methods are based on DIP techniques; therefore, the resulting mosaic consists of a simple image, without additional GPS information. The correlation of orthomosaic images with GPS coordinates and the identification of modules and strings according to the site nomenclature are areas that require further investigation. Another approach to the challenge was developed by Wu et al. [60], with the development of an algorithm that detects PV arrays in power plants and performs the automatic correlation with their string identifiers. This is a promising strategy that could also be used to facilitate the localization of detected faults in the field through aIRT. Besides the study carried out by Wu et al. [60], another 20 studies among the selected literature focused on the development of algorithms to detect PV systems and panels in aerial imagery. However, only three of these studies focused on aerial IRT images of the PV plants, obtaining up to 93.16% precision in the results [63]. On the one hand, 18 papers presented the results of developed algorithms for the detection of individual PV modules in aIRT images, of which three of them applied DL techniques. Although the methods are hardly comparable given their different structures for results (i.e., mask, box or line), their different dataset sizes and the different evaluation metrics used, a method that combined many algorithms (DIP, SVM and DL) for detecting PV modules in aIRT images and obtained an F1 score of 98.4% can be highlighted [85]. On the other hand, the worst metrics were obtained with simple DIP filters [86], which although providing fast results with small datasets required for training, are characterized by a lack in generalization. This is important for the replication of the algorithm in images acquired in different conditions and with a different quality. The algorithms proposed by Carletti et al. [87], Xie et al. [89] and Bommes et al. [88] also performed the tracking of the modules in subsequent frames of an aIRT video. This task is of utter importance for the cross-correlation of detected modules and faults, as well as their location in PV plants.
Most of the selected studies have assessed autonomous fault detection and classification in PV plants through visual (12 papers), IRT (22 papers) and aIRT images (43 papers). Among these studies, 35% used DL techniques for the detection or classification of PV faults, with an increase in developed algorithms using CNNs in recent years. Still, DIP-based algorithms also presented high accuracy results, even though most of them use smaller datasets, and therefore their replication in other sets of data is possibly not feasible. The combination of DL or DIP techniques with classifier algorithms was a promising approach in recent studies. In the field, fault detection can either be processed on board, during the UAV flight, or subsequently through a post-processing procedure after the flight. For the first case, the high computational requirements and the processing time of DL are still a challenge, as even in high-performing computers, the processing of a set of images of a large-scale PV power plant (that consists of some gigabytes of data) can take hours when using a DL algorithm. In the same way as in the detection of PV systems and modules, many types of outputs for the algorithms are possible, namely the segmentation of the faults, the detection of damaged modules or even the classification of faults in separate classes. The classes also differ among authors, and these differences represent a great challenge not only for the comparison between studies, but also for the exchange of data, experiences and algorithms among researchers in PV community, which hinders the advancements in this area. The exchange of data to enable the development of larger and more generalized datasets that consider different environmental conditions is also deaccelerated by data protection clauses.
Besides the different result types, the different evaluation metrics (or the lack of them), dataset sizes and image resolutions of the inputs also make the comparison between studies difficult. However, in general, the algorithms with the highest metrics are the ones dedicated to detecting and classifying a few types of faults compared to those that carry out the classification of many classes of faults. This proves that detection and classification of multiple faults is a complex task and further investigation is required. On this subject, the algorithm developed by Bommes et al. [88] can be highlighted for its encouraging results, with an accuracy of 90% in the detection and classification of faults in ten different anomaly types. In summary, to achieve the goal of an entirely autonomous aIRT procedure, advances in some of the tasks related to the technique must be achieved. Even tasks that were already the focus of many research studies, such as the detection and classification of faults, should be further explored to contemplate different types of datasets and conditions. The exchange of data and academic collaborations are fundamental to allow for a fully automatic procedure that not only detects modules and faults on PV modules but also provides information about the type and location of the faults, in a simple and accessible manner, to enable quick remediation measures.

Conclusions
This paper has conducted a comprehensive review of the literature for methods of automating different tasks of the aIRT framework of PV power plants, since it is a subject that has been intensely investigated by researchers in recent years. Most of these studies (77 studies) focused on the autonomous fault detection and classification of PV plants in visual, IRT and aIRT images. Among these studies, the use of DL algorithms has provided good results with an accuracy of up to 90% in the detection and classification of faults in 10 different anomaly types detected in module segments extracted from aIRT images. On the other hand, only a few studies have explored the automation of other parts of the procedure of aIRT, such as the optimization of the path planning (nine papers) for the inspection flight, the orthomosaicking of the PV plant (14 studies) that is performed to facilitate the localization of the faults in the field and the detection of soiling, and its differentiation from actual faults on PV modules (eight studies). Algorithms for the detection and segmentation of PV modules were presented in 38 papers and achieved a maximum F1 score of 98.4%.
For the automation of these procedures, different algorithms have been investigated, including DIP filters and methods such as canny edge detection and thresholding; DL algorithms such as Fast R-CNN, ImageNET and VGG16; and other ML-based algorithms used for classification tasks such as SVMs, KNNs and RFs. However, the accuracy, robustness and generalization of the developed algorithms are still the main challenges of these studies, especially when dealing with more classes of faults and the inspection of large-scale PV plants. With the ever-increasing capacity and size of utility-scale PV power plants, reaching the scales of gigawatts and hundreds of hectares, automation is increasingly becoming a matter not only of scientific interest, but also of economic importance. Therefore, the autonomous procedure and classification task must still be explored to enhance the accuracy and applicability of the aIRT method.