Towards a Holistic Approach for UAV-Based Large-Scale Photovoltaic Inspection: A Review on Deep Learning and Image Processing Techniques

Abstract

This paper provides an in-depth literature review on image processing techniques, focusing on deep learning approaches for anomaly detection and classification in photovoltaics. It examines key components of UAV-based PV inspection, including data acquisition protocols, panel segmentation and geolocation, anomaly classification, and optimizations for model generalization. Furthermore, challenges related to domain adaptation, dataset limitations, and multimodal fusion of RGB and thermal data are also discussed. Finally, research gaps and opportunities are analyzed to create a holistic, scalable, and real-time inspection workflow for large-scale installation. This review serves as a reference for researchers and industry professionals to advance UAV-based PV inspection.

1. Introduction

Photovoltaic (PV) energy constitutes one of the principal renewable and clean energy sources worldwide, with a cumulative global capacity of 1642 GW at the end of 2023 [1]. The capacity additions utilizing renewable electricity sources reached 510 GW in 2023, representing the most rapid growth in the last two decades at 50% [2], with PV-based electric energy accounting for three-quarters of it. It harnesses the sun’s potential and converts it into electrical energy via the photovoltaic effect [3]. However, installed PV systems may fail to deliver the estimated power in the event of faults and defects that impede their performance. Anomalies can occur due to environmental factors such as dust, foreign objects obstructing the received irradiation, and snow [4]. The PV systems can also experience electrical issues that affect their components and cause energy dissipation [4]. Furthermore, degradation over time can result in energy loss and increase the discrepancy between the estimated and produced energy [5]. Consequently, researchers are currently focusing on preventive and predictive maintenance to detect these anomalies in the preliminary stages. Thus, they adjust their O&M operations and take action to prevent energy loss that affects the Return On Investment (ROI) [6]. Nevertheless, detecting and localizing these anomalies in large-scale installations using manual techniques and electrical data can be challenging [7]. The former depends on the available human resources, the area covered by the installation, and the tools deployed for inspection (IR camera, I-V tracers, etc.). The latter can detect anomalies at the string level, which necessitates the inspection of individual panels in the affected strings. Electrical approaches are intrusive and insufficiently rapid to inspect large-scale installations and require expert knowledge to interpret and inspect effectively [7].

1.1. Types of Anomalies and Detection Methods

An anomaly, also referred to as a fault or defect, is a deviation of the system from its normal behavior, sudden breakdown, or power loss due to various factors such as environmental stresses [8]. Each anomaly possesses a distinct signature and pattern that enables experts to determine its nature and cause. Consequently, they select the appropriate tools required to detect each type effectively. Electrical data that characterize the system’s output facilitate the detection of abnormal behavior in comparison to normal behavior under specific weather conditions. In offline scenarios, this can be detected using I-V tracers. These instruments generate the I-V and P-V characteristics while measuring the temperature, irradiance, and standard curve under Standard Test Conditions (STCs) [9]. This enables operators to analyze the differences between curves and predict anomalies affecting the system. Analysis of the I-V curve can aid in detecting electrical component defects and environmental factors such as aging, partial shading, short circuits, and cracks [9,10,11]. However, this method is time-consuming, requires human resources for large-scale installation, and necessitates the system to be offline to measure the performance of the tested panels, which results in additional energy loss [12]. Figure 1 illustrates an example of I-V curves with different crack (a) and hotspot (b) representations.

Detecting and classifying anomalies using online data requires more sophisticated data analysis methods, such as thresholding and machine learning algorithms, to utilize their automation. The study presented in [13] adopted a system comparison between simulated and actual DC outputs. It employed a fixed threshold to analyze the free fault system. The same approach was proposed in refs. [14,15].

This detection method requires a large amount of labeled data to create a robust machine learning pipeline, which makes it challenging to create or simulate large-scale fields [16]. Thresholding and classical programming overfit the data handled, and generalization to other systems or installations cannot be easily performed. Electrical methods are sensor-based and can be expensive if operators intend to cover a vast area of photovoltaics [17]. Therefore, the most common data acquired concern the strings, and the inspection is consequently focused on the strings and not the individual solar modules.

Some anomalies can be detected through RGB, Infrared (IR), or electroluminescence (EL) images. RGB images detect visual artifacts and defects observable by the human eye. In IR images, each anomaly affects the temperature of the affected components [18,19]. The faults cause the energy to dissipate as heat, and the cell, substring of cells, or whole module appears overheated compared to its normal state. The overheating pattern enables experts to differentiate and distinguish between anomalies and characterize each to act accordingly. However, the need for another image type to further interpret the anomaly has been recommended [20]. EL is primarily adopted for indoor tests, which are based on the principle of measuring the light emission by cells when a current is fed into them [21]. It aids in detecting manufacturing defects, such as microcracks, which are difficult to detect visually or using IR imaging. This method is expensive and challenging to apply outdoors; consequently, testing requires dismounting individual modules from the field [19]. RGB and IR images can be implemented outdoors and indoors, respectively, to detect different anomaly types.

As

Table 1. Anomalies’ descriptions and the image types used for their detection.

Anomaly	Description	Image Appearance	Image Type for Detection
Cracks and micro-cracks	Cracks can occur during installation and maintenance or because of the climate (snow and ice pellets). Mechanical and thermal stress can cause cracks in the panel’s surface [22]. Soldering during production can damage the ribbon, and pressure on the silicon wafers can cause cross-crack lines [21]. In addition, impacts against a rigid object can generate cracks on the edge of the cell.		Thermal images (IR), electroluminescence images (EL), and RGB images
Delamination	Deterioration of the back sheet and the encapsulant (ethylene vinyl acetate [EVA]) allows water and oxygen to penetrate the successive layers of the PV module. This problem can be correlated with reduced solar energy transmitted through the system since the system cannot transmit it optically. This causes a problem with heat dissipation and, therefore, an increase in cell temperature [23,24].	(Image from [25])	IR–RGB
Discoloration	It is a change in the color of the cells towards yellow or brown due to the reduction in the performance of Ethylene Vinyl Acetate (EVA) (chemical reaction), which takes place due to UV radiation combined with water at a temperature of 50 °C. It can also occur due to prolonged exposure to high temperatures and low quality of the encapsulant [17]. This affects the transmittance of the system and, therefore, reduces its production.		IR–RGB
Shading	Shading, which can be caused by buildings, trees, dust, or even snow, reduces the radiation received by the system and, therefore, energy production. This causes high-temperature cells and hotspots. The temperature is lower in partial shading than in total shading [24].		IR–RGB
Soiling	Dust accumulation has a reduced effect on performance, but can worsen if not dealt with promptly. A dust storm can reduce system output by 20% and reach 50% if not cleaned [26].		IR–RGB
Activated diode bypass	The bypass diode solves the interconnection problem. The diode can malfunction due to increased temperature [8]. In this case, the fault can be seen in the thermal image when 1/3 or 2/3 of the module is hotter than the other parts of the same module [27,28].		IR
Short circuit	Short-circuited cells result from interconnection faults. They manifest themselves on thermal images by one or more rows being hotter than the others in the same panel [28] and by a patchwork of overheated cells spread across the short-circuited module [25,29].		IR
Hotspots	Hotspots occur when a cell produces less energy than the cells in the exact string [30]. They are not explicitly linked to specific defects, but they do warn of failures such as delamination points, cracks due to mechanical stress, corrosion, soldering defects, and others [31].		IR
Offline module	Interruption of a complete string or panel connection to the system due to fuse connection failures or junction box problems prevents many modules from being switched on [31].	-	IR
Cells	The panel’s hot region has the cell’s shape in the module. This could be due to shadowing, a defect in the cell, or delamination, which requires visual inspection to detect the probable cause [25].		IR–RGB
PID	This is distinguishable when cells closer to the frame are warmer than those in the middle [25].		IR

indicates, these anomalies are occasionally interconnected, and each may be related to another. Soiling and shading exhibit nearly identical thermal responses in the IR images, as both phenomena cause the affected area to overheat. A diode may be faulty or activated due to soiling covering the module’s surface. Furthermore, cell overheating can result from cracks, shadowing, or delamination. These causal relationships between anomalies are summarized in an illustration in [8] (see Figure 2). Consequently, RGB images are necessary to interpret and identify the causes of thermal anomalies.

These types of images can be acquired by handheld cameras to manually inspect the PV systems or embedded in UAVs to expedite the time of acquisition and take advantage of the large coverage provided by these aerial vectors.

1.2. UAVs as a Fast Inspection Tool

Interest in imaging to detect and classify anomalies in photovoltaic installations has increased as the prices of cameras have decreased and their availability has increased [32,33]. Conversely, to solve the problem of coverage of large-scale installations, industrial UAVs equipped with both types of cameras have emerged to reduce the time and cost of operations. These aerial vectors can capture high-resolution images from an altitude that can be adjusted according to the precision required for the inspection mission. In [34], the optimal altitude for the mission was approximately 5 to 20 m, depending on the anomaly type, using Nikon-V1 for visual faults. Similarly, refs. [35,36] specified the Ground Sampling Distance (GSD) required for a comprehensive inspection using RGB and IR cameras to meet the requirement of [37], which identified a GSD of 3 cm/pixel for the thermal camera and 1 cm/pixel for RGB. In [38], a minimal spatial resolution of 5 × 5 pixels per cell was specified.

Inspections using drones are significantly more cost-effective compared with field inspections using handheld cameras or electrical tools, which require substantial time and human resources. Regarding time efficiency, ref. [39] compared manual inspection to UAV-based inspection of 20 MWp installations equipped with 52,630 solar modules of 380 Wp. The comparison demonstrates that the UAV can perform a four-hour inspection flight, whereas human inspection can take up to 220 h. In terms of cost-effectiveness, utilizing UAVs for inspection is estimated to result in savings of up to USD 7.4 million per year in a 1 GWP plant [40].

1.3. Image Processing Techniques for Anomaly Classification

These imaging tools can detect and classify anomalies either manually or visually by operators or automatically using classical or advanced artificial intelligence algorithms such as machine learning and deep learning. For image processing, visual inspection of the acquired images has proven effective with the requisite expertise, as each type of anomaly can be analyzed and distinguished from the others. However, using UAV for acquisition produces a considerable amount of data that depend on the area covered by the installation and the altitude fixed for the operations. Therefore, visual inspection of images can be challenging for operators, as they are susceptible to errors when the amount of data is significant. Regarding automatic image processing, classical approaches rely on pixel value analysis. RGB images represent the light reflected from objects in the Red, Green, and Blue spectral ranges. Meanwhile, IR sensors, which operate in the short-wave infrared range, are commonly embedded in industrial cameras and represent the thermal response of objects within the scene.

1.3.1. Conventional Methods

Classical computer vision employs knowledge-based programming that utilizes manual feature extraction of pixel values or the temperature of the affected locations in images. Subsequently, thresholds are established for each image channel to automatically detect and classify anomaly types [17,41]. Additionally, geometrical properties can facilitate the detection and classification of defects. Edges, contours, and shapes are frequently utilized for the classification of anomalies in photovoltaics, as certain anomalies can cause abrupt shifts in pixel values and temperature or possess specific shapes (cell shapes), which aids in distinguishing them from normal cells [18,30,42]. However, this level of automation exhibits limited generalization capabilities. Thresholding and morphological operations are data-dependent and become invalid if the image properties differ. This occurs when environmental conditions vary (time of shooting, weather conditions) and depends on the installed photovoltaic technology. Consequently, these methods are not end-to-end and are valid only for specific features [43].

1.3.2. AI-Based Methods

Artificial intelligence methods utilize neural networks to interpret images and associate them with provided data labels, thereby establishing rules and knowledge for subsequent inferences and classification. This approach differs between machine learning and deep learning methodologies. The former necessitates human intervention prior to training by introducing salient features that characterize each image in the dataset. These features are subsequently input into the network along with the corresponding class labels. This approach requires feature engineering and selection to identify relevant attributes, ensuring model generalizability and effective class distinction. Conversely, deep learning methods exhibit a higher degree of automation compared to machine learning methods. This model conducts feature extraction without human intervention. However, to fully leverage the potential of this approach, deep learning models require substantial amounts of data to recognize and categorize each class within the datasets [44,45]. Moreover, increased data availability facilitates the achievement of robust generalization, provided the model is appropriately trained and optimized. Nevertheless, deep learning remains the most suitable tool for inspecting and classifying photovoltaics in large-scale installations. This has been empirically demonstrated in previous studies, where deep learning outperformed machine learning and classical computer vision algorithms [46,47].

• Training and finetuning:

Deep learning algorithms, particularly Convolutional Neural Networks (CNNs), can simulate operator decision-making in classifying the images received as inputs. This simulation requires adjusting the weights and biases (see Figure 3) of each convolution layer in the network to optimize the decision made by the model [48]. The optimization process focuses on reducing the margin between the correct class and the predicted class obtained by the model at each iteration. This margin is quantified by a loss function that measures it. The feature extraction component comprises convolution layers that filter images to extract features relevant to the decision step (feature maps). In deep learning, the model determines the features by adjusting the filter values (weights and biases) in each iteration to reduce the loss function. Furthermore, these values are adjusted through the backpropagation process, where the difference between the correct output class and the predicted class is considered in the optimization process through an optimizer selected according to the complexity of the problem. After each convolution layer, an activation function introduces nonlinearity and complexity in the architecture and detects the features relevant to each class. A pooling layer is commonly incorporated to condense the information and reduce the output size of each convolution layer. As the model output is the probability of an image belonging to an output class, the 2D feature maps are converted to 1D values through the fully connected layers [46,48].

Figure 3 illustrates a typical deep learning architecture for image classification. However, numerous advancements, such as residual blocks [49], dropout [50], and regularization layers [51], have been introduced to optimize the learning process and enhance the efficiency of the trained model for detection.

The selection of hyperparameters and the initial training point can optimize the learning curve of the model. This necessitates skilled users to possess knowledge while adjusting and fine-tuning the hyperparameters [52]. However, it requires manual execution of trials and testing, which can be challenging and time-intensive.

An alternative approach for deep learning model training involves utilizing pre-trained models on similar or different tasks and fine-tuning them to the specific task at hand [53]. This method leverages existing architecture and pre-trained weights as starting points while further optimizing them based on the hyperparameters established during fine-tuning. Subsequently, the final trained model is deployed to process new input images and generate predictions. This process of applying the trained model to unseen data is referred to as inference.

• Generalization and optimization for real-time applications:

For datasets required for training, the quantity of images for each class is essential to improve not only the model’s performance, but also the diversity in representation of the objects to classify, the data quality, and, in some instances, the modalities introduced as input to the model. To enhance generalization to unseen data, utilizing data captured in different timeframes and for various types of instances within the same class can improve the robustness and generalization rate. Consequently, it can perform more effectively on unseen data and mitigate the domain shift effect. Furthermore, analogous to human interpretation, having multiple data sources enables significantly more insight into the data and better characterizes the classes concerned [12,43].

To incorporate deep learning classification for large-scale installation monitoring, optimization for real-time inference should be addressed to further enhance the operation. This encompasses not only the inference speed of the algorithms, but also the efficiency of their execution in real time across various devices, including those with limited computational capacity [54,55].

1.4. Aims, Contributions, and Paper Organization

Anomalies impacting PV systems can be detected through electrical data analysis and manual inspection to identify defects at the string and panel levels, respectively. However, these methods are limited in their scope, requiring significant effort and cost for comprehensive assessments. Consequently, UAV inspections combined with DL have gained traction, utilizing IR and RGB images for anomaly detection and classification. While prior studies have classified both image types, the potential of combining RGB and IR data to enhance classification remains underexplored. It can improve failure detection and deepen understanding of underlying causes [12,43].

Despite advancements in deep learning models for PV inspection, their efficacy depends heavily on the availability and quality of training data. A robust classifier should handle images from diverse environments to detect anomalies and overcome domain shifts. However, existing datasets often suffer from low resolution, limited diversity, and class imbalance. Model selection and optimization are also critical for real-world performance.

Another critical challenge is to complement electrical with individual PV module inspection, focusing deep learning on module-level analysis. PV module segmentation should undergo similar optimizations for effective deployment. Large-scale PV installations necessitate a geolocation step to facilitate in-field inspection. In previous studies, the geolocation of the extracted modules was based on photogrammetric processing, which is computationally intensive and unsuitable for real-time applications. Thus, individual module geolocation from raw UAV images represents a significant improvement opportunity.

This paper presents a comprehensive review of literature on anomaly detection and classification in PV modules. This state-of-the-art (SOTA) study encompasses recently adopted methodologies for segmenting and diagnosing PV systems utilizing UAV, satellite, and handheld camera images. It examines both classical and advanced computer vision techniques, evaluating their efficacy, limitations, and recent advancements.

Beyond anomaly detection and classification, developing a holistic UAV-based PV monitoring approach requires addressing the optimization of data acquisition, preprocessing, storage, and visualization of results. Consequently, this study complements previous review studies [4,8,16,19,32,56] and focuses on these aspects to identify the best practices and highlight research gaps that future studies may address to develop a holistic approach for PV inspection (see Figure 4). A comprehensive review of SOTA methods will facilitate the assessment of current requirements and identify unexplored solutions to enhance the decision-making process of deep learning models. It will also serve as a foundation for future research, guiding researchers toward the development of a holistic approach for large-scale installation inspections using UAV imagery.

To underscore the relevance of this paper in relation to existent literature reviews, this initial section outlines the context and rationale underlying it. The subsequent sections provide a comprehensive analysis of the literature pertaining to anomaly detection and classification in large-scale photovoltaic installations through the utilization of drone imagery. Ultimately, the paper delineates and discusses research gaps and areas for potential improvement, intending to contribute to the development of a holistic approach for monitoring large-scale installations using UAV data.

2. Guidelines and Best Practices for UAV Data Acquisition in PV Inspection

The inspection of PV modules through UAV thermal and RGB images necessitates adherence to specific guidelines. For thermal camera usage, the IEC Technical Specification 62446-3 [37] establishes the protocols for outdoor infrared thermography and delineates the considerations that must be observed regarding minimum camera specifications, suitable environmental conditions for image capture, and imaging procedures. It also encompasses instructions for visual cameras, which provide valuable supplementary insights into thermal imaging.

Moreover, previous studies have focused on elucidating data acquisition guidelines to ensure successful inspections and facilitate the identification of anomalies in the acquired images. These guidelines encompass camera attitude and distance to the panel to enhance anomaly detection.

Firstly, the thermal camera should meet minimum specifications to fulfill the requirements, such as a thermal object temperature range from −20 °C to 120 °C and the ability to function in ambient temperatures between −10 °C and 40 °C [37,38]. The inspection should be conducted at an irradiance exceeding 600 W/m², with wind speeds below 28 km/h, and cloud coverage of less than two oktas of cumulus clouds to mitigate undesirable reflectance on the module surface [37,57]. These conditions are essential to prevent noisy and blurred thermal images that may compromise the accuracy of the thermal response and panel representation [43].

The attitude of the UAV for aerial inspection should be adjusted to prevent sun glare on the module surface and self-shading of the module by the UAV. To this end, the technical specifications stipulate maintaining a perpendicular angle during data capture. For handheld cameras, where ensuring a perpendicular angle may be challenging, an angle greater than 30° is recommended [37]. The angle of acquisition should be perpendicular to and exceed 60° relative to the normal of the solar module surface [58]. For both automatic and manual inspections, the flight should be conducted parallel to the arrays [35,36].

The altitude at which the mission should be conducted depends on the camera resolution, the GSD at which anomalies are identifiable through the images, and the types of faults under investigation. In this regard, refs. [20,34,36] estimated that the optimal flight height for anomaly inspection ranges between 5 and 20 m. Lastly, the technical specifications require operators to indicate the precise locations of detected anomalies in their reports.

3. Data Preprocessing and Preparation for the Inspection

Before image classification and segmentation, the process begins with preprocessing images to enhance relevant features that improve data quality and deep learning capacity. In addition to preprocessing, the data preparation incorporates geolocation as a step to facilitate in-site inspection and data curation to mitigate the imbalance impact.

3.1. UAV Image Pre-Processing

Image processing aims to reduce noise and enhance image quality and features to facilitate detection. Moreover, the data utilized for advanced image classification should be curated to optimize model learning. To this end, previous attempts to detect and classify anomalies have proposed various strategies for data preprocessing.

To improve classifier performance, manual removal of background objects and image cropping have previously been adopted to reduce false detection of anomalies. Refs. [53,59] adopted this approach in addition to resizing to obtain uniform image size. Similarly, ref. [60] applied cropping, scaling, and interference deletions. Ref. [61] converted 16-bit images into 8-bits before normalizing them to the range of 0 to 255. Standardization was performed to reduce the domain shift within the dataset.

Computer vision techniques are employed to denoise and augment data quality before classification or segmentation. Ref. [59] applied thresholding, Sobel filter, and wavelet transform before data usage in CNN training. Ref. [62] performed noise removal, reshaping, and resizing to improve the performance of the proposed CNN. Refs. [17,63] employed Gaussian blurring before filtering. Ref. [64] evaluated the impact of preprocessing before fault detection using image preprocessing. The results showed an improvement in accuracy when using a bilateral filter combined with histogram equalization. Ref. [65] used a Laplacian filter for image sharpening and discontinuity removal. After contour detection using traditional computer vision, some studies adopted morphological opening and closing to remove noise from the detected mask [63,66]. Pixel value normalization is also added to machine and deep learning to speed up the convergence of the model during training [48,59] and to enhance image features [60]. Normalization was also adopted in [67] to reduce the variance in the temperature values in the images.

3.2. Image Geolocation

Some studies have focused on the geolocation of the detected PV anomalies; therefore, the preprocessing of these images includes photogrammetric processing to generate orthorectified images [29,68,69] and SFM–MVS to rectify the geolocation errors [70]. To keep track of the individual PV modules in each video frame, ref. [71] employed ORB key point detection and RANSAC for outlier removal. Consequently, they linked the ID of each PV module to the previously localized plant ID. Ref. [41] presented a method for the local positioning of modules in the acquired videos. In this context, these methods do not provide the geolocation of the extracted modules and are limited to local positioning. Moreover, photogrammetric processing, which is efficient and precise for geolocation, necessitates extensive computational resources.

3.3. Addressing Data Imbalance and Augmentation

Deep learning algorithms provide enhanced automation for classification. They surpass traditional machine learning methods through automatic feature extraction, which facilitates the characterization of each class. However, this advantage necessitates a large dataset with sufficient samples to represent each class [46]. To address this issue, previous studies have employed data augmentation to expand datasets. Data augmentation is one of the solutions that addresses the problem of insufficient data in machine learning tasks, particularly in multiclass classification. This involves incorporating slightly modified replicas that closely resemble the original data [48]. Additional data are generated using geometric and radiometric operations such as rotation, translation, flip, Gaussian filter, and Fourier transform. The generated images have labels identical to those of the original images because they represent potential variations that could occur under different circumstances of data acquisition for a similar application. For example, refs. [71,72] augmented the dataset using geometric operations to increase the number of images. Ref. [53] improved classifier performance by training the model on an augmented dataset.

Moreover, large datasets acquired for the inspection of photovoltaic installations are generally skewed toward the normal class. This is due to the high number of normal panels compared to faulty ones. Therefore, the trained model can be biased toward a more represented class and performs poorly in classifying classes with a low number of images and in the dataset used for training. The techniques proposed and employed in previous studies to address this issue include weighting, data augmentation, undersampling, and oversampling. Data augmentation was previously employed by [73,74,75,76] to achieve data balance. For instance, ref. [73] applied data augmentation to all 12 classes in the dataset [77], whereas [74,75] applied data augmentation exclusively to anomalous classes in the context of multiclass classification.

An alternative approach to addressing imbalance is to undersample the classes that have more images and potentially cause the model to be biased toward them. In their prior research, refs. [74,75] opted to maintain the number of images in the dominant class. Instead, they chose to exclude the more frequently occurring class from the multiclass classification, focusing on the anomalous classes. One disadvantage of undersampling is that a significant quantity of data must be disregarded to establish the required equilibrium. The scarcity of publicly available datasets for anomaly identification in photovoltaic plant inspection reduces the desirability of this approach.

Conversely, oversampling is another solution for mitigating the impact of imbalance on the dataset. Oversampling algorithms allows users to increase the number of images per class by generating similar images based on the original data. The SMOTE approach [78], also known as the synthetic minority oversampling technique, was employed by [73] to generate new images using the k-nearest neighbor (K-NN) algorithm [78]. The selection of the optimal number of neighbors required to generate a new sample depends on the intended number of images and the number of images in the minority class.

To handle data imbalance, our previous study [79] proposed a model-centric approach combined with data augmentation to improve the multiclass classification of imbalanced datasets. It proposes intermediate classification and leverages lower levels of classification to enhance the final classification decision.

4. Photovoltaic Module Segmentation and Identification

Photovoltaic module segmentation and identification have been introduced in previous studies before diagnosis to establish the region of interest for subsequent analysis [80]. Solar module segmentation facilitates the identification of distributed photovoltaic systems.

Initially, classical computer vision algorithms were employed for this task. These algorithms are based on filtering, thresholding, or a combination of both to segment individual panels. Filtering methods utilize colorimetric and radiometric techniques to delineate the features of individual panels and eliminate background noise. Using geometrical filters, refs. [17,42] segmented photovoltaic modules from thermal videos. Ref. [81] detected PV strings using Sobel and Canny detectors to detect individual modules within strings. Ref. [82] leveraged the rectangular shape of PV modules to filter the segmented modules from RGB images converted to grayscale. Ref. [83] employed edge detection to segment PV modules from RGB images. The utilization of geometrical attributes was overfitted to the datasets in these studies, and filtering based on module shape was dependent on the size of the modules in the image space. This shape varies with the mission altitude and the drone attitude during the mission. This variation affects the accuracy of the approach and increases the false detection rate. Ref. [80] incorporated a post-processing approach to address missing modules. They previously identified these modules by edge detection and geometric properties while adopting SVM to eliminate false positives. Other studies focused on utilizing colorimetric features. Ref. [63] applied thresholding on RGB images converted to HSV color space, and ref. [84] employed MATLAB’s Find peaks function to detect the boundaries of the modules and filtered the results with DBSCAN. Ref. [66] used automatic thresholding to generalize this approach. In this study, the Otsu method was implemented to segment photovoltaic modules.

However, interest in a more autonomous approach for individual solar module segmentation has increased [85]. This approach not only enhances the performance of the operation, but also demonstrates greater potential for generalizability when adequately trained. Ref. [80] demonstrated the performance of mask RCNN compared with classical computer vision in terms of performance metrics and the shape of the generated masks. The same architecture was trained by [71] on thermal images extracted from videos, and by [86] on RGB images. Refs. [87,88] trained YOLOv3 to identify solar modules for inspection while proving the real-time integration ease. Meanwhile, ref. [89] fine-tuned YOLOv5 to segment solar cells from RGB and IR images. Ref. [90] used Faster RCNN on the orthorectified tile to identify the PV modules. Ref. [91] used U-Net for array extraction from thermal images.

On the other hand, some efforts were made to segment individual PV modules and installations for the identification areas equipped with this type of installation. Ref. [92] proposed Solar Mapper based on U-Net to detect Photovoltaics from satellite images. Solar Finder was proposed by [93] for the same application. This solution is based on the combination of ML and DL to improve the detection and identification of panels within the rooftop contours. In ref. [94], the researchers were interested in detecting arrays using Mask R-CNN and ResNet-50 as the backbone. Ref. [95] proposed a GeoAI based on U-NET and VGG16 for the same application. To detect installation boundaries, ref. [86] adopted Mask R-CNN and [96] trained U-NET for the same application.

5. Deep Learning for Anomaly Detection and Classification Using UAV Images

Previous studies have addressed the problem of photovoltaic system inspection using either traditional computer vision or advanced algorithms based on ML or DL. However, deep learning approaches have demonstrated superior performance compared to conventional methods in terms of performance metrics and automation capabilities [47]. For instance, ref. [97] suggested that DL-based methods outperform ML and traditional techniques. The latter methods exhibit lower accuracy and fail to generalize when applied to novel faults. In this regard, previous researchers have investigated anomaly detection using RGB images [72,83], IR images [71], and EL images [76,98]. On the other hand, the combination of multiple data sources remains recommended [12], but has been underexplored in previous studies.

To classify anomalies based on their thermal nature, ref. [68] proposed FPN-DenseNet architecture on orthorectified images. In contrast, the study proposed in [71] classified raw IR images using ResNet-50. Ref. [99] trained a lightweight CNN from scratch on IR images for anomaly classification. Ref. [100] classified thermal images using the proposed and custom CNN. Ref. [101] combined two architectures, U-Net and DenseNet, as U-DenseNet to leverage the advantages of both for classifying thermal images. Ref. [102] employed a Visual Transformer and demonstrated its performance compared to other DL approaches to classify IR images. Ref. [73] proposed a residual block-based CNN to classify a 12-class dataset of thermal images. It also used ensemble model performance by training several models using different strategies. Ref. [59] trained a binary CNN from scratch to classify thermal faults in two datasets acquired by a handheld camera and UAV.

Object detection techniques have been employed in numerous studies to identify and delineate faulty components in photovoltaic (PV) panels. One such investigation [72] adopted a modified Single Shot Detector (SSD) with a VGG16 backbone, effectively reducing classification time through a decrease in model parameters. In a related study, ref. [103] adapted and fine-tuned Yolov3-tiny for the detection of thermal anomalies in solar modules. Additionally, ref. [88] applied Yolov3 to segment hotspots within pre-segmented strings using an identical architectural approach.

Research involving RGB imagery has also yielded significant advancements. The DeepSolarEye system, proposed by [26], was developed to analyze and estimate dirt accumulation and soiling on PV module surfaces. Another study, [83], focused on training an encoder–decoder framework specifically for bird dropping detection. In contrast, ref. [104] employed VGG16 solely for feature extraction, followed by SVM classification to identify five distinct anomaly types in RGB images. Furthermore, ref. [105] introduced a stacking ensemble methodology for RGB image classification, aimed at detecting visual anomalies such as discoloration.

In EL images, their application was predominantly associated with manufacturing processes, such as the detection of micro-cracks. Ref. [98] proposed a concatenation model of Xception and ResNet50V2 to classify EL images. This study compared this model with VGG19, DenseNet201, and InceptionResNetV2, which are frequently utilized for image classification in photovoltaics. Vision Transformers were also introduced for the classification of EL panel images into four classes and by technology type [106]. Ref. [76] proposed a light-depth CNN and achieved 98% accuracy for 8-class classifications. Ref. [107] developed an improved Yolov7 with an ELAN block to detect faults in EL cells and monitor the manufacturing processes. Ref. [108] employed knowledge distillation from a pre-trained VGG16 to train a custom CNN to classify EL cell images.

Conversely, the integration of multiple modalities and methods has been addressed in previous research. Ref. [60] developed two Convolutional Neural Networks (CNNs) for the classification of infrared (IR) and RGB images, with decision-making conducted through cross-validation of both models. Ref. [109] proposed a weighted decision approach utilizing three models trained on IR, RGB, and meteorological data to classify anomalies. Ref. [26] combined RGB and environmental factors as inputs to predict power loss, dirt location, and dirt type.

The results reported by [104] suggest that binary classification exhibits superior performance compared to multiclass classification. Additionally, the researchers noted that the model demonstrated a propensity for shadowing due to the prevalence of images in this category.

6. Challenges in Model Generalization and Domain Adaptation

In the context of photovoltaic plant inspection using UAV images, additional factors must be considered when training a DL classifier. Generally, deep learning models are susceptible to generalization issues. Variations in data between training, testing, and real-world deployment can lead to model drift and diminished performance despite satisfactory results during training and testing [90]. Solar panels utilized in power plants exhibit numerous technological, generational, morphological, and dimensional variations [110]. Consequently, the absence of a particular type or representation of solar panel in the training data may result in suboptimal generalization rates.

Ref. [61] proposed contrastive learning to enhance the generalizability of the model and mitigate the impact of domain shift. Furthermore, the visual representation of solar modules in images, particularly in thermal images, is influenced by environmental factors such as irradiance, cloud coverage, mission altitude, and image resolution. These elements alter the thermal response of individual solar modules, which are represented differently in the image domain, resulting in a domain shift and a reduction in the generalization potential of the model if they are absent from the training data. Refs. [80,90] employed a training dataset comprising four distinct sites to address the domain shift. The researchers evaluated the trained model on an unseen site and achieved performance inferior to that of the model trained on a mixed dataset encompassing all sites. Ref. [111] proposed a GenPV model encompassing a range of resolutions and photovoltaic module sizes. The model employed data augmentation and feature pyramid networks to address imbalances in the resolution. The model was structured to accommodate modules of varying sizes through inductive transfer learning, and focal loss was utilized to balance the distribution of classes. Moreover, as indicated by the authors in ref. [91], the proposed model should be evaluated for its generalization capacity when applied to an expanded dataset. Ref. [61] implied that large datasets from different environments are required to achieve better results and mitigate the domain shift. Conversely, GenPV [111] performed poorly on an unaligned dataset, indicating the need for datasets from different environments. Our previous work [112] proposed a data-based domain generalization approach to improve the generalization of the segmentation models toward data from different installations. This approach enhances the performance of the model on unseen data by training it on different photovoltaic models acquired at different locations and environmental conditions. A retraining cycle was proposed in [113] to ensure the continuous adaptation and deployment of the model for future inference.

7. Towards a Holistic Approach for End-to-End UAV-Based PV Inspection Workflow

Prior research has endeavored to establish a comprehensive workflow for detecting and classifying anomalies, encompassing the entire process from data collection to visualization and decision-making. Researchers in [88,114] introduced a workflow that combined the segmentation and classification of thermal and RGB images. Another study proposed SunMap as an unattended maintenance approach for photovoltaic plants, utilizing a photogrammetric-based approach [69]. This method generates an orthophoto map of the inspected sites, facilitating identification of the affected solar modules. Each inspected image underwent thermal analysis. The study in [99] posited that the data were stored in Firebase. The retrieved images were subsequently processed on a Raspberry Pi and either displayed or transmitted to the user as a notification. Our previous work [115] developed a hybrid and scalable approach for monitoring PV plants that manages data acquisition, processing, inspection, storage, and visualization for O&M operations. The authors of [116] proposed a solution for monitoring multiple plants by relying on sensor data processing and AI-based power prediction using five distinct models. Furthermore, in [117], the researchers developed a platform that enables users to upload thermal images. These images underwent a two-step segmentation process for array detection, followed by hotspot identification. The platform subsequently generates a report based on this analysis.

8. Critical Analysis and Future Research Directions

The literature review indicates an increased interest in anomaly detection and classification in photovoltaics. These investigations aimed to preserve the advantages of solar energy and ensure the optimal performance of PV systems throughout their operational lifespan. Regarding the use of UAV images and imaging in general, the technical specifications delineate the protocols for a successful mission to avoid image blurring and unusable data for inspection. When implemented correctly, UAV imaging can reduce operational costs and enhance inspection efficiency.

Recent studies have also demonstrated interest in the utilization of DL algorithms for anomaly detection. The performance of such algorithms facilitates the automation of the process with high-performance metrics compared to classical computer vision and ML. Researchers have attempted to improve performance by proposing custom CNN architectures, pre-trained models, and hybrid approaches. Custom CNNs are based on previously developed blocks in the SOTA architectures, such as residual and attention blocks. Some previous investigations compared pre-trained usage through transfer learning and training models from scratch.

Other studies have incorporated the segmentation of PV installations, arrays, strings, and solar modules. This step enhances anomaly detection by defining the region of interest and reducing the background noise for the classifier’s decision.

While DL algorithms are dependent on data quality and quantity, previous research has introduced solutions for these aspects. Firstly, data scarcity in the field of solar energy, primarily due to confidentiality concerns, necessitates data augmentation to increase the number of images in training, either through geometric and radiometric operations or by adopting Generative Adversarial Networks (GANs) to synthetically expand datasets. Generally, the number of normal panels exceeds that of panels with anomalies in solar fields. Consequently, the photovoltaic panel classification dataset is imbalanced between the normal class and classes with anomalies. Regarding the generalizability of the workflow, it has been suggested that data integration from different installations should enhance the model’s robustness against domain shift.

In addition, the data types utilized for anomaly detection and classification are diverse, depending on the nature of the anomalies studied. However, RGB and thermal images are the primary modalities adopted for large-scale outdoor environments. Previous studies have examined both modalities separately to classify anomalies. More recent research has investigated multimodal approaches, including cross-validation steps and voting decision steps, between classifiers trained on each modality.

Finally, there has been increasing interest in developing a holistic approach that incorporates all essential components required for monitoring photovoltaic installations. This approach optimizes the practical aspects of image analysis for anomaly detection and classification by encompassing data acquisition, storage, anomaly detection, classification, and visualization modules.

However, a critical examination of the literature on anomaly detection and classification revealed several areas for improvement, as shown in

Table 2. Gaps in the SOTA for each component of the holistic approach for the inspection of large-scale installation.

Aspect	Study	Method	Advantages/Gaps
Data	[75]	Data augmentation (for anomalous classes), Undersampling (by discarding the dominant class in the multiclass classification)	The multiclass classifier discarded the normal class.
	[73]	Class weighting, Data augmentation, SMOTE	Lower accuracy in minority classes; potential overfitting with SMOTE.
	[74]	Data augmentation (for anomalous classes), Undersampling (by discarding the dominant class)	The multiclass classifier discarded the normal class.
	[76]	Data augmentation	Potential model bias due to insufficient variations (big difference in number between classes).
	[104]	-	Model bias towards shadowing due to image quantity; reduced performance in multiclass classification.
	[29]	Combination of undersampling, augmentation, and oversampling	It is a complex implementation and may not solve significant gaps between class frequencies.
	[79]	Cascading decision system	Takes advantage of the decision made on lower levels of classification, handling balanced classes; this improves the classification made in the multiclass classification.
Domain shift	[111]	GenPV to mitigate the domain shift by training on different resolutions and photovoltaic module sizes Uses feature pyramid networks to resolve imbalances in resolution	Results revealed poor performance on unaligned datasets.
	[61]	Contrastive learning	Large datasets from different environments are required to achieve better results.
	[80,90]	The dataset comprises four distinct sites	Lower performance on unseen data than on mixed dataset encompassing all sites.
	[91]	-	They affirm that the model should be evaluated for its generalization capacity when applied to an expanded dataset.
	[112]	Data-based domain generalization: data acquisition from different photovoltaic sites and in different environmental conditions	Improves models generalization toward unseen data, ensuring efficient inference in real-world settings
	[113]	Production-ready strategy with a retraining cycle for DL models to improve future inference	Improvement of the model using production environment data. However, there is a need to study the impact of newer data on performance.
Solar panel segmentation	[81]	String detection uses Sobel and Canny detectors to detect individual modules within strings	Classical computer vision techniques are not generalizable.
	[82]	Geometric properties and rectangular shapes of modules to detect single modules	The rectangular shape of panels is not generalizable to other datasets.
	[84]	Find peaks function in MATLAB	Classical computer vision techniques are not generalizable.
	[66]	Automatic thresholding using Otsu	A good attempt to try automatic thresholding; however, it is still not generalizable.
	[63]	RGB images converted to HSV + geometric properties (i.e., circumference) and shape to eliminate background noise	The properties of the panels depend on their shape in the image, which can be difficult to generalize.
	[80]	filtering and geometric properties of the solar panel + SVM to eliminate false positives	Missing panels and geometric properties based on image measurements
	[80]	Mask R-CNN + SVM to eliminate false positives	outperformed the classical, but required labeled data.
	[87]	Yolov3 for solar panel segmentation from thermal images	Fast inference is suitable for real-time detection, but requires labeled data.
	[90]	R-CNN on tiled visual orthomosaics	Photogrammetric preprocessing and labeling are required.
Geolocation	[71]	ORB key point detection and RANSAC for outliers’ removal	Localize each detected panel by its ID.
	[69]	Module’s ID linking to the plant ID to localize each image	Computational requirements not adapted for real-time application.
	[29,68]	Photogrammetric processing	Computational requirements not adapted for real-time application.
	[70]	Photogrammetric processing	Not suitable for real-time application.
	[60]	SFM–MVS to rectify the geolocation errors	Image stitching is required before inspection for fault detection.
	[69]	Classical image processing for module segmentation and two CNNs for the detection and classification of RGB and thermal images, respectively	Localization in image space and not geographical localization.
Anomaly classification	[118]	Multimodal classification in large-scale photovoltaic systems	The multimodality concerns only the features extracted from the electrical data.
	[60]	Two CNNs to classify anomalies in RGB and IR images	The combination of IR and RGB models was only used to cross-validate the decision of both models.
	[109]	Three models for the classification of RGB, IR, and meteorological data	The weighting makes the final decision of these models. There is no interpretation of the correlation between modalities.
	[89]	Yolov5 and ResNet for the segmentation of panels and their classification through RGB and IR images	Both modalities are processed separately.
	[88]	Yolov3 to segment solar arrays and classify faults in RGB and thermal images	Only one modality was used for this study. However, it produces a model with low inference for segmentation and classification.
	[101]	U-DenseNet based on DenseNet and U-Net combination for 12-class dataset classification	Proposed GAN for data augmentation to cure data imbalance; focuses only on IR modality.
	[105]	AlexNet for feature extraction, J48 for feature selection, and four stacked ensemble machine learning models for the classification	Interest only in visual faults such as discoloration.
	[119]	Yolov7 for anomaly classification	Interest in deployment in the production environment and the optimization needed was not studied adequately for real-world settings of PV plants.
	[73]	Residual blocks-based CNN to classify 12-classes dataset of thermal images and 15 model ensemble models	Several models’ combinations for the classification of anomalies, with an accuracy of 88%.
	[59]	Comparison of the detection and classification of images acquired by a handheld camera and a UAV	UAV image classification produced better results.
	[72]	Modified SSD by changing the backbone with VGG16 to detect faults in RGB images
	[26]	Combined RGB and environmental factors as inputs to predict power loss, dirt location, and dirt type
	[104]	Applied VGG16 for feature extraction only and then the SVM classifier for five anomaly classifications in RGB images	Binary classification outperforms multiclass classification in terms of performance.
Holistic approach	[88,114]	Provided a workflow for segmentation and classification using thermal and RGB images	The multimodality was not studied, and both modalities were processed separately.
	[69]	SunMap as a solution for unattended maintenance methods for PV plants	Based on statistical analysis of the thermal response to detect hotspots; used photogrammetric processing for segmentation and localization.
	[99]	IoT-based approach for image classification based on Firebase and Raspberry Pi	The implementation assumed the images were stored in Firebase and did not perform segmentation.
	[117]	Two-step analysis starting with the array’s segmentation, followed by hotspot detection using CNNs	They have provided a platform for uploading images to process and display results. However, they adopted a Single modality (IR), and no geolocation step was provided to localize anomalies.
	[116]	Sensor-based monitoring using five AI models for anomaly detection in multiple plants	The cost is high due to the need for sensors.

. Firstly, there were sampling deficiencies in the datasets used for training the DL models. Both publicly available and private datasets exhibit class imbalance, low resolution, or insufficient representation for each anomaly type. The performance in most studies exceeded 90% accuracy for binary classifiers and 85% accuracy for multiclass classification. However, these models have not been rigorously tested and deployed in production under varying environmental conditions. To address this issue, our previous study proposed a data collection protocol to create an optimized dataset for anomaly classification and solar module segmentation. This protocol is based on varying module representations, environmental conditions, and factors that impact the images.

Furthermore, automatic segmentation necessitates a large, labeled dataset for solar module segmentation. Labeling such datasets requires substantial time and effort, particularly for large-scale installations. No significant efforts have been made to automate the labeling process while making it adaptive to changes in the UAV shooting conditions. Moreover, the geolocation of the segmented modules using an optimized inference strategy for raw images should be incorporated into the process. This approach outperforms photogrammetric processing, which is time-intensive and computationally expensive.

Regarding the empirical gaps, multimodality usage for anomaly detection and classification remains largely unexplored. The human interpretation of anomalies via images relies on prior domain knowledge. It considers the difference in patterns between anomaly types and depends on the RGB images for further insights when infrared images alone are insufficient. However, in previous studies, models were developed and studied separately. The combination included a voting stage between both modalities and cross-validation.

Lastly, the interest in a holistic monitoring approach should prioritize real-time inference optimization and streamline the entire execution pipeline to facilitate the deployment of the trained models in production environments.

9. Conclusions

This work provides a comprehensive assessment of the SOTA of photovoltaic inspection through advanced UAV image processing. It focuses on data acquisition, preprocessing, solar module segmentation, and classification to guide future studies in their critical analyses, to propose novel holistic approaches, and to address the issues that affect each component. The components discussed in this review have been demonstrated to be essential for improving the performance of inspection approaches, their adaptability to real-world applications, and their optimization for real-time execution. Integrating the segmentation and classification steps while optimizing the data acquisition workflow enhances the performance of the inspection operations. However, the efficiency of their execution in terms of production and generalization capacity remains a significant challenge. The images used for segmentation and classification are primarily classified separately, encouraging the study of hybrid and data fusion for both applications to assimilate human operators’ logic and analysis for the decision-making process.

In this regard, future research should focus on introducing multimodal inputs to further optimize the decision-making processes of the developed models, in addition to increasing interest in improving the models’ generalization and adaptability to different photovoltaic systems and environmental conditions. In terms of implementation, proposing a holistic approach from data acquisition to inference in real-world settings would provide more tangible benefits to the field. Finally, the trend toward unattended monitoring motivates the optimization of real-time inferences for expeditious and streamlined inspection.