Infrared–Visible Multi-Sensor Fusion for UAV Photovoltaic Defect Detection Under Real-World Weak Misalignment

Highlights

What are the main findings?

• Frequency-Aware Selective Fusion: Infrared thermal responses and visible structural textures are selectively fused to reduce misaligned-edge interference and background mis-fusion under weak cross-modal misalignment.
• Lightweight FAF-YOLO Detection Framework: FAF-YOLO improves photovoltaic defect detection accuracy and robustness while reducing model parameters and computational cost.

What is the implication of the main finding?

• Registration-Free UAV Inspection: Reliable infrared–visible photovoltaic defect detection can be achieved without strict frame-by-frame image registration.
• Edge-Deployable Real-Time Detection: The proposed method provides a lightweight and deployable solution for real-time photovoltaic inspection on resource-limited UAV edge platforms.

Abstract

For large-scale photovoltaic plant inspection, UAV-based infrared–visible real-time detection can combine thermal abnormality information with appearance and structural cues. This is useful for improving inspection and maintenance efficiency. However, in real UAV inspection, differences in sensor resolution, field of view, and flight attitude can cause weak misalignment between the two modalities. Since complex image registration is difficult to perform before real-time inference, this misalignment can affect cross-modal feature fusion and defect localization. To address this problem, this paper proposes Frequency-Aware Fusion YOLO (FAF-YOLO) for dual-modal photovoltaic defect detection. We also build a real-scene infrared–visible dual-modal photovoltaic defect dataset, named DM-PV, which covers six defect categories related to thermal anomalies and external environmental interference. FAF-YOLO is based on a dual-branch YOLO detection framework. The C3k2-DPRG module is used to enhance defect boundaries, local details, and neighborhood context. The Frequency-aware Selective Fusion (FSF) module models low-frequency structural information and high-frequency detail responses separately, which reduces edge ghosting and background mis-fusion caused by weak misalignment. A Multi-Scale Differentiated Decoupled Head is then used to handle scale-specific prediction and improve small-defect localization and regional-anomaly discrimination. Experimental results show that FAF-YOLO achieves 92.5% Precision, 86.7% Recall, 91.7% ${\mathrm{mAP}}_{50}$, and 61.4% ${\mathrm{mAP}}_{50:95}$ on the DM-PV dataset. It outperforms several mainstream dual-modal detection methods and has lower parameters and computational complexity. Further tests for real-time inspection show that the proposed method keeps more stable performance under weak misalignment perturbations. It also reaches an inference speed of 33 FPS on the Jetson Orin Nano edge platform, which verifies its effectiveness and deployability for UAV-based real-time photovoltaic inspection.

1. Introduction

Photovoltaic (PV) power plants are an important scenario in infrastructure inspection. With the rapid growth of PV plants, efficient and timely defect detection has become essential for maintaining power generation efficiency and system safety. PV modules are exposed to outdoor environments for long periods. They are affected by mechanical stress, environmental erosion, ultraviolet radiation, and other factors. These factors may cause abnormal conditions such as hot spots, diode failures, and surface contamination. If these abnormalities are not detected in time, they may reduce output power and energy conversion efficiency, accelerate material aging, and even lead to safety risks. Therefore, timely and accurate detection of PV abnormalities is important for PV operation and maintenance [1,2].

Traditional PV inspection mainly depends on manual visual inspection, handheld thermal cameras, and local electrical testing. These methods are inefficient, costly, and highly dependent on expert experience. They are difficult to meet the demand for frequent inspection in large-scale PV plants. UAV-based inspection can collect module images over large areas with higher efficiency. However, if the collected data are only analyzed offline, abnormal locations may still be reported with delay. This also increases the cost of later field verification. Therefore, real-time defect detection for UAV-based PV inspection is important for improving the efficiency of PV plant maintenance [3,4,5].

In UAV-based PV inspection, different imaging modalities describe abnormalities in different ways. Infrared thermal images can reflect local overheating and internal electrical faults through temperature distribution. However, they usually have low spatial resolution, blurred boundaries, and limited texture details. As a result, they cannot describe the surface shape of abnormalities well. In contrast, visible images have higher spatial resolution and richer texture information. They can better show surface contamination, occlusion, and structural damage. However, they are less sensitive to internal electrical faults and early thermal abnormalities [6,7]. Therefore, the joint use of infrared and visible information can describe both thermal abnormalities and appearance features. This can improve the completeness and robustness of PV defect detection [8].

With the development of deep learning, PV abnormality recognition and localization have gradually shifted to data-driven detection frameworks [9]. Some studies have also introduced infrared–visible dual-modal information to improve detection performance [10,11]. However, most existing dual-modal PV detection methods are built and evaluated on image pairs with good spatial correspondence. In some datasets, the image pairs are cropped, resized, or registered after data collection, which reduces cross-modal spatial deviation before model training and testing. Under this setting, weak misalignment caused by sensor installation, imaging time differences, and UAV attitude changes is often not treated as a core problem for the detection model. In contrast, real-time UAV inspection needs to process continuous dual-modal images under limited latency and computing resources. It is difficult to apply complex fine registration before each inference. Therefore, weak misalignment directly affects the stability of dual-modal fusion and defect localization. This problem is more evident in PV scenes because PV modules have regular array structures and repeated textures, while some abnormal targets are small and have blurred boundaries. Once cross-modal features cannot be well matched, direct fusion may introduce feature ghosting, background interference, and localization bias, which limits detection performance [12,13].

To address these problems, this paper proposes FAF-YOLO, a frequency-aware fusion detection framework for real-time UAV-based photovoltaic inspection. Based on a dual-branch YOLO detector, the framework does not require strict pixel-level registration before inference. It can achieve stable dual-modal feature fusion and defect detection under weak misalignment.

The main contributions of this paper are as follows.

• We construct DM-PV, an infrared–visible dual-modal photovoltaic defect detection dataset. It is collected from real photovoltaic plants and covers two types of anomalies: internal electrical faults and external environmental interference.
• We design a Frequency-aware Selective Fusion (FSF) module. It models low-frequency structural information and high-frequency detail responses separately, and performs frequency-guided selective cross-modal fusion. This reduces edge ghosting and redundant background responses caused by weak misalignment.
• To address blurred defect boundaries and large scale differences in photovoltaic defects, we optimize the feature extraction module and detection head for this task. These designs improve local detail representation, multi-scale prediction ability, and detection stability in complex backgrounds while controlling computational cost.
• The proposed model shows good overall detection performance. Quantitative results and visual comparisons show that FAF-YOLO improves detection performance while keeping low parameters and computational cost. Embedded-platform tests further verify its feasibility for edge deployment.

The rest of this paper is organized as follows. Section 2 reviews related work. Section 3 describes the construction of the DM-PV dataset. Section 4 presents the overall framework and key modules of FAF-YOLO. Section 5 reports the experimental results and analysis, including ablation studies, comparative experiments, weak-misalignment robustness tests, visual analysis, and edge deployment validation. Finally, Section 6 concludes the paper and discusses future research directions.

2. Related Work

2.1. UAV-Based Photovoltaic Defect Detection

In recent years, defect detection for intelligent PV operation and maintenance has gradually moved from manual inspection and electrical testing to automated detection frameworks based on UAV images and deep learning [14]. Since PV abnormalities show different physical characteristics in different imaging modalities, existing studies mainly follow three technical routes: infrared thermal imaging, visible imaging, and infrared–visible multimodal imaging.

In single-modal PV detection, infrared thermal imaging and visible imaging play different roles. Thermal infrared images directly reflect the surface temperature distribution of PV modules and are widely used to detect electrical abnormalities such as hot spots, diode failures, and local overheating. Methods such as CEMP-YOLO, dynamic adaptive small-object detection networks, and ST-YOLO improve infrared thermal abnormality detection through feature extraction, small thermal target enhancement, and structural optimization [15,16,17]. Visible images provide richer texture, color, and structural information, and are suitable for detecting appearance abnormalities such as surface contamination, occlusion, and local reflection [18,19]. However, a single infrared modality lacks fine texture and boundary information, while a single visible modality cannot directly capture internal electrical faults and early thermal abnormalities. Therefore, single-modal methods are difficult to use for both thermal and appearance abnormalities.

With the development of multimodal vision, some studies have started to use infrared and visible information together for PV defect detection. Kuo et al. used UAV infrared and RGB imaging for defect detection, classification, and localization in large-scale PV plants. Their study showed the value of dual-modal imaging in PV inspection [8]. Guo et al. proposed RFE-YOLO and built a fusion framework for multimodal fault detection in PV modules [20]. These studies show that infrared–visible dual-modal information is effective for PV defect detection. However, their main focus is on complementary modal representation and improved fusion accuracy. Their experiments are usually based on image pairs after field-of-view cropping, size normalization, or with good spatial correspondence. In contrast, fusion stability under weak misalignment in real-time UAV inspection has not been fully modeled or systematically evaluated.

Existing PV defect detection datasets are still mainly single-modal. Datasets such as PVF-10 and Aerial Infrared Health Monitoring are mainly designed for UAV thermal infrared PV fault classification or single-modal health monitoring [21,22]. They are difficult to use for infrared–visible feature-level fusion studies. Existing dual-modal PV datasets are limited in number. Some of them are not publicly available, and some cover only a small range of defect categories. As a result, they cannot fully support unified detection of multiple heterogeneous abnormalities. Therefore, it is still necessary to build an infrared–visible dual-modal PV defect detection dataset that reflects real UAV-based real-time inspection and covers both internal electrical faults and external environmental interference.

2.2. Infrared–Visible Multi-Sensor Fusion for Object Detection

In infrared–visible object detection, the fusion strategy determines how thermal responses and structural texture information are used for object representation. Existing approaches can generally be grouped into early, middle, and late fusion according to the stage at which cross-modal integration is performed, as illustrated in Figure 1.

Early fusion, also called input-level fusion, directly combines infrared and visible images before feature extraction and feeds them into a single-branch detection network. Nataprawira et al. compared the effects of multimodal inputs on pedestrian detection and found that input-level fusion can improve the detection accuracy of small targets at night [23]. Later studies further analyzed early fusion strategies for visible and thermal infrared images, and tried to reduce low-level feature interference between modalities through shape priors or distillation mechanisms [24,25]. Middle fusion, also called feature-level fusion, usually uses a two-branch network to extract infrared and visible features separately, and then performs cross-modal interaction in the backbone, feature pyramid, or detection neck. In recent years, attention mechanisms, Transformers, and dynamic fusion strategies have been introduced to improve cross-modal feature representation. Guided Attentive Feature Fusion, Cross-Modality Fusion Transformer, and ICAFusion improve infrared–visible feature fusion from the perspectives of attention weighting, global feature interaction, and iterative cross-attention, respectively [26,27,28]. Zhao et al. further proposed a coarse-to-fine dynamic fusion strategy to improve the use of cross-modal features [29]. Late fusion, also called decision-level fusion, usually keeps the infrared and visible branches independent, and only fuses the final prediction results through score fusion, box fusion, or probability integration. Weighted Boxes Fusion and ProbEn fuse multi-model or multi-modal detection results from the perspectives of bounding box integration and probabilistic modeling, respectively [30,31].

In summary, early fusion has a simple structure but is easily affected by low-level modal differences. Late fusion is more flexible for engineering purposes but offers limited feature-level interaction. In contrast, mid-level fusion allows deep feature interaction while keeping modalities independent, making it more suitable for modeling the complementary relationship between infrared and visible information. Existing studies also show that infrared–visible fusion has shifted from simple modal merging to effective information selection, redundancy reduction, and task-specific feature enhancement.

2.3. Weakly Misaligned Infrared–Visible Object Detection

In UAV-based infrared–visible inspection, the two sensors usually differ in field of view, spatial resolution, and imaging time. Platform attitude changes and flight vibration can also introduce local translation, scale variation, or slight rotation. If features are directly fused at the same spatial location, misaligned cross-modal responses may cause feature ghosting, semantic conflict, and localization bias. This weak misalignment problem has been studied in infrared–visible tasks such as multispectral pedestrian detection, vehicle detection, and UAV object detection.

Traditional methods mainly use image-level registration or region-level alignment. Image-level registration estimates the geometric transformation between infrared and visible images before detection [32,33,34]. Region-level alignment reduces the alignment range from the whole image to candidate regions or object boxes [35]. These methods have clear geometric meanings. However, image-level registration usually needs extra preprocessing, and region-level alignment depends on the quality of candidate regions or object boxes. Recent studies have gradually moved toward feature-level adaptive alignment and fusion. OAFA models cross-modal spatial offsets and uses offset-guided deformable fusion for feature alignment [11]. CF-Deformable DETR further uses cross-modal deformable attention to adaptively sample key feature points, reducing the influence of non-aligned inputs on detection results [12]. YOLO-Adaptor introduces a lightweight adaptive structure into a one-stage detection framework to improve detection adaptability under non-aligned visible–infrared inputs [36]. C2Former uses a calibrated complementary Transformer to model the interaction between infrared and visible features [37]. In addition, reliability-aware methods adjust fusion weights through modality confidence or uncertainty estimation [38,39]. Frequency-domain methods reduce the effect of local spatial shifts on fusion by separating structural information and detail responses [40].

Overall, existing methods provide useful ideas for weakly misaligned infrared–visible detection, but they still have limitations in real-time UAV-based photovoltaic inspection. On the one hand, offset learning, deformable sampling, cross-modal attention, and reliability estimation increase model complexity and inference cost. On the other hand, many methods rely on a fixed dominant modality or an annotated reference, which makes it difficult to adapt to the modality-dependent nature of photovoltaic defects. Specifically, internal electrical anomalies mainly depend on infrared thermal responses, while surface appearance anomalies mainly depend on visible textures and structural information. In this case, using a fixed dominant modality may weaken useful defect evidence from the other modality. Therefore, UAV-based real-time photovoltaic inspection requires lightweight and reliable cross-modal information selection under weak misalignment.

2.4. Real-Time Object Detection in UAV Images

Object detection in UAV images usually works on continuous aerial image streams. Objects need to be quickly identified in large fields of view, dynamic viewpoints, and complex backgrounds. Unlike fixed-view images, UAV imaging brings target scale variation, local detail weakening, and background interference at the same time. Small targets can lose boundary and texture information during deep feature extraction. Changes in view angle and flight height increase the difficulty of multi-scale modeling. Ground textures, shadows, and occlusions may also produce responses similar to the targets.

To address these problems, existing studies have improved UAV image detection from localization loss, feature enhancement, scale modeling, and background suppression. Xu et al. proposed normalized Wasserstein distance to reduce the sensitivity of IoU to small position shifts in tiny object detection [41]. Boundary-aware feature fusion improves local representation of small targets by strengthening edge information [42]. FFCA-YOLO and SCLNet improve small object detection in remote sensing or UAV images through context feature enhancement and scale-robust learning, respectively [43,44]. In addition, HWANet uses Haar wavelet attention to enhance useful structural information and suppress redundant background responses [45]. These methods show that detail preservation, context modeling, scale awareness, and background suppression are important for improving detection performance in UAV images.

Besides detection accuracy, real-time performance is another key constraint in UAV image detection. Onboard computing platforms are usually limited by size, weight, power, and computing capacity. Detection models need to keep high inference speed under limited resources. RTUAV-YOLO, AccuLiteFastNet, and efficient multi-scale fusion frameworks balance detection accuracy and computational cost through lightweight network design, fast inference, and efficient feature aggregation, respectively [46,47,48]. These studies show that detectors for UAV platforms should consider model size, computational complexity, and practical deployment efficiency at the same time.

Overall, UAV image object detection has formed clear research directions around small object localization, multi-scale representation, background suppression, and lightweight real-time deployment. However, under limited onboard computing resources, it is still important to maintain stable object representation, detection accuracy, and inference efficiency at the same time.

3. Dataset

3.1. Data Acquisition and Preprocessing

Existing photovoltaic defect datasets are mainly based on a single modality. Public infrared–visible dual-modal datasets are still limited. In addition, some dual-modal datasets are manually selected or offline registered, so they cannot well reflect weak misalignment in real-time UAV inspection. To build a dataset closer to real inspection conditions, we used a DJI Matrice 3TD UAV platform to collect infrared–visible images from several real photovoltaic plants and built the Dual-Modal Photovoltaic (DM-PV) dataset. The platform is equipped with visible and thermal infrared imaging units. The visible and infrared images have original resolutions of 5184 × 3888 and 640 × 512, respectively. During data collection, the flight altitude was set between 18 m and 25 m. The flight routes mainly followed the direction of the photovoltaic module arrays, with a small number of oblique routes used to cover samples under different viewing geometries. The gimbal pitch angle was mainly set between $-60°$ and $-90°$, so that both near-nadir and oblique-view images could be collected. The data were collected under sunny, thin-cloud, and overcast conditions. The collection time covered morning, noon, and afternoon, introducing imaging changes caused by different illumination levels, background temperatures, and surface reflections. In total, 2375 infrared–visible image pairs were collected, corresponding to 4750 single-modal images. Figure 2 illustrates the UAV-based acquisition process and representative paired samples.

After the original dual-modal images were collected, a fixed-parameter batch preprocessing procedure was used for coarse field-of-view alignment and size normalization. This process was used to reduce the time cost of sample-wise manual processing. Since the visible images usually cover a larger area than the infrared images, the visible images were first center-cropped with fixed parameters, so that their main observation area was roughly consistent with that of the infrared images. Then, both modalities were resized to 1280 × 1024. This process did not include sample-wise manual correction or complex image registration. Therefore, the processed image pairs still keep weak misalignment caused by real acquisition conditions, such as random translation and rotation. Figure 3 shows the cross-modal position offsets that remain after preprocessing. Compared with detection pipelines that rely on offline precise registration, this setting is more consistent with real-time inspection, where complex registration for each image pair is difficult to perform.

3.2. Dataset Annotation and Split

Based on the collected infrared–visible image pairs, DM-PV was built with six anomaly categories: single hotspot, large-area hotspot, diode short circuit, bird droppings, weed occlusion, and sunlight reflection. The first three categories mainly appear as thermal anomalies or internal electrical faults. The last three categories mainly correspond to external environmental interference. Together, they cover common thermal and appearance anomalies in photovoltaic plants.

For data annotation, LabelImg was used to draw bounding boxes around defective regions. Given that different defect types have different levels of visibility in infrared and visible images, a class-dependent reference-modality annotation protocol was adopted. Single hot spots, large-area hot spots, and diode short circuits were annotated based on the thermal response regions in infrared images. Bird droppings, weed occlusion, and sunlight reflection were annotated based on surface texture, occlusion patterns, and reflection regions in visible images. This strategy avoids low-confidence annotations in the modality where the defect response is weak or the boundary is unclear.

All annotations were recorded in a shared coordinate system with a unified image size. They were then merged into a single set of ground-truth labels for each infrared–visible image pair and used for model training and evaluation. Figure 4 shows representative dual-modal samples of the six defect types and their unified annotations.

For the experiments in this paper, DM-PV was divided into training, validation, and test sets using each infrared–visible image pair as the basic unit. This ensured that the same dual-modal sample did not appear in different subsets. The final split includes 3136 training images, 794 validation images, and 820 test images, accounting for about 66.0%, 16.7%, and 17.3% of all images, respectively. To evaluate model generalization under different collection conditions, the validation and test sets were selected from data collected under different environmental conditions and flight routes. Therefore, they differ from the training set in weather, background texture, viewing angle, and imaging distribution.

4. Methods

4.1. Overall Architecture of FAF-YOLO

Considering the requirements of detection accuracy, inference speed, and deployment cost in UAV-based photovoltaic inspection, we select YOLO11n as the base detector and build an infrared–visible mid-level fusion baseline [49]. The baseline uses two independent backbones to extract infrared and visible features separately. Features at the same scale are concatenated at intermediate layers and then processed by a $1\times 1$ convolution for channel reduction and feature mapping. The fused features are then fed into the neck and detection head for prediction. This approach can directly use dual-modal information, but simple concatenation lacks an explicit mechanism to select structural features, detailed responses, and background redundancy. In addition, when the spatial correspondence between modalities is unstable, it can introduce mismatched features and ineffective interactions. Thus, a more robust dual-modal fusion strategy is needed.

Based on this, we propose FAF-YOLO (Frequency-Aware Fusion YOLO). Its overall structure is shown in Figure 5. The framework is designed to handle intra-modal feature representation, weak misalignment-aware fusion, and scale-differentiated prediction. First, the C3k2-DPRG module is designed and applied to the backbone and neck feature extraction. This enhances defect boundaries, local details, and neighborhood context, providing stable intra-modal features for subsequent fusion. On this basis, the Frequency-aware Selective Fusion (FSF) module is introduced to separately model low-frequency structural information and high-frequency detail responses in infrared and visible features. Joint gating and residual cross-modal injection reduce interference from misaligned edges and redundant background responses. The fused multi-scale features are then fed into the lightweight MDD Head. Its differentiated prediction strategy improves small-scale defect localization and enhances the distinction of medium- and large-scale defects from complex backgrounds. Overall, FAF-YOLO can more stably utilize complementary information from infrared and visible modalities under weak misalignment, while improving multi-scale photovoltaic defect detection.

4.2. C3k2-DPRG Module

In UAV-based photovoltaic defect detection, small defects such as single hot spots and bird-dropping occlusions usually have weak boundaries. Their fine-grained edges and local abnormal responses can be weakened during the downsampling process in the backbone. In addition, bird droppings and weed occlusions can be confused with module edges, array textures, and background regions. This further increases the difficulty of defect discrimination. To address these problems, this study designs a DPRG Block. Its structure is shown in Figure 6. The DPRG Block is then embedded into the C3k2 module, forming C3k2-DPRG. Here, CBS and DWCBS denote Conv-BN-SiLU and DWConv-BN-SiLU, respectively. They are referred to as the convolution block and the depthwise separable convolution block in the following text.

Given an input feature X, the DPRG Block first performs channel mapping to obtain $X_p$. Then, $X_p$ is split into two parts along the channel dimension. One part uses a $3\times 3$ convolution block to extract local textures and edge details. The other part uses a $5\times 5$ depthwise separable convolution block to enlarge the receptive field with low computational cost. This process can be expressed as follows:

$$F_s={\varphi }_{3\times 3}\left(X_{p1}\right),F_l={\varphi }_{\mathrm{dw}5\times 5}\left(X_{p2}\right)$$

(1)

where $X_{p1}$ and $X_{p2}$ denote the two feature groups obtained by splitting $X_p$ along the channel dimension. ${\varphi }_{1\times 1}(·)$, ${\varphi }_{3\times 3}(·)$, and ${\varphi }_{\mathrm{dw}5\times 5}(·)$ denote the $1\times 1$ convolution block, the $3\times 3$ convolution block, and the $5\times 5$ depthwise separable convolution block, respectively.

The two branch outputs are merged through channel-wise concatenation. Then, channel shuffle is used to reorganize cross-branch information and obtain the reorganized feature. To reduce the interference from array textures, module borders, and complex backgrounds on abnormal responses, the DPRG Block further adopts residual spatial gating. This operation uses a spatial gate to generate spatial modulation weights. The weights are then injected into the reorganized feature through a residual form. This helps improve spatial discriminability and reduces the risk of over-suppressing weak responses. The process is defined as follows:

$$G\left(U\right)=\sigma \left(\mathrm{BN}\left({\mathrm{DWConv}}_{5\times 5}\left(U\right)\right)\right)$$

(2)

$$\tilde{U}=U+U\odot G\left(U\right)$$

(3)

where U denotes the reorganized feature, ${\mathrm{DWConv}}_{5\times 5}(·)$ denotes the $5\times 5$ depthwise convolution, $\sigma (·)$ denotes the Sigmoid activation function, $G\left(U\right)$ denotes the spatial modulation weight, and ⊙ denotes element-wise multiplication.

Finally, $\tilde{U}$ is passed through channel transformation. When the input and output channels are the same and the shortcut is enabled, the DPRG Block uses a residual connection. Its output can be expressed as follows:

$$Y=\left\{\begin{array}{cc}X+{\varphi }_{1\times 1}\left(\tilde{U}\right),\hfill & C_{\mathrm{in}}=C_{\mathrm{out}}\mathrm{and}\mathrm{shortcut}\mathrm{is}\mathrm{enabled}\hfill \\ {\varphi }_{1\times 1}\left(\tilde{U}\right),\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(4)

The original C3k2 module adopts a cross-stage partial connection. One part of the feature is directly preserved. The other part is sent into a transformation branch composed of Bottleneck units. The two parts are then concatenated and fused through channel mapping. This structure has good information flow and computational efficiency. However, its transformation branch mainly relies on a single-path local convolution. It has limited ability to jointly model fine-grained abnormal responses and neighborhood context. Therefore, this study replaces the original transformation units with DPRG Blocks to construct C3k2-DPRG. Figure 7 compares the structures of the original C3k2 and the proposed C3k2-DPRG.

For C3k2-DPRG, the input feature is mapped and then split into $Y_1$ and $Y_2$. Here, $Y_1$ is the cross-stage preserved branch. It is used to maintain the original information flow and gradient propagation. $Y_2$ passes through n DPRG Blocks for progressive feature refinement:

$$Y_{i+2}=\mathcal{D}\left(Y_{i+1}\right),i=1,2,\dots ,n$$

(5)

$$Z={\varphi }_{1\times 1}\left(\mathrm{Concat}(Y_1,Y_2,\dots ,Y_{n+2})\right)$$

(6)

where $\mathcal{D}(·)$ denotes the DPRG Block, and n is set to 2 in this paper. C3k2-DPRG is used in both the backbone and the neck. In the backbone, it enhances the modality-specific representation of the infrared and visible branches, and provides more stable inputs for the following FSF module. In the neck, it strengthens multi-scale aggregated features and provides a more effective feature basis for classification and regression in the MDD Head.

4.3. Frequency-Aware Selective Fusion Module

Under weak misalignment, infrared and visible features still provide complementary semantic information. However, their spatial positions are not always strictly matched. If the two features are directly concatenated or added at the same spatial location, misaligned edges, repeated textures, and irrelevant thermal responses may be introduced as false complementary information. This can cause feature ghosting and background interference. Motivated by the above analysis, a Frequency-aware Selective Fusion (FSF) module is designed in this paper, as shown in Figure 8. Wavelet transform is used in the module to separate low-frequency structural components and high-frequency detail components. It then generates low-frequency context guidance and high-frequency spatial attention. These two cues are combined into a joint gate to selectively inject cross-modal complementary information.

Given the infrared feature $F_{\mathrm{ir}}$ and the visible feature $F_{\mathrm{vis}}$, FSF first applies two-dimensional discrete wavelet transform to each modality. The features are decomposed into low-frequency and high-frequency components:

$$L^{ir},H^{ir}=\mathrm{DWT}\left(F^{ir}\right),L^{vis},H^{vis}=\mathrm{DWT}\left(F^{vis}\right)$$

(7)

where $L(·)$ and $H(·)$ denote the low-frequency and high-frequency components, respectively. The low-frequency component mainly represents structural and contextual information. It is relatively stable under misalignment caused by local translation and slight rotation. Therefore, it is more suitable as a structural constraint for cross-modal fusion. The high-frequency component is obtained by concatenating multiple directional high-frequency subbands. It reflects detail changes such as boundaries, textures, and local variations. However, it is also more sensitive to misaligned edges and background textures. Therefore, FSF models low-frequency and high-frequency information in separate branches. The two frequency responses are used for structural guidance and spatial selection, respectively.

In the low-frequency branch, FSF first normalizes the low-frequency components of the infrared and visible features. This reduces the feature distribution difference between the two modalities. Then, a low-frequency mixed representation is built using a summation term and an absolute difference term. An average operation is used to keep the feature magnitude stable. The summation term preserves the common structural responses of the two modalities. The difference term supplements the structural differences between modalities. The mixed feature is then compressed by a $1\times 1$ convolution block. SS2D is introduced to model long-range structural relations in the regular PV module arrays [50]. The low-frequency context guidance map is generated as follows:

$${\widehat{L}}^{ir}=\mathrm{GN}\left(L^{ir}\right),{\widehat{L}}^{vis}=\mathrm{GN}\left(L^{vis}\right)$$

(8)

$$L^{mix}={\displaystyle \frac{{\widehat{L}}^{ir}+{\widehat{L}}^{vis}+\left|{\widehat{L}}^{ir}-{\widehat{L}}^{vis}\right|}{3}}$$

(9)

$$A_l=\sigma \left({\mathrm{Conv}}_{1\times 1}\left(\mathrm{S}\mathrm{S}2\mathrm{D}\left({\varphi }_{1\times 1}\left(L^{mix}\right)\right)\right)\right)$$

(10)

where $A_l$ denotes the low-frequency context guidance map, $\mathrm{GN}$ denotes Group Normalization, $SS2D$ denotes the two-dimensional selective scan module, and ${\mathrm{Conv}}_{1\times 1}(·)$ denotes the $1\times 1$ convolution for generating the attention map. The other symbols follow the definitions given above.

In the high-frequency branch, FSF does not directly add high-frequency features into the fusion output. Instead, it uses them to generate a spatial attention map. This map is used to constrain the positions where cross-modal information is injected. Specifically, the high-frequency components of the infrared and visible features are concatenated along the channel dimension. A $1\times 1$ convolution and a Sigmoid activation are then used to generate the high-frequency spatial attention map as follows:

$$A_h=\sigma \left({\mathrm{Conv}}_{1\times 1}\left(\mathrm{Concat}\left(H^{ir},H^{vis}\right)\right)\right)$$

(11)

where $A_h$ denotes the high-frequency spatial attention map.

After obtaining the low-frequency context guidance map and the high-frequency spatial attention map, FSF multiplies them through broadcasting to form a joint gating map. The map is then upsampled to the original resolution of the feature. Next, the gating map modulates the feature of the other modality through element-wise multiplication. The modulated feature is injected into the current modality branch in a residual form. Finally, the two modality features are concatenated as the fusion output:

$$A=\mathrm{Up}\left(A_l\odot A_h\right)$$

(12)

$${\tilde{F}}^{ir}=F^{ir}+F^{vis}\odot A,{\tilde{F}}^{vis}=F^{vis}+F^{ir}\odot A$$

(13)

$$F^{fuse}=\mathrm{Concat}\left({\tilde{F}}^{ir},{\tilde{F}}^{vis}\right)$$

(14)

where $\mathrm{Up}(·)$ denotes the upsampling operation, and ⊙ denotes element-wise multiplication. Compared with direct gate-based filtering or uniform weighting, this cross-enhancement strategy keeps the original response and introduces useful complementary information from the other modality.

Through this design, FSF takes the infrared and visible branch features enhanced by C3k2-DPRG as input. It performs selective cross-modal fusion under frequency guidance. The fused features are then fed into the MDD Head as multi-scale prediction features for classification and boundary localization.

4.4. Multi-Scale Differentiated Decoupled Head

After front-end feature extraction and frequency-selective fusion, the network obtains multi-scale fused features that contain thermal responses, texture structures, and local details. However, the P3, P4, and P5 layers differ in spatial resolution, semantic strength, and target-scale suitability. If the same prediction branch is used at all three scales, it is difficult to achieve both accurate localization for small defects and semantic discrimination for regional anomalies. To address this problem, this paper designs a Multi-scale Differentiated Decoupled Head (MDD Head). Its structure is shown in Figure 9.

The P3 layer has a high spatial resolution. Its regression branch directly affects the localization accuracy of small-scale anomalies. Under weak misalignment, the same cross-modal offset can cause a larger relative position error for small targets. This increases the difficulty of boundary regression. Therefore, a PixelShuffle path is introduced into the P3 regression branch [51]. It uses channel-to-space rearrangement to supplement high-resolution spatial representation. Then, depthwise separable convolution is used for boundary regression. This process is formulated as:

$$F_{box}^{P3}={\mathrm{Conv}}_{1\times 1}\left({\varphi }_{dw3\times 3}\left(PS\left({\varphi }_{1\times 1}\left(F^{P3}\right)\right)\right)\right)$$

(15)

where $F^{P3}$ denotes the input feature of the P3 layer, $PS(·)$ denotes the PixelShuffle operation, and ${\mathrm{Conv}}_{1\times 1}(·)$ denotes the prediction output convolution. The other symbols follow the definitions given above.

The classification branch of the P3 layer uses depthwise separable convolution for feature mapping. This branch extracts discriminative texture information for small-scale anomalies while reducing redundant computation. It is formulated as:

$$F_{cls}^{P3}={\mathrm{Conv}}_{1\times 1}\left({\varphi }_{dw3\times 3}\left({\varphi }_{1\times 1}\left(F^{P3}\right)\right)\right)$$

(16)

Different from the P3 layer, the P4 and P5 layers have larger receptive fields. They are more suitable for modeling regional structure information of medium- and large-scale anomalies. PV modules and some anomalies usually show clear row-wise and column-wise distributions. Local convolution alone is not enough to capture this directional context. Therefore, this paper builds a Strip Context module in the classification branches of P4 and P5. Its internal structure is shown at the bottom of Figure 9. This module aggregates context information along the horizontal and vertical directions. It improves the discrimination of abnormal shapes and PV module structural distributions. Given the input feature X, let $X_r$ denote the feature obtained by the $1\times 1$ convolution mapping. The Strip Context module can be expressed as follows:

$$C=Poo{l}_h\left(X_r\right)+Poo{l}_w\left(X_r\right)$$

(17)

$$M=\sigma \left({\mathrm{Conv}}_{1\times 1}\left({\varphi }_{1\times 1}\left(C\right)\right)\right)$$

(18)

$$Y=X\odot M$$

(19)

where $Poo{l}_h(·)$ and $Poo{l}_w(·)$ denote height-wise and width-wise strip pooling, respectively [52]. C denotes the direction-aware context aggregation feature, and M denotes the spatial weight. Therefore, the classification branch of P4/P5 can be expressed as follows:

$$F_{cls}^{P_i}={\mathrm{Conv}}_{1\times 1}\left({\varphi }_{dw3\times 3}\left(SC\left({\varphi }_{1\times 1}\left(F^{P_i}\right)\right)\right)\right),i\in \{4,5\}$$

(20)

where $\mathrm{SC}(·)$ denotes the Strip Context module, and $F^{P_i}$ denotes the input feature at the corresponding scale.

For the regression branches of P4 and P5, boundary prediction depends more on local position and edge responses. To avoid introducing irrelevant regional responses through context aggregation, Strip Context is not used in these branches. Instead, a convolutional prediction path is used for boundary regression:

$$F_{box}^{P_i}={\mathrm{Conv}}_{1\times 1}\left({\varphi }_{dw3\times 3}\left({\varphi }_{1\times 1}\left(F^{P_i}\right)\right)\right),i\in \{4,5\}$$

(21)

Unlike homogeneous detection heads, the MDD Head builds differentiated branches according to the prediction needs of features at different scales. The high-resolution P3 regression branch focuses on the localization of small-scale anomalies. The semantically richer P4/P5 classification branches focus on regional anomaly discrimination. The remaining branches stay lightweight. This design better adapts to the fused multi-scale features and balances detection adaptability with computational cost.

5. Experiments and Results

5.1. Experimental Settings

All experiments in this study were conducted on Ubuntu 22.04. The implementation used Python 3.10 and PyTorch 2.1.2. The hardware platform consisted of one NVIDIA RTX 4090 GPU with 24 GB memory and an Intel Xeon Platinum 8352V CPU with 16 vCPUs at 2.10 GHz. CUDA 11.8 was used for GPU acceleration. Model training, validation, and testing were all performed in this environment. The main training settings are listed in

Table 1. Training parameter settings.

Parameter	Value
Input image size	$640\times 640$
Epochs	500
Batch size	16
Optimizer	AdamW
Initial learning rate	0.002
Warmup epochs	5
Learning rate scheduler	Cosine annealing
Mixed precision training	Disabled

.

This study uses the online data augmentation mechanism of the YOLO framework. To improve scene adaptability, random geometric transformations are applied during training, including horizontal flipping, vertical flipping, rotation, translation, and scaling. For paired infrared and visible images, these operations are performed synchronously on both modalities to preserve cross-modal spatial correspondence. The augmentation settings are listed in

Table 2. Online data augmentation settings.

Parameter	Value
Horizontal flip	0.5
Vertical flip	0.5
Rotation (degrees)	10.0
Translation	0.1
Scaling	0.5

, and examples of augmented training batches are shown in Figure 10.

5.2. Evaluation Metrics

To evaluate the detection performance and deployment cost of the model in photovoltaic defect detection, this study uses Precision (P), Recall (R), ${\mathrm{mAP}}_{50}$, ${\mathrm{mAP}}_{50:95}$, Params, and GFLOPs as evaluation metrics. Precision and Recall measure detection accuracy and recall ability, respectively. ${\mathrm{mAP}}_{50}$ and ${\mathrm{mAP}}_{50:95}$ measure detection performance under different localization requirements. Params and GFLOPs measure model complexity and computational cost. The definitions are as follows:

$$\mathrm{Precision}={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}}$$

(22)

$$\mathrm{Recall}={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}}$$

(23)

$$\mathrm{AP}={\int }_0^{1}P\left(r\right)\mathrm{d}r$$

(24)

$${\mathrm{mAP}}_{50}={\displaystyle \frac{1}{N}}{\displaystyle \sum _{c=1}^N}{\mathrm{AP}}_c^{\mathrm{IoU}=0.50}$$

(25)

where $TP$, $FP$, and $FN$ denote true positives, false positives, and false negatives, respectively. $AP$ is calculated from the Precision–Recall curve. $A{P}_c^{\mathrm{IoU}=0.50}$ denotes the AP of class c at the IoU threshold of 0.50, and N denotes the number of classes. ${\mathrm{mAP}}_{50:95}$ is averaged over IoU thresholds from 0.50 to 0.95 with a step size of 0.05.

5.3. Experiments on Modality Configuration and Fusion Strategy

5.3.1. Infrared-Only Configuration

To verify the effectiveness of infrared–visible dual-modal modeling and to select the base detector for later experiments, this study compares infrared-only and dual-modal configurations based on several lightweight YOLO detectors [49,53,54,55]. The results are shown in

Table 3. Results of infrared-only and dual-modal configurations.

Methods	Modality	P (%)	R (%)	${\mathbf{mAP}}_{50}$ (%)	${\mathbf{mAP}}_{50:95}$ (%)
YOLOv8n	IR	71.0	65.5	66.4	46.6
YOLOv8n	RGB + IR	88.8	84.4	89.2	57.6
YOLOv10n	IR	62.0	65.5	64.3	40.6
YOLOv10n	RGB + IR	87.0	81.1	87.9	57.6
YOLO11n	IR	69.2	66.9	67.2	44.7
YOLO11n	RGB + IR	87.6	85.0	89.2	59.3
YOLO12n	IR	68.0	66.5	66.5	44.6
YOLO12n	RGB + IR	86.9	85.0	89.3	57.5

. Infrared is used as the single-modal input because appearance anomalies may still produce thermal responses in infrared images, while internal electrical anomalies usually lack stable visible features. The results show that the infrared-only configuration has some detection ability, but its overall performance is clearly lower than that of the dual-modal configuration. This indicates that visible structural and texture information can complement infrared thermal responses. At the same time, YOLO11n achieves the highest ${\mathrm{mAP}}_{50:95}$ under the dual-modal configuration. Therefore, YOLO11n is selected as the base detector for later improvements.

5.3.2. Effect of Fusion Stage and Operation

To study the effect of fusion stage and operation on dual-modal detection, we compare early fusion and mid-level fusion schemes. Each scheme is implemented with either Addition or Concatenation. All experiments in

Table 4. Comparison of different fusion strategies.

Fusion Stage	Fusion Operation	P (%)	R (%)	${\mathbf{mAP}}_{50}$ (%)	${\mathbf{mAP}}_{50:95}$ (%)	Params (%)	FLOPs (%)
Early	Addition	83.9	75.0	79.2	48.1	2.58	6.3
Early	Concatenation	84.2	77.3	83.7	53.1	2.58	6.4
Middle	Addition	87.2	84.0	88.4	57.3	3.46	7.5
Middle	Concatenation	87.6	85.0	89.2	59.3	4.14	10.0

use YOLO11n as the base detector, changing only the fusion settings to ensure a fair comparison.

Table 4 shows that mid-level fusion generally performs better than early fusion. Concatenation also outperforms Addition in both fusion stages. Among the tested configurations, mid-level concatenation achieves the best results, with Precision, Recall, ${\mathrm{mAP}}_{50}$, and ${\mathrm{mAP}}_{50:95}$ reaching 87.6%, 85.0%, 89.2%, and 59.3%, respectively. This indicates that concatenating features after independent branch extraction better preserves dual-modal responses and leverages complementary information. Therefore, mid-level concatenation with YOLO11n is selected as the baseline fusion setting for subsequent improvements.

5.4. Ablation Experiments

To verify the effectiveness of the proposed core components and the rationality of their structural designs, ablation experiments are conducted under the same training and evaluation settings. The experiments are analyzed from three aspects: the overall component combination, the internal design of C3k2-DPRG, and the key components of FSF. All experiments use the same dataset split and training strategy to ensure a fair comparison.

5.4.1. Overall Module Ablation

To verify the effectiveness of the three core components, we conduct ablation experiments on C3k2-DPRG, FSF, and MDD Head. Each component is introduced alone and in combination with the others. The results are shown in

Table 5. Ablation study on different module combinations in FAF-YOLO11.

Baseline	C3k2- DPRG	FSF	MDD	P (%)	R (%)	${\mathbf{mAP}}_{50}$ (%)	${\mathbf{mAP}}_{50:95}$ (%)	Params (M)	FLOPs (G)
YOLO11n				87.6	85.0	89.2	59.3	4.14	10.0
	✓			90.3	84.5	89.5	59.6	4.07	10.7
		✓		88.2	85.3	89.6	59.8	4.19	9.7
			✓	88.7	83.4	89.9	59.7	4.23	9.8
	✓	✓		89.1	86.4	90.8	60.8	4.12	10.5
	✓		✓	88.7	85.7	89.8	59.8	3.84	9.3
	✓	✓	✓	92.5	86.7	91.7	61.4	3.90	9.0

. When C3k2-DPRG is used alone, Precision increases by 2.7%, while Recall decreases by 0.5%. This indicates that C3k2-DPRG improves the discrimination of defect candidate regions. When FSF is used alone, all four performance metrics are improved. At the same time, FLOPs are reduced by 0.3 G, and the number of parameters increases by only 0.05 M. This shows that frequency-guided fusion can improve cross-modal feature interaction with a low extra cost. MDD Head improves Precision and ${\mathrm{mAP}}_{50}$ by 1.1% and 0.7%, respectively, but Recall decreases to 83.4%. This suggests that scale-differentiated prediction helps improve localization quality, but its effect on recall is limited.

The combination results further show that these components are complementary. When C3k2-DPRG and FSF are used together, Recall, ${\mathrm{mAP}}_{50}$, and ${\mathrm{mAP}}_{50:95}$ increase by 1.4%, 1.6%, and 1.5% over the baseline, respectively. This suggests that stronger feature representation can provide more effective inputs for frequency-guided fusion. The best performance is achieved when all three components are introduced. Compared with the baseline, Precision, Recall, ${\mathrm{mAP}}_{50}$, and ${\mathrm{mAP}}_{50:95}$ increase by 4.9%, 1.7%, 2.5%, and 2.1%, respectively. Meanwhile, the number of parameters and FLOPs decrease by 0.24 M and 1.0 G, respectively. These results show that the proposed method improves detection performance while reducing model complexity.

To analyze the effect of C3k2-DPRG and FSF on dual-modal feature representation, typical samples from different defect categories are selected for visualization, as shown in Figure 11. The last two rows show the fused feature response maps of the Concat baseline and the C3k2-DPRG+FSF design. It can be seen that the responses of the Concat baseline tend to spread along grid lines, panel boundaries, and background textures. The defect-related responses are not sufficiently concentrated. After introducing C3k2-DPRG and FSF, the fused features show clearer and more focused responses in defect regions. Specifically, the dual-branch convolution and residual gating in C3k2-DPRG strengthen local boundaries and neighboring context, which helps small-scale abnormalities form more distinct structural responses. FSF suppresses interference from background textures and misaligned edges through frequency-aware feature selection. This makes the responses of regional abnormalities more complete. These results are consistent with the performance improvement brought by the combined modules in Table 5.

5.4.2. Ablation Study on the C3k2-DPRG Module

To analyze the effect of the dual-branch convolution configuration in the DPRG Block, ablation experiments are conducted on different kernel sizes and convolution types while keeping FSF, the MDD Head, and other training settings unchanged. The results are shown in

Table 6. Ablation study of different kernel sizes and convolution types in the C3k2-DPRG module.

C3k2-DPRG		P (%)	R (%)	${\mathbf{mAP}}_{50}$ (%)	${\mathbf{mAP}}_{50:95}$ (%)	Params (M)	FLOPs (G)
Branch1	Branch2
$3\times 3$	$3\times 3$ DW	89.6	81.4	90.2	59.8	3.88	9.0
$3\times 3$	$7\times 7$ DW	89.3	85.1	89.3	59.2	3.89	9.0
$3\times 3$ DW	$5\times 5$ DW	85.4	82.8	87.5	57.4	3.72	8.5
$3\times 3$	$5\times 5$	87.8	82.7	88.5	59.0	4.35	10.4
$3\times 3$	$5\times 5$ DW	92.5	86.7	91.7	61.4	3.90	9.0

Note: Layers without DW denote standard convolutions, while layers with DW denote depthwise separable convolutions.

. For kernel size, Branch 2 with $3\times 3$ depthwise separable convolution gives 81.4% Recall, showing a small receptive field cannot capture enough context. A $7\times 7$ kernel increases Recall but lowers ${\mathrm{mAP}}_{50}$ due to background interference. In comparison, a $5\times 5$ depthwise separable convolution balances context and accuracy. For convolution type, using depthwise separable convolutions in both branches reduces parameters and computation, but ${\mathrm{mAP}}_{50}$ drops to 87.5%. Using a $5\times 5$ standard convolution in Branch 2 raises FLOPs to 10.4 G with no performance gain. Overall, $3\times 3$ standard convolution combined with $5\times 5$ depthwise separable convolution achieves the best performance, with ${\mathrm{mAP}}_{50}$ and ${\mathrm{mAP}}_{50:95}$ of 91.7% and 61.4%, respectively.

To examine the effect of DPRG depth on detection performance and computational complexity, the actual number of stacked DPRG blocks, n, was set to 1, 2, 3, and 4, while all other network structures and training settings were kept unchanged. As shown in

Table 7. Sensitivity analysis of DPRG depth.

DPRG Depth n	P (%)	R (%)	${\mathbf{mAP}}_{50}$ (%)	${\mathbf{mAP}}_{50:95}$ (%)	Params (M)	FLOPs (G)
1	87.2	86.0	89.3	58.4	3.41	7.3
2	92.5	86.7	91.7	61.4	3.90	9.0
3	89.9	86.4	90.7	60.8	4.40	10.7
4	88.3	86.4	90.1	59.6	4.91	12.4

Note: Bold values indicate the DPRG depth configuration selected for the final model.

, when n increases from 1 to 2, $\mathrm{mAP}50$ and $\mathrm{mAP}50:95$ increase by 2.4% and 3.0%, respectively. Further increasing the DPRG depth leads to higher parameter counts and computational costs, while detection performance decreases. Considering both accuracy and complexity, $n=2$ was selected in the final model.

5.4.3. Ablation Study on the FSF Module

To verify the contributions of the main components of FSF, ablation experiments were conducted on frequency decomposition (FD), the high-frequency gate (HF Gate), the low-frequency gate (LF Gate), and cross-modal residual injection (CMI). As shown in

Table 8. Ablation study on key components in FSF module.

Methods	FD	HF Gate	LF Gate	CMI	P (%)	R (%)	${\mathbf{mAP}}_{50}$ (%)	${\mathbf{mAP}}_{50:95}$ (%)
SDU-residual					88.0	83.7	88.1	57.6
FD-residual	✓				87.1	84.2	89.6	57.8
HF-only	✓	✓		✓	86.7	86.6	89.0	59.1
LF-only	✓		✓	✓	89.0	84.3	89.6	59.9
Without CMI	✓	✓	✓		89.1	83.5	88.1	59.4
FSF	✓	✓	✓	✓	88.2	85.3	89.6	59.8

. FD-residual improves ${\mathrm{mAP}}_{50}$ by 1.5 percentage points, indicating that frequency decomposition helps enhance defect-related responses. The HF-only and LF-only variants show complementary advantages. When CMI is removed, Recall decreases to 83.5%, indicating that cross-modal residual injection helps use complementary information from the two modalities. The complete FSF achieves a more balanced overall performance.

Figure 12 shows the feature responses of SDU-residual, FD-residual, and FSF. In Figure 12a, SDU-residual produces clear background activation in the green box, while FD-residual suppresses part of it through wavelet decomposition. FSF further reduces redundant background responses through selective gating. In Figure 12b, SDU-residual gives a weak response in the yellow box, and FD-residual activates the abnormal region but with scattered responses. FSF produces a stronger and more concentrated response by introducing cross-modal residual information. These visual results are consistent with Table 8, confirming that frequency decomposition and selective cross-modal interaction improve fused feature quality under weak misalignment.

To further verify the internal design of FSF, three controlled variants were constructed on the baseline network, while all other settings were kept unchanged. LF-Sum retains only the summation term in low-frequency interaction, LF-$3\times 3$ Conv replaces SS2D with a $3\times 3$ convolution, and HF-Avg replaces high-frequency feature concatenation with element-wise averaging. As shown in

Table 9. Ablation study of internal design choices in the FSF module.

Variant	P (%)	R (%)	${\mathbf{mAP}}_{50}$ (%)	${\mathbf{mAP}}_{50:95}$ (%)
LF-Sum	84.5	84.0	87.6	57.0
LF-$3\times 3$ Conv	82.8	83.2	86.5	57.0
HF-Avg	85.4	82.9	87.8	57.9
Full FSF	88.2	85.3	89.6	59.8

, all three variants show lower performance, indicating that the low-frequency absolute-difference term, long-range context modeling, and the preservation of modality-specific high-frequency details all improve feature fusion. The complete FSF achieves the best overall performance.

5.4.4. Ablation Study on the MDD Module

To verify the scale-aware and task-specific design of the MDD Head, controlled ablation experiments were conducted with different branch configurations, while all other settings were kept unchanged. The detailed configurations are listed in

Table 10. Ablation study of branch-specific configurations in the MDD Head.

Variant	P3 Box	P4/P5 Box	P4/P5 Cls	P (%)	R (%)	${\mathbf{mAP}}_{50}$ (%)	${\mathbf{mAP}}_{50:95}$ (%)
Plain Head	N	N	N	89.1	86.4	90.8	60.8
PS variant	PS	N	N	88.6	86.9	91.0	61.1
Box-SC variant	PS	SC	N	86.6	82.7	85.5	57.8
Dual-SC variant	PS	SC	SC	88.0	84.3	88.9	59.6
Proposed MDD	PS	N	SC	92.5	86.7	91.7	61.4

Note: Box and Cls denote the regression and classification branches, respectively. N denotes the standard lightweight Conv–DWConv path, PS denotes the PixelShuffle-based refinement path used in the P3 Box branch, and SC denotes the insertion of Strip Context into the corresponding P4/P5 branch.

. The PS variant increases Recall, ${\mathrm{mAP}}_{50}$, and ${\mathrm{mAP}}_{50:95}$ to 86.9%, 91.0%, and 61.1%, respectively, showing that PixelShuffle improves the localization and detection of small-scale defects. In contrast, adding Strip Context to the P4/P5 regression branches causes a clear performance drop, indicating that it is not suitable for bounding-box regression. Compared with the Plain Head, the complete MDD improves Precision, Recall, ${\mathrm{mAP}}_{50}$, and ${\mathrm{mAP}}_{50:95}$ by 3.4%, 0.3%, 0.9%, and 0.6%, respectively. These results support the use of PixelShuffle for P3 regression and Strip Context for P4/P5 classification.

5.5. Comparison with Different Models

To evaluate the overall performance of the proposed method, we compare it with TAF-YOLO [56], mid-concatenation fusion models based on different YOLO detectors, and representative dual-modal methods, including ICAFusion [28], DEYOLO [57], C2DFF-Net [58], CF-Deformable DETR [12] and RFE-YOLO [20]. The results are reported in

Table 11. Quantitative comparison with representative methods on the DM-PV dataset.

Methods	Fusion Type	P	R	${\mathbf{mAP}}_{50}$	${\mathbf{mAP}}_{50:95}$	Params	FLOPs	Weight Size
		(%)	(%)	(%)	(%)	(M)	(G)	(MB)
TAF-YOLO	Early	83.8	80.4	83.4	47.8	2.28	14.7	4.73
YOLOv8n	Mid-Concat	88.8	84.4	89.2	57.6	4.45	11.6	8.83
YOLOv10n	Mid-Concat	87.0	81.1	87.9	57.6	3.63	9.9	8.26
YOLO11n	Mid-Concat	87.6	85.0	89.2	59.3	4.14	10.0	8.36
YOLO12n	Mid-Concat	86.9	85.0	89.3	57.5	4.14	10.2	8.51
DEYOLO	Mid	87.1	78.0	84.9	57.6	6.00	16.8	11.84
ICAFusion (YOLOv5n-based)	Mid	90.3	82.4	88.3	57.4	6.66	249.2	13.16
ICAFusion (YOLO11n-based)	Mid	88.1	85.3	88.8	57.0	7.52	343.8	14.88
C2DFF-Net (YOLOv8n-based)	Mid	82.4	77.6	80.5	46.2	6.58	14.6	13.00
C2DFF-Net (YOLO11n-based)	Mid	82.3	74.7	78.4	42.7	6.67	16.4	13.17
CF-Deformable DETR	Mid	93.5	85.5	92.7	62.8	80.55	140.1	162.1
RFE-YOLO (YOLO11n-based)	Mid	86.6	84.0	87.9	56.6	5.14	12.2	10.67
Ours	Mid	92.5	86.7	91.7	61.4	3.90	9.0	7.94

. RFE-YOLO is designed for photovoltaic module fault detection, while the other methods mainly target general infrared–visible detection, UAV-based detection, or remote sensing small-object detection. For a more consistent comparison, several methods are adapted to the YOLO11n framework.

As shown in Table 11, the proposed method achieves the highest Recall and competitive Precision, ${\mathrm{mAP}}_{50}$, and ${\mathrm{mAP}}_{50:95}$, reaching 92.5%, 86.7%, 91.7%, and 61.4%, respectively. Compared with the YOLO11n mid-concatenation baseline, these metrics improve by 4.9, 1.7, 2.5, and 2.1 percentage points. Meanwhile, the parameters, FLOPs, and weight size decrease from 4.14 M, 10.0 GFLOPs, and 8.36 MB to 3.90 M, 9.0 GFLOPs, and 7.94 MB, respectively. This shows that the proposed method improves detection performance while reducing model complexity.

Among the compared dual-modal methods, TAF-YOLO obtains relatively lower accuracy under weak misalignment, while ICAFusion introduces substantially higher computational cost. C2DFF-Net and DEYOLO also show limited performance on the DM-PV dataset, suggesting that general dual-modal structures may not fully adapt to complex photovoltaic defect scenes. CF-Deformable DETR achieves slightly higher Precision, ${\mathrm{mAP}}_{50}$, and ${\mathrm{mAP}}_{50:95}$, whereas FAF-YOLO achieves higher Recall. However, CF-Deformable DETR requires substantially more parameters, computation, and model storage, making it less suitable for resource-constrained UAV deployment. RFE-YOLO obtains competitive results as a photovoltaic-oriented method, but FAF-YOLO still achieves better accuracy and efficiency. Overall, FAF-YOLO provides a more favorable accuracy–efficiency trade-off for UAV-based infrared–visible photovoltaic defect detection.

5.6. Robustness Test Under Weak Misalignment

To evaluate robustness under weak cross-modal misalignment, we perturb one image in each paired dual-modal sample during testing and compare the proposed method with the Mid-Fusion-YOLO11n baseline. Since UAV-based infrared–visible inspection often involves local translation, slight rotation, or both, we design three test groups: translation, rotation, and compound perturbations. Recall and ${\mathrm{mAP}}_{50}$ are used to measure defect detection ability and overall detection performance under misalignment.

5.6.1. Robustness Analysis Under Single Perturbations

Table 12. Quantitative comparison of detection performance under different translation shifts.

Methods	Criterion	Translation (px)
		0	20	25	30	35	40	45	50	55
Baseline	R (%)	85.0	81.0	77.8	77.0	71.4	71.5	65.1	67.1	63.1
	${\mathrm{mAP}}_{50}$ (%)	89.2	85.1	83.3	81.1	77.2	73.9	71.0	72.5	69.5
Ours	R (%)	86.7	83.7	81.8	78.0	77.8	76.7	75.0	70.0	67.4
	${\mathrm{mAP}}_{50}$ (%)	91.7	88.0	86.1	84.4	81.6	78.5	77.1	75.1	74.9

and Figure 13 show the results under translation perturbation. As the translation distance increases, Recall and ${\mathrm{mAP}}_{50}$ decrease for both models. This shows that the performance of dual-modal fusion is affected when the spatial correspondence between the two modalities is weakened. Compared with the baseline, the proposed method maintains higher Recall and ${\mathrm{mAP}}_{50}$ at all translation levels. Its performance curves also decrease more slowly, which indicates better adaptability to local position shifts.

Table 13. Quantitative comparison of detection performance under different rotation angles.

Methods	Criterion	Rotation Angle (°)
		0	3.0	4.0	5.0	6.0	7.0	8.0	9.0	10.0
Baseline	R (%)	85.0	82.8	76.2	71.3	66.8	62.3	58.0	55.2	54.3
	${\mathrm{mAP}}_{50}$ (%)	89.2	88.1	82.9	75.8	70.6	66.0	60.1	56.6	55.6
Ours	R (%)	86.7	84.7	81.4	79.6	74.2	67.4	59.6	58.8	61.4
	${\mathrm{mAP}}_{50}$ (%)	91.7	90.2	86.8	84.2	79.9	72.8	68.5	67.2	64.9

Note: To maintain layout clarity, the performance metrics from $3.0°$ to $10.0°$ are reported at $1°$ intervals. The overall degradation trend remains consistent with the dense sampling results.

and Figure 14 show the results under rotation perturbation. As the rotation angle increases, both models show a clear performance drop. This suggests that rotation misalignment also weakens cross-modal feature correspondence and affects detection stability. The proposed method maintains higher Recall and ${\mathrm{mAP}}_{50}$ in most cases, especially at larger rotation angles. Its curves also decline more slowly than those of the baseline. These results show that the proposed fusion strategy can reduce the effect of rotation misalignment and improve robustness under single-type weak misalignment.

5.6.2. Robustness Analysis Under Compound Perturbation

To simulate more complex misalignment in UAV inspection, translation and rotation are applied simultaneously. As shown in

Table 14. Quantitative comparison of detection performance under combined translation and rotation perturbations.

Methods	Criterion	Translation (px) & Rotation Angle (°)
		0 & 0	10 & 1.0	15 & 1.5	20 & 2.0	25 & 2.5	30 & 3.0	35 & 3.5	40 & 4.0
Baseline	R (%)	85.0	83.3	82.4	75.8	77.1	69.3	69.2	67.2
	${\mathrm{mAP}}_{50}$ (%)	89.2	88.9	86.3	82.5	82.8	76.0	72.6	72.1
Ours	R (%)	86.7	85.0	84.2	82.0	80.8	76.1	70.5	69.9
	${\mathrm{mAP}}_{50}$ (%)	91.7	90.7	88.8	88.3	86.7	80.0	76.8	77.3

and Figure 15, both models degrade as the compound perturbation increases, suggesting weakened infrared–visible feature correspondence. The proposed method consistently outperforms the baseline under all compound perturbation levels. At the highest level, it achieves 69.9% Recall and 77.3% ${\mathrm{mAP}}_{50}$, compared with 67.2% and 72.1% for the baseline. The slower decline of its curves further indicates better stability under compound misalignment.

We further visualize the fused feature maps under the highest compound perturbation level. As shown in Figure 16, the concatenation-based fusion strategy in the baseline produces clear edge ghosting around the PV panel boundaries. This suggests that direct concatenation can introduce misaligned edge information into the fused feature map. The misalignment also increases background responses and weakens the discriminative response in defect regions. In contrast, the proposed method clearly reduces edge ghosting and preserves strong responses at defect locations. This shows that the proposed fusion module has more stable feature selection ability under severe cross-modal misalignment.

5.7. Visual Comparison Analysis

To further analyze the differences between the proposed method and the baseline model, we present heatmap visualizations and detection result visualizations, as shown in Figure 17, Figure 18 and Figure 19.

As shown in Figure 17, the proposed method produces more focused heatmap responses on paired infrared and visible samples. The responses in defect regions are stronger, while responses in non-defect regions are weaker. Specifically, in the visible modality, the baseline model’s responses often spread to module textures, local reflections, and normal background areas. In contrast, the proposed method highlights defect-related regions more accurately and reduces redundant activations. In the infrared modality, the proposed method maintains clear responses in thermal anomaly regions without introducing noticeable noise in non-defect areas. These results indicate that the proposed fusion strategy increases the concentration of responses in defect-related features and reduces irrelevant activations in background regions.

Figure 18 shows the detection results on visible images. The baseline model has more false detections and missed detections in the visible modality. Some small defects or defects with unclear boundaries are not detected reliably. In addition, visually similar regions, such as bird droppings, weed occlusion, and local background textures, can cause class confusion or false detections. Compared with the baseline, the proposed method detects more small defects and gives more stable class predictions under complex backgrounds and similar appearance interference. This shows that FAF-YOLO can better use the texture and structural information provided by visible images.

Figure 19 shows the detection results on infrared images. Infrared images provide thermal response information, but some defects are small and have blurred boundaries. This can lead to missed detections or inaccurate localization. The baseline model is less stable in regions with weak thermal responses or complex background heat distributions. In contrast, the proposed method localizes thermal anomaly regions more accurately and reduces false responses in non-defect thermal areas. This indicates that FAF-YOLO can use cross-modal complementary information while maintaining the discriminative ability of infrared thermal cues.

5.8. Edge Deployment and Real-Time Performance Analysis

To verify the edge deployment capability of FAF-YOLO for real-time UAV-based photovoltaic inspection, we first test the model under FP32 and FP16 precision in the PyTorch framework. As shown in

Table 15. Inference performance under PyTorch and edge-side TensorRT deployment.

Inference Format	Precision Format	FPS	${\mathbf{mAP}}_{50}$ (%)
PyTorch weights	FP32	40	91.73
PyTorch weights	FP16	52	91.59
TensorRT engine	FP32	25	—
TensorRT engine	FP16	33	—

, converting the model from FP32 to FP16 increases the FPS from 40 to 52. The mAP@50 only decreases from 91.73% to 91.59%. This shows that half-precision inference can improve inference speed without causing a clear drop in detection performance.

We further deploy the model on a real edge device to test its real-time inference ability. The best PyTorch weights are exported to ONNX format. Then, FP32 and FP16 TensorRT inference engines are built on the NVIDIA Jetson Orin Nano platform. Since the effect of FP16 on detection accuracy has been tested in PyTorch, this edge-side test focuses mainly on inference speed. The results show that the TensorRT-FP16 engine reaches 33 FPS, while the TensorRT-FP32 engine reaches 25 FPS. This gives an improvement of about 32%. These results indicate that FP16 TensorRT deployment can further improve edge-side inference efficiency and support real-time defect detection in UAV-based photovoltaic inspection.

6. Conclusions

This paper studies the weak misalignment problem between infrared and visible images in UAV-based real-time photovoltaic inspection. A real-scene dual-modal photovoltaic defect dataset, named DM-PV, is constructed. Based on this dataset, a dual-modal defect detection method, named FAF-YOLO, is proposed. The method is designed for edge-side real-time inference. It does not rely on complex frame-by-frame image registration. Instead, it improves the fusion of infrared thermal responses and visible structural textures through feature enhancement, frequency-selective fusion, and scale-specific prediction.

The experimental results show that FAF-YOLO improves detection performance while keeping low model complexity. Compared with the mid-level concatenation fusion baseline, the proposed method improves Precision, Recall, ${\mathrm{mAP}}_{50}$, and ${\mathrm{mAP}}_{50:95}$. It also reduces the number of parameters and computational cost. The ablation experiments verify the contribution of each key design to the overall performance. The visualization results further show that the proposed method produces more concentrated responses in defect regions. It also reduces interference from normal panel textures and misaligned edges.

In the robustness and deployment tests, FAF-YOLO shows more stable detection performance under translation, rotation, and compound perturbations. This indicates that the method is more adaptable to weak misalignment in UAV inspection. The edge deployment results show that the TensorRT FP16 engine achieves real-time inference on the Jetson Orin Nano, further indicating the practical feasibility of the proposed method for edge-side UAV-based photovoltaic inspection. In future work, temporal information from UAV inspection videos will be introduced to improve detection consistency under complex flight conditions. Model compression, quantization, and inference acceleration will also be further studied.