Skip to Content
You are currently on the new version of our website. Access the old version .
ProcessesProcesses
  • Article
  • Open Access

13 February 2026

19 Pages

Domain-Adaptive Multimodal Large Language Models for Photovoltaic Fault Diagnosis via Dynamic LoRA Routing

,
,
,
,
and
1
State Grid Wenzhou Electric Power Supply Company, Wenzhou 317101, China
2
State Grid Ruian Electric Power Supply Company, Wenzhou 317101, China
3
School of Computer Science, Wuhan University, Wuhan 430072, China
4
School of Electrical and Automation, Wuhan University, Wuhan 430072, China

Abstract

The reliability of photovoltaic (PV) equipment is vital for ensuring the safe and stable operation of power systems. While multimodal large language models (MLLMs) open up promising avenues for intelligent fault diagnosis, they often falter when confronted with the heterogeneity of PV data—where visual observations come from different sensor modalities (e.g., visible, infrared, and thermal) and display strong domain-dependent variations. Conventional Low-Rank Adaptation (LoRA) is not expressive enough to model such modality-aware differences, which can result in insufficient exploitation of informative patterns. To overcome this limitation, we propose PV-FaultExpert, a domain-adaptive MLLM designed specifically for PV equipment fault analysis. PV-FaultExpert is built upon DyLoRA (Dynamic Expert Routing with LoRA), a dynamic routing strategy that reformulates standard LoRA into a shared low-rank component coupled with multiple expert-specific adapters. A routing module then selects expert paths according to input characteristics, allowing the model to adapt to diverse modalities while maintaining parameter efficiency. Moreover, we construct a PVfault diagnosis dataset via ChatGPT-4o-assisted chain-of-thought reasoning and subsequent expert verification, which both supports model training and enables rigorous evaluation of our method. Extensive experiments demonstrate that PV-FaultExpert consistently surpasses strong baselines, including GPT-4 and Claude-3, across multiple evaluation criteria, producing fault analysis reports that are accurate, interpretable, and aligned with safety-critical requirements.

1. Introduction

Photovoltaic (PV) equipment plays a vital role in ensuring the safe and efficient operation of modern power systems [1]. Faults in PV systems can lead not only to energy yield loss and increased operational cost, but also to significant safety hazards such as fire risks or grid instability [2]. Traditionally, PV fault diagnosis has relied on periodic manual inspections and expert-driven analysis based on accumulated field experience. However, these approaches are inherently limited in scalability, timeliness, and consistency, especially as PV systems become increasingly large-scale, decentralized, and diverse in deployment [3].
Unlike conventional substations, PV systems operate under highly dynamic environmental conditions—fluctuating irradiance, temperature changes, dust accumulation, and partial shading. These factors lead to non-stationary operating behaviors and variable fault manifestations [4].
Compounding this, PV equipment (e.g., modules, inverters, combiner boxes, transformers) is monitored through heterogeneous data sources such as RGB images, synthesized or real thermal imagery, sensor logs, and inspection records—often collected via different sensors with varying resolutions and modalities [5]. This multimodal, cross-device, and cross-condition variability introduces substantial challenges in learning robust fault representations and generating consistent diagnostics [6].
Recent advances in Multimodal Large Language Models [7,8,9,10] (MLLMs) offer a promising new paradigm for intelligent fault analysis. These models are capable of jointly processing visual and textual inputs, allowing them to interpret equipment states holistically and reason across data modalities. When applied to PV diagnosis, MLLMs have the potential to unify image understanding, thermal interpretation, and expert-level reporting into a single, interpretable framework. Compared to rule-based systems, MLLMs can generalize across scenarios, uncover subtle fault patterns, and generate human-readable reports with actionable insights.
However, directly adapting MLLMs to real-world PV fault analysis presents two major challenges. First, the data collected in PV systems is heterogeneous and multimodal, encompassing images from diverse sensor types (e.g., visible light, infrared, thermal simulation) and exhibiting domain-specific variability across component types, vendors, and environmental conditions. While parameter-efficient methods are crucial for adapting large models, existing approaches like Low-Rank Adaptation [11] (LoRA) apply a static, “one-size-fits-all” adaptation logic to all inputs. Specifically, LoRA injects a single pair of low-rank matrices into each target layer, forcing the model to learn a unified adaptation for all data types. This makes it difficult to capture the modality-aware distinctions crucial for accurate diagnosis when facing the diverse data in PV systems. Second, naive instruction tuning on limited PV data can cause catastrophic forgetting [12] and reduce the model’s general reasoning capabilities, especially under noisy supervision.
To overcome these limitations, we propose PV-FaultExpert, a domain-adaptive multimodal fault analysis model designed specifically for PV equipment. At the core of our method is DyLoRA, a dynamic expert routing mechanism that improves upon standard LoRA by introducing a shared adapter matrix and multiple expert-specific branches [13]. Based on the input modality and content, DyLoRA selectively activates expert paths via a lightweight routing module, enabling fine-grained adaptation to diverse input types without increasing parameter overhead. Moreover, to further support our PV-FaultExpert, we construct a PV diagnosis dataset featuring over 35,000 samples with heterogeneous visual inputs and structured diagnostic reports. We employ Chain-of-Thought [14] (CoT) enhanced prompting with ChatGPT-4o [15] to generate pseudo-reports and further refine them through expert validation, ensuring data reliability and domain fidelity.
The contributions of this paper are summarized as follows:
  • We develop a multimodal large model for PV fault analysis by designing a novel fine-tuning method and constructing a dedicated dataset. To the best of our knowledge, this is the first multimodal large model specifically developed for PV fault analysis.
  • We propose DyLoRA, a dynamic expert routing mechanism that enables input-aware adaptation. It advances beyond the limitations of standard LoRA by integrating a shared low-rank matrix with multiple, expert-specific branches, significantly improving model flexibility and robustness in complex multimodal PV scenarios.
  • We construct a PV fault dataset over 35,000 samples with CoT-based pseudo-annotations and multimodal inputs.
Extensive experiments show that PV-FaultExpert outperforms strong baselines, including GPT-4 [7], Claude-3 [8], and several open-source MLLMs, in generating accurate, interpretable, and safety-aware diagnostic reports under real-world PV conditions.
The remainder of this paper is organized as follows: Section 2 reviews the relevant literature and highlights the limitations this work seeks to overcome. Section 3 introduces the proposed method and the constructed dataset in detail. Section 4 presents the experimental results. Finally, Section 5 concludes this paper by summarizing the contributions of the proposed approach.

3. Method

3.1. Overview

PEFT methods have become a cornerstone for adapting large pre-trained models, with Low-Rank Adaptation (LoRA) [11] being one of the most prominent techniques. The core idea of LoRA [11] is to inject trainable, low-rank matrices into the model’s architecture, allowing for efficient adaptation without updating the entire set of original weights. Conventional LoRA [11] performs low-rank adaptation by introducing two trainable matrices, A and B, whose product A B serves as a learnable offset to the frozen pre-trained weight W 0 , as described as,
y ′ = y + Δ y = W 0 x + B A x ,
where y ∈ R d represents the output vector, and x ∈ R k denotes the input. The matrices A ∈ R r × k and B ∈ R d × r define the low-rank adaptation, where the rank r is significantly smaller than both d and k. To ensure that the model behaves identically to the pre-trained baseline at initialization, B is typically initialized with all zeros, while A is initialized using Kaiming Uniform initialization [47], resulting in an initial offset Δ y = 0 . However, the original LoRA design fine-tunes a unified set of parameters across all inputs, which becomes suboptimal in heterogeneous multimodal settings such as ours, where visual inputs originate from diverse sensor types and exhibit domain-specific variability.
To address this limitation, we propose a dynamic expert routing mechanism that decomposes the standard LoRA structure into a central shared matrix A and multiple specialized matrices B i , each dynamically selected based on the input characteristics (as illustrated in Figure 1). This asymmetric structure allows the model to maintain generalizable knowledge through the shared component while enabling adaptive specialization through expert-specific branches, thereby enhancing its ability to learn from heterogeneous multimodal data. We denote this architecture as follows:
W = W 0 + Δ W = W 0 + ∑ i = 1 N ω i · B i A ,
where W 0 represents the frozen pre-trained weight, and Δ W is the low-rank offset composed through a weighted aggregation of multiple expert branches. Each B i ∈ R d × r corresponds to an expert-specific projection, while A ∈ R r × k is a shared transformation matrix across all experts. The scalar ω i serves as a learned gating coefficient that dynamically adjusts the contribution of each expert B i based on the input context. The number of expert branches is defined by the hyperparameter N, which controls the model’s capacity for specialization under heterogeneous conditions.
Figure 1. Comparison of standard LoRA and our proposed DyLoRA architecture. (a) Standard LoRA injects low-rank adaptation by optimizing two trainable matrices A and B on frozen weights. (b) DyLoRA introduces a shared matrix A and multiple expert-specific matrices B i , dynamically selected via a routing mechanism based on heterogeneous multimodal representations. This design enhances adaptability to heterogeneous input modalities while preserving parameter efficiency. MHSA refers to the MultiHead Self-Attention module.

3.2. Dynamic Expert Routing with LoRA

To address the complexity of heterogeneous multimodal data—such as PV equipment images collected from different monitoring devices, imaging modalities (e.g., visible and infrared), sensor brands, and environmental conditions—we first determine the number of expert branches N, either manually based on domain-specific knowledge or through clustering algorithms like k-means applied to input features.
As shown in Figure 2, our DyLoRA is built upon a Mixture-of-Experts structure with dynamic routing, where each expert is implemented as a lightweight LoRA adapter B i , and all experts share a common low-rank matrix A. During training, the base model remains frozen, and only the expert modules and the routing mechanism are updated, enabling efficient parameter adaptation. The key innovation lies in the input-dependent dynamic routing mechanism, which allows the model to selectively activate and combine expert branches at inference time. Instead of statically assigning input to a fixed adapter, a gating network predicts soft routing weights conditioned on each input’s modality and characteristics. This enables DyLoRA to adaptively specialize its internal representations for different types of multimodal data while still benefiting from shared global knowledge captured by the common matrix A. The proposed DyLoRA can be expressed as follows:
y = W 0 x + ∑ i = 1 N ω i E i A x ,
ω i = sigmoid ( W g ⊤ x ) ,
where the output y is computed by combining a frozen pre-trained projection W 0 x with dynamically weighted expert contributions. Each expert branch E i ∈ R d × r operates on a shared matrix A ∈ R r × k , while ω i represents the input-dependent routing weight for expert i, allowing selective specialization. In addition, the routing weights ω = ( ω 1 , … , ω N ) are obtained by passing the intermediate token representation x through a trainable transformation W g ∈ R r × N , followed by a sigmoid layer. This mechanism enables the model to adaptively determine which expert(s) to activate based on the input characteristics.
Figure 2. The overall architecture of our proposed DyLoRA. Given heterogeneous multimodal inputs (e.g., visible, thermal, infrared images) and task instructions, we first extract unified multimodal representations. These representations are fed into a shared low-rank adapter A and multiple expert-specific adapters B i , where a trainable dynamic router predicts expert weights based on semantic content. The selected expert outputs are aggregated to adaptively generate downstream task outputs, achieving both parameter efficiency and robust generalization to diverse modalities. Attention, LN, and FNN refer to MultiHead Self-Attention module, Layer Normalization, and Feedforward Neural Network [48], respectively.

3.3. Dataset Curation

To support robust multimodal fault diagnosis for PV equipment, we curate a high-quality dataset that combines real-world annotated data and self-constructed pseudo-labeled samples, as illustrated in Figure 3b. Our dataset is primarily built upon PVEL-AD [49], a large-scale benchmark comprising over 30,000 near-infrared images of PV cells with corresponding defect annotations. While PVEL-AD provides valuable bounding box and class-level information, such annotations are not sufficient for downstream report generation tasks that require structured, human-readable diagnostic content. To address this, we leverage ChatGPT-4o [15] with CoT prompting to automatically transform raw detection outputs into rich diagnostic reports. Each generated report includes four key components, i.e., fault type, cause analysis, maintenance recommendation, and a concise summary, enabling better alignment with practical inspection workflows.
Figure 3. Illustration of the data construction pipeline. (a) We leverage domain-specific CoT prompts with ChatGPT-4o to automatically generate diverse power-related Q&A pairs, synthesize missing modalities (e.g., thermal from visible), and produce pseudo-reports for multimodal PV equipment analysis. (b) The annotation pipeline refines raw detection annotations into structured reports using step-by-step prompting, transforming bounding box metadata into rich semantic annotations that capture defect type, cause, and spatial extent.
Moreover, to further enhance the dataset’s modality diversity and component coverage, we additionally collect several hundred images of PV equipment (e.g., PV transformers and PV panels) and their corresponding thermal infrared views from publicly available online resources. These images capture broader equipment types and more complex visual patterns under varying real-world conditions (e.g., illumination, temperature). For these samples, we again employ ChatGPT-4o, a closed-source multimodal model, to generate pseudo-reports based on RGB and thermal content. As shown in Figure 3a, each report is generated using carefully crafted CoT-style prompts that guide the model to perform appearance analysis, thermal interpretation, and risk assessment. To ensure annotation quality, we adopt a two-stage report construction and validation workflow. First, pseudo-reports are automatically generated using CoT-enhanced prompting with ChatGPT-4o. Second, three domain experts with Master’s degrees in electrical engineering verify the generated reports, with primary focus on the self-curated subset. When a report is judged as incorrect by two or more experts, it is manually corrected and then re-entered into the expert checking process until it passes the review. This step guarantees domain fidelity and factual reliability across the dataset.
As a result, the final dataset includes structured, multimodal samples across diverse PV equipment, with each entry composed of RGB/thermal imagery and a corresponding diagnostic report. This curated dataset serves as the foundation for fine-tuning our PV-FaultExpert model, providing rich supervision signals for learning domain-specific reasoning patterns.

3.4. Loss Function

3.4.1. Pre-Training for Feature Alignment

Given the characteristics of our dataset—which includes pseudo-labeled diagnostic reports and heterogeneous multimodal imagery captured from various sensor types and modalities—establishing robust image–text correspondence is particularly crucial. The inherent noise in pseudo-labels, combined with the distributional shifts across different data sources (e.g., visible and thermal imaging from distinct devices), makes direct instruction tuning suboptimal. To address these challenges, we introduce a contrastive pre-training stage to explicitly align visual and textual representations. This step ensures that the model learns to associate semantically relevant visual cues with their corresponding diagnostic descriptions, even when labels are weak or data is heterogeneous. We follow the vision–language alignment framework of LLaVA [9], which optimizes the following contrastive objective:
L align = − 1 M ∑ i = 1 M log exp ( sim ( v i , t i ) / τ ) ∑ j = 1 M exp ( sim ( v i , t j ) / τ ) ,
where v i and t i represent the image and text embeddings of the i-th pair, sim ( · , · ) denotes cosine similarity, and τ is a learnable temperature parameter. This alignment mechanism provides a strong initialization for downstream expert-aware instruction tuning.

3.4.2. Fine-Tuning on Fault Analysis

We perform diagnostic report generation based on multimodal inputs from PV equipment, where visual and textual modalities must be jointly modeled to enable accurate fault description. Given an input image I, we first encode it using a pre-trained vision encoder (i.e., CLIP ViT-L/14 [18]), obtaining visual features V = Encoder vis ( I ) . These features are then projected into the language embedding space through a learnable projection layer,
H 0 = Proj ( Encoder vis ( I ) ) ,
y ^ i j = Decoder LLM ( y ^ < t , H 0 , T ) ,
where H 0 is the projected visual representation and T = { t 1 , t 2 , … , t n } is the tokenized instruction prompt. The decoder [20] generates the output sequence token-by-token, conditioned on both the visual input and the prompt.
To train the model, we use a supervised cross-entropy objective that minimizes the difference between the predicted and ground-truth tokens,
L gen = − 1 l ∑ i = 1 l ∑ j = 1 V d y i j log ( y ^ i j ) ,
where l is the length of the report, V d is the vocabulary size, y i j is the ground-truth one-hot vector, and y ^ i j is the predicted probability for the j-th token at position i.
Overall, our model is trained by jointly minimizing the objective consisting of a report generation loss and a multimodal alignment loss that promotes cross-modal semantic consistency,
L total = L gen + L align .

4. Experiments

4.1. MLLMs for Fault Analysis

MLLMs sometimes lack sufficient electrical domain expertise and contextual relevance in fault diagnosis. For example, in analyzing faults within PV equipment—such as a damaged transformer breather—the model may generate a generic response like “Breather damage detected, please repair promptly,” without providing root cause analysis or specific repair guidance. By integrating a domain-specific knowledge base tailored for PV equipment, these models can produce more professional, accurate, and actionable fault analysis reports. Specifically, the analysis report includes the fault type, fault cause, and maintenance recommendations. An example report is as follows: (i) Fault Type: Damage to the breather of a PV panel transformer. (ii) Fault Cause: Excessive internal pressure leading to aging or failure of the breather’s sealing material. (iii) Maintenance Recommendations: Inspect the breather for cracks or physical damage, and replace it if necessary. Check the internal pressure of the transformer to ensure it remains within the normal operating range.
MLLMs that have been fine-tuned with domain-specific knowledge are capable of providing more precise fault descriptions, identifying specific causes, and offering targeted maintenance recommendations, thereby greatly enhancing the professionalism and applicability of the analysis. Moreover, our proposed efficient fine-tuning method leverages dynamic routing to effectively integrate heterogeneous multimodal data, further enhancing the LLM’s capability for fault representation learning.

4.2. Datasets

The dataset used in this study is primarily derived from PVEL-AD [49], a large-scale dataset containing over 30,000 near-infrared images of PV cells, along with corresponding defect annotations. ChatGPT-4o [15], with the help of CoT, converts raw detection annotations into structured diagnostic reports that combine fault types, cause analysis, and maintenance recommendations to support more interpretable and actionable downstream applications. Furthermore, to enrich the dataset, we collected several hundred images of PV equipment (e.g., PV transformers and PV panels) and their corresponding thermal images from publicly available online sources. We then employed ChatGPT-4o, the latest and most advanced closed-source multimodal model, to generate pseudo-labels in the form of diagnostic reports based on the visual content. To ensure the accuracy and reliability of these generated reports, three electrical experts were invited to conduct a secondary review, during which any unreasonable or inaccurate content was carefully identified and corrected. All three experts followed the same inspection and evaluation criteria defined in the Chinese national standard Code of Operation for Photovoltaic Power Station (GB/T 38335-2019) [50]. The self-curated dataset in this paper is available upon request by contacting the authors.

4.3. Evaluation Metrics

The evaluation measures model performance on the test set by computing similarity scores between the generated reports and our annotated reference reports. Traditional metrics such as BLEU [51] and ROUGE [52] are primarily designed for tasks like machine translation and text summarization. However, due to the unique characteristics and domain-specific requirements of substation fault analysis, these general-purpose metrics fall short in effectively evaluating the quality of generated diagnostic reports. To structure our assessment, we followed the comprehensive evaluation framework proposed in [53,54]. Drawing insights from discussions with electrical power experts and referencing evaluation criteria from related domains such as healthcare [55,56], education [57], and software engineering [58], we then identified four essential qualities that define a high-quality substation fault analysis report:
  • Accuracy evaluates whether the generated report correctly identifies the fault type and cause. Let T R be the set of key diagnostic elements (e.g., fault types, causes) from the reference report, and T R ^ is from the generated report. Accuracy is defined as follows:
    Accuracy = | T R ∩ T R ^ | | T R | .
  • Clarity evaluates the readability and structural coherence of the generated text. Following recent studies [59,60,61], we use ChatGPT-4o to score the syntactic fluency of the predicted reports.
  • Completeness measures whether the generated report includes all necessary information that appears in the ground truth. Let C R be the set of expected components and analysis items from the reference report, and C R ^ is from the generated report. The computation follows the same set-overlap formulation as Accuracy,
    Completeness = | C R ∩ C R ^ | | C R | .
  • Practicality assesses whether the report provides clear and actionable maintenance instructions. Let A R be the set of expected repair actions in the reference report, and A R ^ is those extracted from the generated report. Similar to Accuracy, it is computed based on the overlap between expected and generated repair instructions:
    Practicality = | A R ∩ A R ^ | | A R | .
  • Average Score is a metric for the report’s overall quality, defined as the arithmetic mean of the four criteria above:
    Average   Score = Acc . + Cla . + Com . + Pra . 4 .
These four dimensions form the foundation of our evaluation framework for substation fault reports. To support consistent and objective assessment, we developed a five-point scoring system for each criterion, ranging from 1 (poor) to 5 (excellent), ensuring that report quality can be quantitatively and independently evaluated. To ensure uniformity across all metrics, while Clarity is rated directly on this 1-to-5 scale, our other objective metrics are first calculated as a value x in the range [ 0 ,   1 ] . These are then mapped to the final scale using a logarithmic transformation, which better reflects the greater significance of score improvements at the higher end of the spectrum. The scaling function is defined as:
S c o r e s c a l e d = 1 + ( α − 1 ) · log ( x + 1 ) log ( 2 ) ,
where α is the maximum score of the scale ( α = 5 ).

4.4. Implementation Details

We adopted LLaVA-1.5-7B as the base architecture for developing our fault analysis model. To adapt it to our constructed dataset, we applied our proposed PEFT method. The training was conducted using four NVIDIA A100 GPUs (Nvidia, Santa Clara, CA, USA), with a learning rate of 1 × 10 − 4  , a batch size of 10, and for 20 epochs. The fine-tuning configuration included a rank of 64, an alpha value of 16, and a dropout rate of 0.05 , enabling efficient adaptation of the model to the substation domain without incurring significant computational cost.

4.5. Baselines

To evaluate the effectiveness of our proposed model, we conducted a comparative study with six representative baseline models, including GPT-4 [7], Claude-3 [62,63,64], LLaVA-1.5-7B [9,65], VisualGLM-6B [66,67], Qwen2-VL-7B [10,68], and MiniGPT-4 [17,69]. We randomly selected 1000 samples from the dataset as the testing set, covering various fault types observed in PV equipment. Each baseline model was used to generate corresponding fault analysis reports based on these samples. To ensure an objective evaluation, three engineers with expertise in PV systems and electrical engineering were invited to review and rate the generated reports according to a predefined set of assessment criteria.

4.6. Reasoning Analysis

As shown in Figure 4, our PV-FaultExpert effectively interprets heterogeneous multimodal inputs and generates structured diagnostic reports that align closely with human expert reasoning. The model accurately identifies key visual attributes from RGB images, incorporates auxiliary cues from thermal data, and integrates domain knowledge to provide fault descriptions, risk assessments, and safety-aware conclusions. These results demonstrate the model’s capability not only to capture fine-grained visual features but also to produce coherent and actionable outputs, highlighting its practical utility in real-world PV equipment inspection.
Figure 4. Example outputs generated by PV-FaultExpert on multimodal PV equipment data. Each case includes the heterogeneous multimodal input and the predicted fault analysis report. The results demonstrate PV-FaultExpert’s ability to interpret heterogeneous multimodal inputs and produce detailed, safety-aware assessments.

4.7. Comparative Results

Model performance on PV fault analysis requires not only an accurate understanding of fault types, but also the ability to generate clear, complete, and practical diagnostic reports. To ensure a fair comparison, all baseline models were evaluated on the same dataset using a standardized evaluation protocol involving four key dimensions, i.e., Accuracy, Clarity, Completeness, and Practicality. As shown in Table 1, our approach first establishes its superiority over traditional deep learning methods, significantly outperforming both classic CNN-based and transformer-based architectures. Our proposed model PV-FaultExpert, trained with domain-specific knowledge and a PEFT strategy, achieves the best overall performance across all metrics. Specifically, PV-FaultExpert outperforms GPT-4 and Claude-3 by an average margin of 0.98 and 0.92 , respectively, demonstrating significant advantages over even the most advanced closed-source models. Additionally, when comparing PV-FaultExpert with LLaVA1.5-7B (fine-tuned), which was also adapted to our dataset, we observe consistent improvements of 0.77 in Accuracy, 0.71 in Clarity, 0.84 in Completeness, and 0.89 in Practicality. These results highlight the effectiveness of our fine-tuning method and the importance of incorporating domain-specific knowledge into multimodal large models. Furthermore, to enable a fairer evaluation of report generation performance in more realistic settings, we additionally constructed a hard and realistic test subset. Specifically, we invited five experts to manually annotate 50 PV multimodal images using free writing styles. This subset was fully annotated by humans without AI assistance. We then re-evaluated all methods on this harder test set. As shown in Table 1, performance decreases for all methods on this subset. The drop is more pronounced for the Clarity metric, which can be attributed to the increased variability introduced by diverse and unconstrained writing styles.
Table 1. Quantitative comparison results (higher is better) for PV fault analysis.

4.8. Ablation Study

As shown in Table 2, the full version of our model, PV-FaultExpert, outperforms all ablated variants across all evaluation dimensions. The fusion integrates specialized insights from multiple experts by aggregating their outputs via a router-determined weighted sum, forming a coherent diagnostic signal that is essential for analyzing complex, multimodal faults. Specifically, removing the fusion operation after routing (Row a) leads to a noticeable drop in average score from 4.41 to 3.93 . This result indicates that our MoE framework is essential for cultivating distinct “experts” capable of mastering the unique features of each data modality. The Router enables conditional computation by analyzing an input to weight and activate only the most relevant experts for a given task. Further removing the dynamic routing mechanism (Row b) results in a decrease of 0.16 in the overall average. This proves that the router plays a critical role in intelligently analyzing each input and assigning the task to the most relevant expert(s), ensuring the right expertise is applied to the right problem. MoE enables specialization by using an ensemble of experts (low-rank adapters) to overcome the representational bottleneck of processing heterogeneous data. When the mixture-of-experts (MoE) module is disabled (Row c), performance drops even further to 3.53 . This confirms the importance of this step in integrating the specialized insights from the selected experts into a single, coherent diagnosis. These results demonstrate that each component of PV-FaultExpert contributes positively to model performance, and that the full architecture is essential for generating high-quality PV fault analysis reports.
Table 2. Ablation studies on our proposed efficient fine-tuning method.
In addition, to investigate the impact of expert quantity in our proposed DyLoRA framework, we conduct some ablation studies with varying numbers of expert branches N ∈ { 1 , 2 , 3 , 4 , 5 } . As shown in Figure 5, the model performance improves steadily from N = 1 to N = 3 , reaching the highest average score of 4.41 at N = 3 . This suggests that incorporating multiple specialized expert branches helps the model better adapt to the heterogeneity in PV multimodal data. However, increasing the number of experts beyond three does not lead to further gains—in fact, a slight performance drop is observed. This may be attributed to over-fragmentation of the data across too many experts, which could reduce the effectiveness of individual branches. Therefore, selecting an appropriate number of expert adapters is essential to balance model capacity and generalization. Also, we use a routing confusion matrix to verify whether DyLoRA experts are specialized for different modalities. Specifically, we re-ran inference on the above hard test set and recorded the routing scores after the sigmoid activation. Based on these scores, we constructed a routing confusion matrix; see Table 3. The results show that, after fine-tuning, the experts in our proposed DyLoRA indeed focus on different modalities, which provides empirical evidence of expert specialization. This further indicates that DyLoRA is beneficial for handling heterogeneous multimodal data.
Figure 5. The performance of our DyLoRA under different numbers of expert branches.
Table 3. Routing confusion matrix on the hard test set (human-annotated test subset).

4.9. Computational Efficiency

As our experimental results show (see Table 4), while our multimodal large model approach is more computationally demanding compared to traditional deep learning methods (e.g., transformers or CNN with LSTM), the performance it achieves is substantially superior. Leveraging large models to push the boundaries of performance in various downstream tasks has become a common and well-established trend in recent research [55,72]. On the one hand, in real-world applications, it is standard practice to deploy large models on cloud-based infrastructure, which effectively meets their computational requirements. On the other hand, a significant and growing body of work is investigating the distillation of knowledge from large models into smaller, domain-specific ones [73,74]. Recent studies such as SmolVLM [75], Nano-R1, and TinyLLaVA [76] have demonstrated that large multimodal model knowledge can be distilled into compact multimodal models for edge deployment. After quantization, these models can even run on resource-constrained devices with CPU-only configurations and around 4 GB of memory. Exploring such techniques could further enhance the deployment possibilities of powerful models like ours. In summary, the rich knowledge inherent in large models is invaluable for specialized, vertical domains such as PV fault analysis. The ongoing research into making these models more efficient and accessible is a highly promising direction.
Table 4. Computational efficiency of the evaluated models.

5. Conclusions

In this work, we propose PV-FaultExpert, a domain-adaptive multimodal model tailored for PV equipment fault analysis. Built upon a curated dataset of multimodal information related to PV equipment, our method incorporates a PEFT strategy to effectively adapt large models to the PV system. At the core of our approach is a LoRA-based fine-tuning method enhanced with a mixture-of-experts mechanism, which enables the model to selectively attend to different components during training. This adaptive focus improves both the precision of fine-tuning and the efficiency of inference, making the model more capable of handling domain-specific challenges in PV diagnostics. Experiments demonstrate that PV-FaultExpert consistently outperforms several recent popular AI models across multiple evaluation metrics. These results confirm the effectiveness of our design in generating more accurate and actionable fault analysis reports for PV equipment.

6. Limitations

Our approach is based on pseudo-labeled reports generated with LLM assistance and subsequently verified by experts, which may still introduce residual bias from the synthetic annotation pipeline. Although we further evaluate on a small human-annotated subset, larger fully human-labeled benchmark datasets would provide a stronger basis for validating real-world diagnostic reliability. In addition, the current dataset is collected under specific hardware settings and operational conditions. PV fault appearances and thermal patterns may vary across module vendors, inverter types, installation configurations, and geographic regions with different climate and irradiance characteristics. As a result, the generalization of the proposed model may be limited when deployed in unseen domains. Future work will expand cross-vendor and cross-region data coverage and explore domain adaptation strategies.

Author Contributions

Conceptualization, J.W. and Y.C.; methodology, J.W. and Q.M.; software, Q.M.; validation, Q.M. and J.W.; formal analysis, J.Z.; investigation, M.Y.; resources, J.Z. and M.C.; data curation, Q.M. and M.C.; writing—original draft preparation, J.W. and Q.M.; writing—review and editing, J.W. and Q.M.; visualization, Y.C.; supervision, J.Z. and M.C.; project administration, J.W.; funding acquisition, J.W. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by the Science and Technology Project of State Grid Corporation of China, grant number 5400-20241918A-1-1-ZN.

Data Availability Statement

The original contributions presented in this study are included in the article. Further inquiries can be directed to the corresponding author.

Conflicts of Interest

Author Junjian Wu was employed by State Grid Wenzhou Electric Power Supply Company. Author Yiwei Chen was employed by State Grid Ruian Electric Power Supply Company. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest. The authors declare that this study received funding from State Grid Corporation of China. The funder was not involved in the study design, collection, analysis, interpretation of data, the writing of this article or the decision to submit it for publication.

References

  1. Osmani, K.; Haddad, A.; Lemenand, T.; Castanier, B.; Ramadan, M. A review on maintenance strategies for PV systems. Sci. Total Environ. 2020, 746, 141753. [Google Scholar]
  2. Aram, M.; Zhang, X.; Qi, D.; Ko, Y. A state-of-the-art review of fire safety of photovoltaic systems in buildings. J. Clean. Prod. 2021, 308, 127239. [Google Scholar] [CrossRef] [Scilit]
  3. Shafiullah, M.; Ahmed, S.D.; Al-Sulaiman, F.A. Grid integration challenges and solution strategies for solar PV systems: A review. IEEE Access 2022, 10, 52233–52257. [Google Scholar] [CrossRef] [Scilit]
  4. Kheirrouz, M.; Melino, F.; Ancona, M.A. Fault detection and diagnosis methods for green hydrogen production: A review. Int. J. Hydrogen Energy 2022, 47, 27747–27774. [Google Scholar] [CrossRef] [Scilit]
  5. Chang, Z.; Han, T. Prognostics and health management of photovoltaic systems based on deep learning: A state-of-the-art review and future perspectives. Renew. Sustain. Energy Rev. 2024, 205, 114861. [Google Scholar] [CrossRef] [Scilit]
  6. Li, Y.; Cao, Y.; Cui, X.; Zhang, Y.; Mukhopadhyay, S.C.; Li, Y.; Cui, L.; Liu, Z.; Li, S. Semantic consistency guided hybrid-invariant transformer for domain adaptation in multi-view echo quality assessment. IEEE Trans. Instrum. Meas. 2025, 74, 4008519. [Google Scholar]
  7. Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F.L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al. Gpt-4 technical report. arXiv 2023, arXiv:2303.08774. [Google Scholar] [CrossRef] [Scilit]
  8. Anthropic. Claude 3 Haiku: Our Fastest Model Yet. 2024. Available online: https://www.anthropic.com/news/claude-3-haiku (accessed on 30 July 2025).
  9. Liu, H.; Li, C.; Wu, Q.; Lee, Y.J. Visual instruction tuning. In Proceedings of the 37th International Conference on Neural Information Processing Systems, New Orleans, LA, USA, 10–16 December 2023. [Google Scholar]
  10. Wang, P.; Bai, S.; Tan, S.; Wang, S.; Fan, Z.; Bai, J.; Chen, K.; Liu, X.; Wang, J.; Ge, W.; et al. Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution. arXiv 2024, arXiv:2409.12191. [Google Scholar]
  11. Hu, E.J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; Chen, W. LoRA: Low-Rank Adaptation of Large Language Models. In Proceedings of the International Conference on Learning Representations, Virtual, 25–29 April 2022. [Google Scholar]
  12. Liang, J.; Huang, W.; Wan, G.; Yang, Q.; Ye, M. LoRASculpt: Sculpting LoRA for Harmonizing General and Specialized Knowledge in Multimodal Large Language Models. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), Nashville, TN, USA, 11–15 June 2025. [Google Scholar]
  13. Xue, F.; Shi, Z.; Wei, F.; Lou, Y.; Liu, Y.; You, Y. Go wider instead of deeper. In Proceedings of the AAAI Conference on Artificial Intelligence, Online, 22 February–1 March 2022. [Google Scholar]
  14. Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Ichter, B.; Xia, F.; Chi, E.; Le, Q.V.; Zhou, D. Chain-of-thought prompting elicits reasoning in large language models. Adv. Neural Inf. Process. Syst. 2022, 35, 24824–24837. [Google Scholar]
  15. OpenAI. ChatGPT-4o. 2025. Available online: https://chat.openai.com/ (accessed on 30 July 2025).
  16. Li, J.; Li, D.; Xiong, C.; Hoi, S. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Proceedings of the International Conference on Machine Learning (PMLR), Baltimore, ML, USA, 17–23 July 2022. [Google Scholar]
  17. Zhu, D.; Ye, M.; Shen, X.; Li, X.; Elhoseiny, M. MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models. arXiv 2023, arXiv:2304.10592. [Google Scholar]
  18. Radford, A.; Kim, J.W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning, PMLR, Virtual, 18–24 July 2021. [Google Scholar]
  19. Fang, Y.; Wang, W.; Xie, B.; Sun, Q.; Wu, L.; Wang, X.; Huang, T.; Wang, X.; Cao, Y. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada, 18–22 June 2023. [Google Scholar]
  20. Chiang, W.L.; Li, Z.; Lin, Z.; Sheng, Y.; Wu, Z.; Zhang, H.; Zheng, L.; Zhuang, S.; Zhuang, Y.; Gonzalez, J.E.; et al. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality. 2023. Available online: https://lmsys.org/blog/2023-03-30-vicuna/ (accessed on 14 April 2023).
  21. Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; et al. Llama 2: Open foundation and fine-tuned chat models. arXiv 2023, arXiv:2307.09288. [Google Scholar] [CrossRef] [Scilit]
  22. Zhu, K.; Wang, Y.; Sun, Y.; Chen, Q.; Liu, J.; Zhang, G.; Wang, J. Continual sft matches multimodal rlhf with negative supervision. In Proceedings of the Computer Vision and Pattern Recognition Conference, Nashville, TN, USA, 11–15 June 2025. [Google Scholar]
  23. Sun, Y.; Zhang, H.; Chen, Q.; Zhang, X.; Sang, N.; Zhang, G.; Wang, J.; Li, Z. Improving multi-modal large language model through boosting vision capabilities. arXiv 2024, arXiv:2410.13733. [Google Scholar] [CrossRef] [Scilit]
  24. Chen, Z.; Wu, J.; Wang, W.; Su, W.; Chen, G.; Xing, S.; Zhong, M.; Zhang, Q.; Zhu, X.; Lu, L.; et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 16–22 June 2024. [Google Scholar]
  25. Wang, Q.; Zhang, J.; Du, J.; Zhang, K.; Li, R.; Zhao, F.; Zou, L.; Xie, C. A fine-tuned multimodal large model for power defect image-text question-answering. Signal Image Video Process. 2024, 18, 9191–9203. [Google Scholar]
  26. Mu, J.; Wang, W.; Liu, W.; Yan, T.; Wang, G. Multimodal Large Language Model with LoRA Fine-Tuning for Multimodal Sentiment Analysis. ACM Trans. Intell. Syst. Technol. 2024, 16, 139. [Google Scholar]
  27. Han, Z.; Gao, C.; Liu, J.; Zhang, J.; Zhang, S.Q. Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv 2024, arXiv:2403.14608. [Google Scholar]
  28. Wang, J.; Song, Q.; Qian, L.; Li, H.; Peng, Q.; Zhang, J. SubstationAI: Multimodal Large Model-Based Approaches for Analyzing Substation Equipment Faults. arXiv 2024, arXiv:2412.17077. [Google Scholar]
  29. Biderman, D.; Portes, J.; Ortiz, J.J.G.; Paul, M.; Greengard, P.; Jennings, C.; King, D.; Havens, S.; Chiley, V.; Frankle, J.; et al. Lora learns less and forgets less. arXiv 2024, arXiv:2405.09673. [Google Scholar] [CrossRef] [Scilit]
  30. Houlsby, N.; Giurgiu, A.; Jastrzebski, S.; Morrone, B.; De Laroussilhe, Q.; Gesmundo, A.; Attariyan, M.; Gelly, S. Parameter-efficient transfer learning for NLP. In Proceedings of the International Conference on Machine Learning, PMLR, Long Beach, CA, USA, 9–15 June 2019. [Google Scholar]
  31. He, J.; Zhou, C.; Ma, X.; Berg-Kirkpatrick, T.; Neubig, G. Towards a unified view of parameter-efficient transfer learning. arXiv 2021, arXiv:2110.04366. [Google Scholar]
  32. Li, X.L.; Liang, P. Prefix-tuning: Optimizing continuous prompts for generation. arXiv 2021, arXiv:2101.00190. [Google Scholar] [CrossRef] [Scilit]
  33. Liu, X.; Ji, K.; Fu, Y.; Du, Z.; Yang, Z.; Tang, J. P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks. arXiv 2021, arXiv:2110.07602. [Google Scholar]
  34. Jia, M.; Tang, L.; Chen, B.C.; Cardie, C.; Belongie, S.; Hariharan, B.; Lim, S.N. Visual prompt tuning. In Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel, 23–27 October 2022; Springer: Cham, Switzerland, 2022. [Google Scholar]
  35. Das, S.S.S.; Zhang, R.H.; Shi, P.; Yin, W.; Zhang, R. Unified low-resource sequence labeling by sample-aware dynamic sparse finetuning. arXiv 2023, arXiv:2311.03748. [Google Scholar]
  36. Lawton, N.; Kumar, A.; Thattai, G.; Galstyan, A.; Steeg, G.V. Neural architecture search for parameter-efficient fine-tuning of large pre-trained language models. arXiv 2023, arXiv:2305.16597. [Google Scholar]
  37. Zhang, Q.; Chen, M.; Bukharin, A.; Karampatziakis, N.; He, P.; Cheng, Y.; Chen, W.; Zhao, T. Adalora: Adaptive budget allocation for parameter-efficient fine-tuning. arXiv 2023, arXiv:2303.10512. [Google Scholar]
  38. Mao, Y.; Mathias, L.; Hou, R.; Almahairi, A.; Ma, H.; Han, J.; Yih, W.t.; Khabsa, M. Unipelt: A unified framework for parameter-efficient language model tuning. arXiv 2021, arXiv:2110.07577. [Google Scholar]
  39. Chen, S.; Jie, Z.; Ma, L. Llava-mole: Sparse mixture of lora experts for mitigating data conflicts in instruction finetuning mllms. arXiv 2024, arXiv:2401.16160. [Google Scholar]
  40. Sheng, Y.; Cao, S.; Li, D.; Hooper, C.; Lee, N.; Yang, S.; Chou, C.; Zhu, B.; Zheng, L.; Keutzer, K.; et al. S-lora: Serving thousands of concurrent lora adapters. arXiv 2023, arXiv:2311.03285. [Google Scholar]
  41. Lopes, F.; Rocha, P.; Coelho, A. Towards Automated Visual Inspection of Electrical Grid Assets for the Smart Grid - An Application to HV Insulators. In Proceedings of the 2024 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm), Oslo, Norway, 17–20 September 2024. [Google Scholar]
  42. Zheng, J.; Wu, H.; Zhang, H.; Wang, Z.; Xu, W. Insulator-Defect Detection Algorithm Based on Improved YOLOv7. Sensors 2022, 22, 8801. [Google Scholar] [CrossRef] [Scilit]
  43. Merkelbach, S.; Diedrich, A.; Sztyber-Betley, A.; Travé-Massuyès, L.; Chanthery, E.; Niggemann, O.; Dumitrescu, R. Using Multi-Modal LLMs to Create Models for Fault Diagnosis (Short Paper). In Proceedings of the 35th International Conference on Principles of Diagnosis and Resilient Systems (DX 2024), Vienna, Austria, 4–7 November 2024; Schloss Dagstuhl–Leibniz-Zentrum für Informatik: Wadern, Germany, 2024. [Google Scholar]
  44. Wang, H.; Li, C.; Li, Y.F.; Tsung, F. An Intelligent Industrial Visual Monitoring and Maintenance Framework Empowered by Large-Scale Visual and Language Models. IEEE Trans. Ind. Cyber-Phys. Syst. 2024, 2, 166–175. [Google Scholar] [CrossRef] [Scilit]
  45. Jin, H.; Kim, K.; Kwon, J. GridMind: LLMs-Powered Agents for Power System Analysis and Operations. In Proceedings of the SC’25 Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis, St. Louis, MO, USA, 16–21 November 2025. [Google Scholar]
  46. Wen, Y.; Chen, X. X-GridAgent: An LLM-Powered Agentic AI System for Assisting Power Grid Analysis. arXiv 2025, arXiv:2512.20789. [Google Scholar]
  47. He, K.; Zhang, X.; Ren, S.; Sun, J. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 7–13 December 2015. [Google Scholar]
  48. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, Ł.; Polosukhin, I. Attention is all you need. Adv. Neural Inf. Process. Syst. 2017, 30, 5998–6008. [Google Scholar]
  49. Su, B.; Zhou, Z.; Chen, H. PVEL-AD: A large-scale open-world dataset for photovoltaic cell anomaly detection. IEEE Trans. Ind. Inform. 2022, 19, 404–413. [Google Scholar]
  50. GB/T 38335-2019; Code of Operation for Photovoltaic Power Station. Standards Press of China: Beijing, China, 2019.
  51. Papineni, K.; Roukos, S.; Ward, T.; Zhu, W.J. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, PA, USA, 7–12 July 2002. [Google Scholar]
  52. Lin, C.Y. Rouge: A package for automatic evaluation of summaries. In Proceedings of the Text Summarization Branches Out, Barcelona, Spain, 25–26 July 2004. [Google Scholar]
  53. Lynch, S.; Savary-Bataille, K.; Leeuw, B.; Argyle, D. Development of a questionnaire assessing health-related quality-of-life in dogs and cats with cancer. Vet. Comp. Oncol. 2011, 9, 172–182. [Google Scholar] [CrossRef] [Scilit]
  54. Lahat, A.; Sharif, K.; Zoabi, N.; Shneor Patt, Y.; Sharif, Y.; Fisher, L.; Shani, U.; Arow, M.; Levin, R.; Klang, E. Assessing generative pretrained transformers (GPT) in clinical decision-making: Comparative analysis of GPT-3.5 and GPT-4. J. Med. Internet Res. 2024, 26, e54571. [Google Scholar] [CrossRef] [Scilit]
  55. Zhang, H.; Chen, J.; Jiang, F.; Yu, F.; Chen, Z.; Li, J.; Chen, G.; Wu, X.; Zhang, Z.; Xiao, Q.; et al. Huatuogpt, towards taming language model to be a doctor. arXiv 2023, arXiv:2305.15075. [Google Scholar] [CrossRef] [Scilit]
  56. Tiffin, P.A.; Finn, G.M.; McLachlan, J.C. Evaluating professionalism in medical undergraduates using selected response questions: Findings from an item response modelling study. BMC Med. Educ. 2011, 11, 43. [Google Scholar] [CrossRef] [Scilit]
  57. Ghafourian, Y.; Hanbury, A.; Knoth, P. Readability measures as predictors of understandability and engagement in searching to learn. In Linking Theory and Practice of Digital Libraries, Proceedings of the 27th International Conference on Theory and Practice of Digital Libraries, TPDL 2023, Zadar, Croatia, 26–29 September 2023; Springer: Cham, Switzerland, 2023; pp. 173–181. [Google Scholar]
  58. Karunaratne, S.; Dharmarathna, D. A review of comprehensiveness, user-friendliness, and contribution for sustainable design of whole building environmental life cycle assessment software tools. Build. Environ. 2022, 212, 108784. [Google Scholar] [CrossRef] [Scilit]
  59. Bai, Y.; Ying, J.; Cao, Y.; Lv, X.; He, Y.; Wang, X.; Yu, J.; Zeng, K.; Xiao, Y.; Lyu, H.; et al. Benchmarking foundation models with language-model-as-an-examiner. Adv. Neural Inf. Process. Syst. 2023, 36, 78142–78167. [Google Scholar]
  60. Li, Z.; Xu, X.; Shen, T.; Xu, C.; Gu, J.C.; Lai, Y.; Tao, C.; Ma, S. Leveraging large language models for NLG evaluation: Advances and challenges. arXiv 2024, arXiv:2401.07103. [Google Scholar]
  61. Bavaresco, A.; Bernardi, R.; Bertolazzi, L.; Elliott, D.; Fernández, R.; Gatt, A.; Ghaleb, E.; Giulianelli, M.; Hanna, M.; Koller, A.; et al. Llms instead of human judges? a large scale empirical study across 20 nlp evaluation tasks. arXiv 2024, arXiv:2406.18403. [Google Scholar] [CrossRef] [Scilit]
  62. Bai, Y.; Jones, A.; Ndousse, K.; Askell, A.; Chen, A.; DasSarma, N.; Drain, D.; Fort, S.; Ganguli, D.; Henighan, T.; et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv 2022, arXiv:2204.05862. [Google Scholar] [CrossRef] [Scilit]
  63. Ziegler, D.M.; Stiennon, N.; Wu, J.; Brown, T.B.; Radford, A.; Amodei, D.; Christiano, P.; Irving, G. Fine-tuning language models from human preferences. arXiv 2019, arXiv:1909.08593. [Google Scholar]
  64. Askell, A.; Bai, Y.; Chen, A.; Drain, D.; Ganguli, D.; Henighan, T.; Jones, A.; Joseph, N.; Mann, B.; DasSarma, N.; et al. A general language assistant as a laboratory for alignment. arXiv 2021, arXiv:2112.00861. [Google Scholar] [CrossRef] [Scilit]
  65. Liu, H.; Li, C.; Li, Y.; Lee, Y.J. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 16–22 June 2024. [Google Scholar]
  66. Du, Z.; Qian, Y.; Liu, X.; Ding, M.; Qiu, J.; Yang, Z.; Tang, J. Glm: General language model pretraining with autoregressive blank infilling. arXiv 2021, arXiv:2103.10360. [Google Scholar]
  67. Ding, M.; Yang, Z.; Hong, W.; Zheng, W.; Zhou, C.; Yin, D.; Lin, J.; Zou, X.; Shao, Z.; Yang, H.; et al. Cogview: Mastering text-to-image generation via transformers. Adv. Neural Inf. Process. Syst. 2021, 34, 19822–19835. [Google Scholar]
  68. Bai, J.; Bai, S.; Yang, S.; Wang, S.; Tan, S.; Wang, P.; Lin, J.; Zhou, C.; Zhou, J. Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond. arXiv 2023, arXiv:2308.12966. [Google Scholar]
  69. Chen, J.; Zhu, D.; Shen, X.; Li, X.; Liu, Z.; Zhang, P.; Krishnamoorthi, R.; Chandra, V.; Xiong, Y.; Elhoseiny, M. Minigpt-v2: Large language model as a unified interface for vision-language multi-task learning. arXiv 2023, arXiv:2310.09478. [Google Scholar]
  70. Xue, Y.; Xu, T.; Rodney Long, L.; Xue, Z.; Antani, S.; Thoma, G.R.; Huang, X. Multimodal recurrent model with attention for automated radiology report generation. In Medical Image Computing and Computer Assisted Intervention—MICCAI 2018: 21st International Conference, Granada, Spain, 16–20 September 2018, Proceedings, Part I; Springer: Cham, Switzerland, 2018; pp. 457–466. [Google Scholar]
  71. Chen, Z.; Song, Y.; Chang, T.H.; Wan, X. Generating radiology reports via memory-driven transformer. arXiv 2020, arXiv:2010.16056. [Google Scholar]
  72. Yang, D.; Wei, J.; Xiao, D.; Wang, S.; Wu, T.; Li, G.; Li, M.; Wang, S.; Chen, J.; Jiang, Y.; et al. Pediatricsgpt: Large language models as chinese medical assistants for pediatric applications. Adv. Neural Inf. Process. Syst. 2024, 37, 138632–138662. [Google Scholar]
  73. Hinton, G.; Vinyals, O.; Dean, J. Distilling the knowledge in a neural network. arXiv 2015, arXiv:1503.02531. [Google Scholar] [CrossRef] [Scilit]
  74. Sanh, V.; Debut, L.; Chaumond, J.; Wolf, T. DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. arXiv 2019, arXiv:1910.01108. [Google Scholar]
  75. Marafioti, A.; Zohar, O.; Farré, M.; Noyan, M.; Bakouch, E.; Cuenca, P.; Zakka, C.; Allal, L.B.; Lozhkov, A.; Tazi, N.; et al. Smolvlm: Redefining small and efficient multimodal models. arXiv 2025, arXiv:2504.05299. [Google Scholar] [CrossRef] [Scilit]
  76. Zhou, B.; Hu, Y.; Weng, X.; Jia, J.; Luo, J.; Liu, X.; Wu, J.; Huang, L. Tinyllava: A framework of small-scale large multimodal models. arXiv 2024, arXiv:2402.14289. [Google Scholar]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Article Metrics

Citations

Article Access Statistics

Multiple requests from the same IP address are counted as one view.