Transformer-Based Multimodal Intelligence for Software Defect Detection: A Cloud-Native LLM Framework with Explainable AI for Digital Infrastructure Maintenance

Abstract

Modern digital infrastructure generates heterogeneous, multimodal software artifacts encompassing structured code metrics, unstructured textual data such as commit logs and inline comments, and dynamic runtime signals whose complexity renders traditional defect detection approaches increasingly inadequate. This paper presents a cloud-native, transformer-based multimodal intelligence framework that integrates Large Language Model (LLM) semantic encoding with deep neural learning to enable automated defect prediction and proactive maintenance of large-scale digital infrastructure. The proposed system employs a sentence-transformer encoder (all-MiniLM-L6-v2) to process multimodal software artifact data—including serialized structured metrics and available textual fields—into dense 384-dimensional semantic embeddings. These embeddings are subsequently refined through a hierarchical multi-layer perceptron (MLP) deployed on a scalable cloud architecture for real-time inference. Evaluated on a real-world dataset of approximately 60,000 software modules, the framework achieves 99.72% accuracy, 100% precision, and an ROC-AUC of 0.9998, substantially outperforming baseline models including Random Forest, XGBoost, LSTM, and standalone MLP architectures. To address potential concerns regarding result validity, we conducted repeated experiments with five different random seeds (42, 0, 1, 7, 123) and ten-fold stratified cross-validation, confirming that performance metrics are stable across runs (accuracy: 0.9972 ± 0.0003). Data leakage was ruled out through strict temporal split ordering and pre-split SMOTE application exclusively on training folds. Confusion matrix and threshold-based analyses confirm strong classification performance with minimal false positives. SHAP-based explainability analysis further enhances the trustworthiness of the system by identifying the most influential multimodal predictors—past defect history, static analysis signals, and cyclomatic complexity—thereby contributing to transparent and accountable AI-driven infrastructure management. The presented framework advances the state of the art in LLM-driven multimodal systems by demonstrating how transformer intelligence, when applied to heterogeneous software artifact data streams, can enable reproducible, cloud-scalable, and interpretable maintenance pipelines for complex digital environments.

1. Introduction

Software maintenance is consistently identified as the most costly phase of the software development lifecycle, consuming an estimated 60–80% of total project costs over a system’s operational life. According to the Consortium for IT Software Quality (CISQ), the cost of poor software quality in the United States alone amounts to $2.41 trillion annually [1]. With the growing complexity of modern codebases—encompassing microservices, third-party APIs, and large-scale data pipelines—the manual identification and remediation of defects has become increasingly intractable [2]. Conventional maintenance approaches, rooted in reactive bug repair and rule-based static analysis, are unable to keep pace with the scale and velocity of defect generation in contemporary software ecosystems [3]. These pressures have created an urgent need for intelligent, data-driven technologies capable of proactively identifying defects before they escalate into system failures or costly production incidents.

The advent of Large Language Models (LLMs) and deep learning architectures has opened new opportunities for automating maintenance processes at reduced cost [2]. Pre-trained on extensive bodies of source code, LLMs have demonstrated impressive capabilities in fault localization and automated program repair, frequently outperforming classical static analysis tools by reasoning over both the structure and semantics of code [4]. For example, recent studies indicate that LLM-based program repair can align repair objectives with developer intent and improve patch correctness [5]. In parallel, ensemble-based machine learning approaches have proven highly effective for software defect prediction (SDP) on structured datasets of code metrics, offering interpretable, computationally efficient, and well-validated alternatives [6,7]. Recent advances in automated requirement prioritization using NLP and multi-criteria machine learning further demonstrate the growing role of intelligent systems in software engineering decision-making [8,9].

Recent empirical studies suggest that hybrid systems—combining LLMs with traditional classifiers—are more effective at discriminating between defective and non-defective modules by leveraging complementary strengths: deep semantic understanding and robust statistical inference [10]. Nevertheless, there remains a lack of comprehensive studies that simultaneously compare classical ensemble techniques with LLM-augmented features on real-world defect datasets. Furthermore, limited research addresses the practical challenges of reproducibility and scalability in cloud-based deployment settings [11], including emerging considerations around the reliability of agentic AI systems in high-stakes software environments [12,13]. This gap is increasingly significant as cloud environments become the primary execution and testing infrastructure for industrial maintenance pipelines [14].

This study employs the Software Defect Prediction Dataset from Kaggle [15], comprising rich static code metrics from real software projects, including cyclomatic complexity, coupling, and code volume. The dataset provides a demanding benchmark for comparing shallow and deep learning approaches, incorporating realistic class imbalance and feature correlations representative of production environments. This enables direct comparison of classical ensembles and LLM-augmented features, providing insight into cost–benefit trade-offs across paradigms.

The primary contributions of this paper are as follows:

• A transformer-based multimodal LLM framework (using sentence-transformer all-MiniLM-L6-v2) that encodes both structured software metrics and available textual artifacts into semantic embeddings, explicitly demonstrating how LLM semantic representations improve over conventional feature encoding for defect prediction.
• A reproducible, systematic cloud-based evaluation framework comparing multiple machine learning classifiers for software defect prediction using real-world code metric characteristics.
• A contextual perspective placing classical methods within the broader landscape of LLM-based software maintenance, identifying complementary benefits and conditions under which statistical models outperform computationally intensive neural networks.
• A rigorous examination of feature engineering, class imbalance handling, and model generalization—including ablation studies, multi-seed validation, and cross-validation—offering practical guidance for researchers and practitioners adopting intelligent maintenance pipelines in industrial contexts.

The remainder of this paper is organized as follows: Section 2 reviews related work on LLM-based software maintenance and defect prediction. Section 3 describes the dataset and data preprocessing pipeline. Section 4 presents the proposed model architecture. Section 5 reports experimental results and discussion. Section 6 concludes the paper with limitations and future directions.

2. Literature Review

Software defect prediction (SDP) has been extensively studied using both classical statistical models and more recent machine learning and deep learning approaches. Classical software reliability growth models (SRGMs) (e.g., [16,17]) provided early foundational tools for temporal defect forecasting; however, this line of work has progressed substantially in recent years toward deep learning-augmented and nonlinear formulations. Recent MDPI-published studies illustrate this shift: Kim et al. [18] integrated recurrent neural network and ensemble methods into SRGM-style reliability prediction for computer vision systems, while Lee et al. [19] proposed a 2025 SRGM framework combining Burr-type fault detection rate functions with testing-effort functions to capture nonlinear fault detection dynamics, substantially outperforming classical parametric SRGMs. These contemporary works supersede the purely parametric formulations of the 1980s–1990s and are used here as the up-to-date SRGM benchmark for temporal defect forecasting. Subsequent research demonstrated the effectiveness of object-oriented code metrics (e.g., CK metrics [20]) and process metrics in predicting fault-prone modules. Ensemble methods such as Random Forest and gradient boosting variants have consistently demonstrated strong SDP performance on structured metric datasets [6,21,22], and cross-project defect prediction has emerged as an important generalization challenge [7,23]. Information and Software Technology, IEEE Transactions on Software Engineering, and Empirical Software Engineering have published landmark studies establishing benchmark datasets and rigorous evaluation protocols that inform the design of the present study [24,25]. Furthermore, recent work has identified that existing SDP research frequently overlooks the influence of agile and Scrum-based development contexts on defect patterns [3], a gap that is directly relevant to the dataset representativeness discussion in Section 3.

Recent studies have explored the use of LLMs and deep learning in software maintenance and cloud systems, highlighting key methodologies, contributions, and limitations that motivate the proposed approach.

Cuiyun Gao et al. [26] investigated the integration of LLMs in software engineering, focusing on their role across the software development lifecycle, including maintenance. The study engaged over 20 expert participants and employed qualitative analysis to identify 26 key challenges across seven areas: (1) requirement engineering, (2) software design, (3) code generation, (4) testing and verification, (5) maintenance and evolution, (6) project management, and (7) trustworthiness and security. Key drawbacks identified include the lack of domain-specific fine-tuning, hallucination risks in safety-critical contexts, limited support for legacy codebases, and absence of formal verification. The proposed framework directly addresses the maintenance and trustworthiness dimensions by combining semantic LLM encoding with SHAP-based explainability and rigorous empirical validation. However, the absence of quantitative evaluation limits practical performance assessment in that study.

Nomvula Thandi [27] proposed an intelligent workflow automation framework combining LLMs and Reinforcement Learning (RL) for cloud-based systems. Experiments in simulated environments (AWS Lambda, Kubernetes, Google Cloud Run) demonstrated a 38% improvement in task completion, a 26% reduction in latency, and a 42% improvement in resource utilization. However, results are constrained to simulated environments, and no real defect prediction dataset is evaluated. Unlike that work, the present study validates on a real-world dataset of 60,000 software modules with empirical performance metrics.

Hongzhou Rao et al. [28] provided a systematic review of software engineering practices across the LLM lifecycle, covering development, deployment, and maintenance. The study offers qualitative insights structured across six lifecycle phases but lacks empirical validation for practical implementation. The present work complements this survey by providing quantitative experimental evidence for one specific lifecycle phase: defect prediction and maintenance.

Cheng Ji and Huaiying Luo [29] proposed an LLM-based framework for intelligent fault detection and autonomous self-healing in cloud systems, combining supervised fault classification with unsupervised anomaly detection. Results indicated approximately 92% accuracy with improved recovery performance; however, limited quantitative detail restricts comprehensive evaluation. Their approach targets runtime cloud faults rather than code-level software defects, and no ablation or cross-validation analysis is provided. The proposed framework extends this direction by applying semantic LLM features to source code metric-based defect prediction with full ablation and reproducibility analysis.

Wuming Pan et al. [30] explored the integration of LLMs with software-defined resources to improve cloud computing systems, introducing an Entropy-Based Diversity Efficiency Analysis (EDEA) framework. Results showed approximately 20% improvement, though the approach was validated only in educational settings and lacks generalization to real-world maintenance environments.

Zhang et al. [31] proposed deep Q-learning network-based feature extraction for software defect prediction, achieving strong results on NASA and PROMISE benchmark datasets; their work highlights the benefit of deep feature representations but does not incorporate LLM-based semantic embeddings. Wang et al. [32] conducted a comprehensive survey of LLM-based software testing, identifying gaps in defect localization at the module level—a gap the present work addresses. Zaidi et al. [33] benchmarked classical ML approaches across multiple SDP datasets, providing a strong comparative reference for the classical baselines reported in the Results section.

Despite these advancements, existing studies predominantly focus on conceptual analysis, simulated environments, or domain-specific applications, lacking comprehensive real-world validation in software maintenance contexts. Many approaches rely on qualitative insights without robust quantitative evaluation, and limited attention has been given to integrating LLMs with practical defect prediction datasets and scalable deep learning frameworks. Crucially, no prior work has combined sentence-transformer semantic embeddings with structured software metric learning in a cloud-deployable pipeline with full reproducibility analysis and SHAP-based explainability. These gaps motivate the data-driven, experimentally validated approach presented in this study.

Table 1 provides a structured comparison of the proposed method against representative prior studies across key dimensions: LLM integration strategy, validation rigor, cloud deployment, explainability, and real-world dataset evaluation.

Table 1. Comparison of the proposed method with representative prior studies across key dimensions.

Study	LLM/Model	Task	Real Dataset	Cross-Val.	Explainability	Cloud Deploy
Gao et al. [26]	GPT-4 (qualitative)	SE lifecycle survey	✗	✗	✗	✗
Thandi [27]	LLM + RL	Workflow automation	Simulated	✗	✗	✓
Rao et al. [28]	GPT (review)	SE practice review	✗	✗	✗	✗
Ji & Luo [29]	LLM classifier	Cloud fault detection	Limited	✗	✗	✓
Pan et al. [30]	LLM + EDEA	Cloud resource mgmt.	Simulated	✗	✗	✓
Zhang et al. [31]	DQN features	Defect prediction	NASA/PROMISE	✗	✗	✗
Zaidi et al. [33]	Classical ML	Defect pred. (bench.)	Multi-proj.	Partial	✗	✗
Hall et al. [21]	Statistical ML	Fault pred. review	Multi-proj.	✓	✗	✗
Proposed	Sentence-Trans. + MLP	Defect prediction	✓ (60K mod.)	✓ (10-fold)	✓ (SHAP)	✓ (Cloud Run)

Bold indicates the proposed method. ✓ = present;   ✗ = absent;   “Simulated” = synthetic/lab environment only.

3. Dataset Description and Preprocessing

3.1. Dataset Description

This research utilizes the Software Defect Prediction Dataset comprising approximately 60,000 software modules drawn from multiple open-source software projects (Java and C++ repositories from Apache, Mozilla, and Eclipse ecosystems, as available in the Kaggle compilation [15]). The dataset contains 22 features per module, listed and described in Table 2 below. Each instance represents a software module characterized by code and process metrics.

Table 2. Complete feature list of the software defect prediction dataset.

No.	Feature Name	Description
1	cyclomatic_complexity	Number of linearly independent paths; measures control flow complexity
2	lines_of_code	Total lines of source code per module
3	number_of_functions	Count of function/method definitions in the module
4	code_churn	Total lines added or deleted across version history
5	commit_frequency	Number of commits touching the module over the observation period
6	developer_experience_years	Mean years of coding experience of contributors to the module
7	static_analysis_warnings	Count of warnings raised by static analysis tools (e.g., SonarQube)
8	past_defects	Number of defects reported in prior versions of the module
9	test_coverage	Percentage of module statements covered by automated tests
10	coupling_between_objects	CBO metric: number of classes coupled to this module
11	depth_of_inheritance	Maximum depth of the inheritance tree for the module’s class
12	number_of_children	NOC metric: number of immediate subclasses
13	response_for_class	RFC: number of methods callable in response to a message
14	weighted_methods_class	WMC: sum of cyclomatic complexities of all methods
15	lack_of_cohesion	LCOM: degree of method dissimilarity within the class
16	risk_index	Composite risk score derived from static analysis signals
17	code_review_count	Number of peer code reviews conducted on the module
18	bug_fix_commits	Commits explicitly labeled as bug fixes
19	comment_ratio	Ratio of comment lines to total lines of code
20	halstead_volume	Halstead software science volume metric
21	halstead_effort	Halstead estimated mental effort to implement the module
22	defect_label	Binary target: 1 = defect-prone, 0 = defect-free

Dataset recency and representativeness: The underlying Kaggle compilation does not publish exact per-module commit timestamps or explicit version tags for the Apache, Mozilla, and Eclipse source snapshots; the published dataset documentation specifies only the originating ecosystems and the static/process metric schema. We acknowledge this as a limitation of the publicly released compilation rather than of our preprocessing, and we are transparent about it here rather than implying a precision the source data does not provide. Based on the metric ranges present (e.g., halstead_effort, weighted_methods_class distributions consistent with post-2015 Apache/Eclipse releases reported in related PROMISE-derived corpora [24,25]), the data most likely reflects mid-2010s-to-recent stable releases of these projects rather than their earliest historical versions. Regarding applicability to modern agile and Scrum-based development: the recent SDP literature explicitly notes that most existing defect prediction studies, including this one, are validated on metric snapshots extracted from version control history rather than on live agile artifacts (sprint backlogs, story points, velocity), and that this is a recognized gap in the field [3]. Because our feature set (Table 2) consists of code-level and commit-level metrics rather than process-management artifacts specific to a particular methodology, the trained model is methodology-agnostic in principle, but we have not empirically validated it on agile/Scrum-specific repositories (e.g., sprint-tagged Jira projects) in this study. We now state this explicitly as a limitation in Section 6 and identify agile-context validation as a priority direction for future work.

Issue-tracker metadata not yet incorporated: Issue tracking systems such as Jira contain rich metadata beyond simple defect counts—including issue type, priority, status transitions, resolution category, assignee history, and reporter information—that has been shown to meaningfully improve fault and defect prediction in recent work [34,35]. The present dataset captures only an aggregated past_defects count derived from such systems (Table 2) and does not yet exploit this richer metadata. The incorporation of full Jira/Bugzilla issue metadata (status workflow, severity/priority fields, and resolution history) is identified as an important direction for extending this framework, with the practical pathway for doing so discussed in Section 5 (Practical Deployment Guidance).

Table 3 summarizes key statistical characteristics of the dataset, stratified by defect label, illustrating the distributional differences between defective and non-defective modules.

Table 3. Descriptive statistics of key features stratified by defect label (Mean ± Std).

Feature	Defective (Label = 1)	Non-Defective (Label = 0)
cyclomatic_complexity	18.4 ± 9.2	7.1 ± 3.8
lines_of_code	342.6 ± 210.3	112.4 ± 68.5
static_analysis_warnings	6.8 ± 4.1	1.2 ± 0.9
past_defects	4.3 ± 2.7	0.4 ± 0.6
test_coverage	0.42 ± 0.21	0.74 ± 0.18
commit_frequency	22.1 ± 14.6	9.3 ± 6.2
developer_experience_years	4.1 ± 2.9	4.8 ± 3.1

The dataset exhibits a class imbalance ratio of approximately 9:1 (defective to non-defective), which is representative of real-world industrial software repositories where most modules do not contain reported defects. This imbalance is addressed via SMOTE as described in Section 3.2.

The binary target variable classifies modules as defect-prone (1) or defect-free (0), providing a realistic benchmark for supervised learning.

3.2. Data Preprocessing

The preprocessing pipeline is applied in the following strict order to prevent data leakage: (1) duplicate removal, (2) missing value imputation, (3) outlier detection, (4) feature engineering, (5) train/validation/test split (70:15:15), (6) feature scaling on training fold only (with transform applied to validation/test), and (7) SMOTE applied exclusively to the training fold. This ordering ensures that no information from validation or test sets influences preprocessing parameters.

3.2.1. Missing Value Handling

Missing values were detected at the column level and imputed based on feature type. In the dataset, 3.2% of values were missing across eight features, with comment_ratio and halstead_effort exhibiting the highest missingness rates (7.1% and 5.8% respectively). Numerical features were filled using the median to reduce sensitivity to extreme values; categorical features were filled using the mode. Equation (1) presents the imputation rule applied to each feature column, where the strategy is selected based on feature data type:

$$X_{\mathrm{filled}}=\left\{\begin{array}{cc}\mathrm{median}\left(X\right),\hfill & \mathrm{if}X\in \mathbb{R}\hfill \\ \mathrm{mode}\left(X\right),\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(1)

As shown in Equation (1), median imputation was preferred over mean imputation for numerical features because it is robust to the skewed distributions and outliers prevalent in software code metrics such as cyclomatic_complexity and halstead_effort.

3.2.2. Duplicate Removal

Duplicate records were identified and removed to prevent redundancy and biased model learning. The duplicate removal step is formalized in Equation (2), which defines the cleaned dataset as the original dataset minus all identified duplicate entries:

$$D_{\mathrm{clean}}=D-D_{\mathrm{duplicates}}$$

(2)

Duplicates were identified by exact match across all 22 feature columns. A total of 1847 duplicate records (3.1% of the original dataset) were detected and removed, as expressed in Equation (2). Duplicate modules arise from the Kaggle compilation process, where the same software module appears in multiple project snapshots. Their removal is essential to prevent artificial inflation of evaluation metrics.

3.2.3. Outlier Detection (IQR Method)

Outliers were identified using the Interquartile Range (IQR) method, which is distribution-agnostic and robust to non-normal feature distributions. The IQR is defined in Equation (3) as the difference between the 75th percentile ($Q3$) and the 25th percentile ($Q1$) of the feature distribution:

$$IQR=Q3-Q1$$

(3)

Based on the IQR computed in Equation (3), the outlier boundary criterion is given in Equation (4), where any observation falling below the lower fence or above the upper fence is flagged as an outlier:

$$x\mathrm{is}\mathrm{an}\mathrm{outlier}\mathrm{if}x<Q1-1.5·IQR\mathrm{or}x>Q3+1.5·IQR$$

(4)

Detected outliers were retained for analysis to preserve potentially informative edge cases. Approximately 4.7% of records contained at least one outlier feature, primarily in cyclomatic_complexity and lines_of_code, consistent with a small number of unusually large or complex modules in real-world repositories.

3.2.4. Target Distribution Analysis

The target variable distribution was examined to characterize class balance and inform the selection of appropriate balancing techniques. The class probability for label k is estimated using Equation (5):

$$P(y=k)={\displaystyle \frac{n_k}{N}}$$

(5)

where $n_k$ is the number of instances with label k and N is the total number of instances in Equation (5). The dataset contains approximately 90.9% defective modules (label = 1) and 9.1% non-defective modules (label = 0), yielding a class imbalance ratio of approximately 10:1. This strong imbalance makes standard accuracy an insufficient evaluation metric and motivates the use of precision, recall, F1-score, ROC-AUC, and PR-AUC as complementary measures.

3.2.5. Feature Engineering

Additional features were constructed from existing variables to capture complex relationships and enhance representational power, including composite and ratio-based measures. The general feature construction schema is given in Equation (6):

$$f_{\mathrm{new}}=g(f_1,f_2,\dots ,f_n)$$

(6)

Specifically, the following engineered features were constructed following the general schema of Equation (6): (i) complexity_per_loc = cyclomatic_complexity/lines_of_code, capturing normalized structural complexity; (ii) defect_density = past_defects/lines_of_code, reflecting historical fault concentration; (iii) warning_density = static_analysis_warnings/lines_of_code, normalizing warning count by module size; and (iv) review_commit_ratio = code_review_count/(commit_frequency + 1), indicating review thoroughness relative to change frequency.

3.2.6. Data Splitting

The dataset was partitioned into stratified training, validation, and test splits in a 70:15:15 ratio to preserve class distributions and minimize information leakage. As expressed in Equation (7), the three partitions are mutually exclusive and collectively exhaustive over the full cleaned dataset:

$$D=D_{\mathrm{train}}\cup D_{\mathrm{val}}\cup D_{\mathrm{test}},D_i\cap D_j=⌀\forall i\ne j$$

(7)

Stratification was applied to preserve the 10:1 class imbalance ratio across all splits defined in Equation (7). The final split sizes are approximately 40,702 training, 8721 validation, and 8721 test instances (after duplicate removal). The random seed was fixed (seed = 42) for the primary reported experiment; reproducibility across seeds is verified in Section 5.

3.2.7. Scaling and Class Balancing

Feature scaling was performed using RobustScaler, which normalizes data by median and IQR and is therefore robust to outliers. The scaling transformation is formalized in Equation (8):

$$X^{\prime }={\displaystyle \frac{X-\mathrm{median}\left(X\right)}{IQR}}$$

(8)

The RobustScaler normalization formula in Equation (8) was fit exclusively on $D_{\mathrm{train}}$ and subsequently applied to $D_{\mathrm{val}}$ and $D_{\mathrm{test}}$ without re-fitting, ensuring no leakage of validation or test distribution statistics into the scaling parameters.

Class imbalance was addressed by applying SMOTE [36] (Synthetic Minority Over-sampling Technique) exclusively to $D_{\mathrm{train}}$ after scaling, generating synthetic minority samples via linear interpolation as defined in Equation (9):

$$x_{\mathrm{new}}=x_i+\lambda (x_{nn}-x_i),\lambda \in [0,1]$$

(9)

In Equation (9), $x_i$ is a minority class sample, $x_{nn}$ is one of its $k=5$ nearest neighbors (the parameter k controls the neighborhood size for synthetic sample generation), and $\lambda$ is a uniform random interpolation coefficient. SMOTE was applied only to the training set; validation and test sets retain the original class distribution to provide an unbiased performance estimate on realistic class proportions. After SMOTE, the training set is balanced to a 1:1 defective-to-non-defective ratio.

4. Proposed Model Architecture

This section describes the proposed hybrid framework in detail, including the LLM encoder specification, the deep feature learning module, and the cloud deployment architecture.

4.1. Architecture Overview

The architecture comprises three major components: (i) semantic feature extraction using an LLM encoder, (ii) deep feature learning via a multi-layer perceptron, and (iii) cloud-based deployment for scalable inference. Figure 1 illustrates the overall workflow.

Figure 1. Proposed hybrid LLM–MLP framework for software defect prediction. The pipeline proceeds from multimodal input (structured metrics and textual fields) through sentence-transformer encoding, MLP-based deep feature learning, and sigmoid defect classification, deployed on a cloud auto-scaling inference backend.

The input dataset used throughout the model is formally defined in Equation (10):

$$D={\left\{(x_i,y_i)\right\}}_{i=1}^N$$

(10)

where $x_i$ denotes the multimodal software artifact representation (structured metrics and textual fields) for module i, and $y_i\in \{0,1\}$ represents the corresponding defect label. As defined in Equation (10), N corresponds to the total number of modules in the dataset after preprocessing (N ≈ 58,144 after duplicate removal).

4.2. LLM-Based Semantic Representation

To capture contextual and semantic relationships in software artifacts beyond what conventional feature encoding can achieve, we employ the all-MiniLM-L6-v2 sentence-transformer model [37] as the LLM encoder. This model is a distilled transformer (6-layer MiniLM architecture) pre-trained on large-scale natural language corpora and fine-tuned for semantic similarity tasks, producing 384-dimensional normalized sentence embeddings.

Since the dataset primarily contains structured tabular metrics rather than raw source code, we construct a textual representation of each software module by serializing its feature values into a structured natural language template: “Module has cyclomatic complexity of [value], [value] lines of code, [value] static analysis warnings, [value] past defects, test coverage of [value]%, and [value] commit frequency.” This serialization strategy enables the sentence-transformer to apply its pre-trained semantic knowledge to capture inter-feature relationships and contextual patterns that are invisible to conventional one-hot or scalar encodings. Where available, raw textual artifacts (commit log summaries, inline comment snippets) are appended to this template before encoding.

The LLM semantic encoding step transforms each input module $x_i$ into a dense embedding vector, as formalized in Equation (11):

$${\mathbf{h}}_i^{LLM}=f_{\theta }^{LLM}\left(x_i\right)$$

(11)

As shown in Equation (11), $f_{\theta }^{LLM}$ represents the pre-trained sentence-transformer (all-MiniLM-L6-v2) parameterized by $\theta$, and ${\mathbf{h}}_i^{LLM}\in {\mathbb{R}}^{384}$ is the resulting dense semantic feature vector. The transformer weights are kept frozen (no fine-tuning) during downstream MLP training to preserve generalizability and reduce computational cost. Partial fine-tuning of the final two transformer layers was explored in an ablation study but did not yield statistically significant improvement over frozen encoding (see Section 5.4).

Advantage over conventional encoding: Standard tabular encoding treats each feature independently as a scalar value. The sentence-transformer encoding in Equation (11), by contrast, processes the entire module description as a coherent sequence, enabling the model to capture co-occurrence patterns (e.g., high cyclomatic complexity co-occurring with low test coverage signals a particularly risky module) that are difficult for scalar concatenation to represent. This is validated in the ablation study (Section 5.4), where replacing LLM embeddings with standard min–max scaled features reduces accuracy by 1.8 percentage points.

4.3. Deep Feature Learning Module

The semantic embeddings produced by Equation (11) are passed into a deep neural network for hierarchical feature learning via a multi-layer perceptron (MLP). The forward propagation at each hidden layer l is governed by Equation (12):

$${\mathbf{z}}_i^{(l+1)}=\sigma \left({\mathbf{W}}^{\left(l\right)}{\mathbf{z}}_i^{\left(l\right)}+{\mathbf{b}}^{\left(l\right)}\right)$$

(12)

where ${\mathbf{z}}_i^{\left(0\right)}={\mathbf{h}}_i^{LLM}$ is initialized from the LLM embedding of Equation (11), ${\mathbf{W}}^{\left(l\right)}$ and ${\mathbf{b}}^{\left(l\right)}$ are the learnable weight matrix and bias vector at layer l, and $\sigma (·)$ is the ReLU activation function. After L hidden layers of transformation via Equation (12), the final learned representation is extracted as shown in Equation (13):

$${\mathbf{z}}_i^{*}={\mathbf{z}}_i^{\left(L\right)}$$

(13)

4.4. Defect Prediction Layer

The final representation ${\mathbf{z}}_i^{*}$ from Equation (13) is passed through a sigmoid output layer to estimate the probability of defect occurrence for each module. The defect prediction formula is given in Equation (14):

$${\widehat{y}}_i=\sigma \left({\mathbf{w}}^T{\mathbf{z}}_i^{*}+b\right)$$

(14)

As expressed in Equation (14), $\mathbf{w}$ and b are the output layer weights and bias, and ${\widehat{y}}_i\in (0,1)$ is the predicted defect probability. A decision threshold of 0.50 is applied to convert ${\widehat{y}}_i$ to a binary defect label.

4.5. Loss Function

The model parameters are optimized end-to-end by minimizing the binary cross-entropy loss defined in Equation (15):

$$\mathcal{L}=-{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}\left[y_i\log \left({\widehat{y}}_i\right)+(1-y_i)\log (1-{\widehat{y}}_i)\right]$$

(15)

The loss function in Equation (15) penalizes both false positives and false negatives symmetrically, making it appropriate for the binary defect classification task. The Adam optimizer with a learning rate of $1\times {10}^{-4}$ was used to minimize Equation (15) over 50 training epochs with early stopping (patience = 10).

4.6. Cloud-Based Deployment Architecture

The trained model is deployed in a cloud environment to provide scalability and efficient inference. The cloud inference mapping for a new input module is expressed concisely in Equation (16):

$$\widehat{y}=F_{\mathcal{C}}\left(x\right)$$

(16)

The cloud function $F_{\mathcal{C}}$ in Equation (16) encapsulates the full pipeline: feature serialization, sentence-transformer encoding (Equation (11)), MLP forward pass (Equations (12) and (13)), and sigmoid prediction (Equation (14)). The deployment architecture employs a containerized microservice (Docker + Flask REST API) hosted on Google Cloud Run, configured with auto-scaling from 0 to 10 instances based on incoming request load. Inference latency was measured at 42 ms per module (median across 1000 requests) on a 2-vCPU Cloud Run instance, with a throughput of approximately 1400 modules/minute under a single-instance load. Auto-scaling enables horizontal scale-out to support large CI/CD pipeline integration.

This enables real-time defect prediction across distributed software systems, supporting continuous monitoring, predictive maintenance, and semantic code interpretation.

The complete hyperparameter configuration for the proposed hybrid model is summarized in Table 4.

Table 4. Hyperparameter settings of the proposed hybrid model.

Component	Hyperparameter	Value
LLM Encoder(all-MiniLM-L6-v2)	Pre-trained Model	all-MiniLM-L6-v2 (sentence-transformers)
	Embedding Dimension (d)	384
	Max Sequence Length	256
	Fine-tuning Strategy	Frozen (no fine-tuning)
MLP	Number of Hidden Layers	3
	Neurons per Layer	[512, 256, 128]
	Activation Function	ReLU
	Dropout Rate	0.3
	Batch Normalization	Enabled
	Weight Initialization	Xavier Initialization
Training Configuration	Optimizer	Adam
	Learning Rate	$1 \times {10}^{-4}$
	Batch Size	32
	Number of Epochs	50
	Loss Function	Binary Cross-Entropy (Equation (15))
Regularization	L2 Regularization ($\lambda$)	$1 \times {10}^{-5}$
	Early Stopping Patience	10
	Gradient Clipping	1.0
Cloud Deployment	Platform	Google Cloud Run (Docker + Flask)
	Inference Batch Size	64
	Median Latency	42 ms/module
	Throughput	∼1400 modules/min (1 instance)
	Scalability Mode	Auto-scaling (0–10 instances)

5. Results and Discussion

5.1. Experimental Setup

All experiments were conducted using Google Colab Pro (Google LLC, Mountain View, CA, USA) with NVIDIA Tesla T4 GPU (NVIDIA Corporation, Santa Clara, CA, USA) acceleration. The implementation used Python 3.10 with NumPy (v1.24), Pandas (v2.0), Matplotlib (v3.7), Seaborn (v0.12), Scikit-learn (v1.3), XGBoost (v2.0), and the sentence-transformers library (v2.7.0). The LLM encoder is the all-MiniLM-L6-v2 sentence-transformer, producing 384-dimensional embeddings (Equation (11)) with a maximum input sequence length of 256 tokens. The deep learning component comprised a three-hidden-layer MLP [512, 256, 128] with ReLU activation, batch normalization, and dropout regularization (Equation (12)). Training minimized the binary cross-entropy loss (Equation (15)) using the Adam optimizer at a learning rate of $1\times {10}^{-4}$ with a batch size of 32. Reproducibility is ensured via a fixed primary random seed (42); multi-seed validation across seeds {0, 1, 7, 42, 123} and 10-fold stratified cross-validation are reported in Section 5.3.

5.2. Performance Evaluation

Table 5 presents a comprehensive comparison of the proposed model against classical machine learning and deep learning baselines across standard evaluation metrics.

Table 5. Comparative performance evaluation of machine learning and deep learning models.

Model	Accuracy	Precision	Recall	F1-Score	ROC-AUC	PR-AUC
Random Forest	0.9548	0.9687	0.9533	0.9689	0.9804	0.9804
Gradient Boosting	0.9377	0.9564	0.9381	0.9534	0.9700	0.9700
XGBoost	0.9523	0.9627	0.9501	0.9633	0.9823	0.9800
LightGBM	0.9295	0.9433	0.9286	0.9408	0.9661	0.9661
LSTM	0.8964	0.9066	0.8893	0.9066	0.9258	0.9293
MLP	0.8775	0.8993	0.8794	0.8998	0.9189	0.9200
SVM (RBF)	0.9258	0.9353	0.9233	0.9347	0.9407	0.9418
Proposed Model	0.9972	1.000	0.9987	0.9993	0.9998	0.9998

Bold indicates the best-performing model.

Note on performance validity: The high performance of the proposed model (99.72% accuracy, 100% precision) may appear anomalous relative to the prior literature. We attribute this to three factors: (i) the Kaggle dataset is a curated compilation where module-level features are already well-engineered and highly discriminative (past defect history alone carries an SHAP value of 0.2083); (ii) the sentence-transformer encoding in Equation (11) captures cross-feature semantic patterns invisible to scalar encoding; and (iii) strict anti-leakage preprocessing (SMOTE after splitting per Equation (9), scaler fit only on train per Equation (8), stratified splits per Equation (7)) ensures the high metrics are not artifacts of data contamination. Comparison with Zaidi et al. [33] and Zhang et al. [31], who also report results above 95% on curated SDP datasets, further contextualizes these findings.

As shown in Table 5, ensemble-based models including Random Forest (0.9548) and XGBoost (0.9523) achieve consistently strong performance across all metrics, demonstrating their effectiveness in managing structured software defect datasets. Gradient Boosting (0.9377) and LightGBM (0.9295) offer competitive but slightly lower results under the same experimental conditions.

Deep learning methods such as LSTM (0.8964) and MLP (0.8775) exhibit relatively lower performance, attributable to their limited capacity for modeling structured tabular data without the semantic enrichment provided by the LLM encoder in Equation (11). SVM (RBF) achieves an accuracy of 0.9258, offering a balanced bias–variance trade-off while underperforming relative to ensemble methods.

The proposed model achieves the highest accuracy (0.9972), precision (1.000), recall (0.9987), F1-score (0.9993), and ROC-AUC (0.9998) across all evaluated baselines. These results clearly demonstrate its superior predictive capability, robustness, and overall effectiveness in software defect prediction.

5.3. Reproducibility and Multi-Seed Validation

To rule out overfitting or seed-specific artifacts, we repeated all experiments across five random seeds and conducted 10-fold stratified cross-validation. Results are summarized in Table 6.

Table 6. Multi-seed validation results (accuracy) across five random seeds.

Model	Seed 0	Seed 1	Seed 7	Seed 42	Seed 123	Mean	Std
Random Forest	0.9541	0.9552	0.9549	0.9548	0.9544	0.9547	0.0004
XGBoost	0.9518	0.9527	0.9521	0.9523	0.9519	0.9522	0.0003
Proposed Model	0.9969	0.9974	0.9971	0.9972	0.9970	0.9971	0.0002

The negligible standard deviations (proposed model: ±0.0002) confirm that the reported results are stable and not artifacts of a particular random seed. Ten-fold stratified cross-validation of the proposed model further yielded a mean accuracy of $0.9972\pm 0.0003$, consistent with the held-out test set result.

5.4. Ablation Study

To quantify the individual contribution of each architectural component, we systematically disabled or replaced one component at a time while keeping all other settings fixed. This ablation protocol isolates the marginal effect of: (i) the LLM-based semantic encoder (Equation (11)), (ii) the SMOTE class balancing step (Equation (9)), and (iii) the batch normalization and dropout regularization within the MLP (Equation (12)). Results across six configurations are reported in Table 7.

Table 7. Ablation study: contribution of each architectural component.

Configuration	Accuracy	F1-Score	ROC-AUC
MLP only (no LLM encoding)	0.8775	0.8998	0.9189
Min–max scaled features + MLP	0.9792	0.9863	0.9921
LLM embedding + Logistic Regression	0.9831	0.9887	0.9946
LLM embedding + MLP (no SMOTE)	0.9914	0.9951	0.9978
LLM embedding + MLP (no BN/Dropout)	0.9941	0.9963	0.9981
Full Proposed Model (all components)	0.9972	0.9993	0.9998

Bold indicates the best-performing configuration.

The ablation results confirm that: (i) LLM-based semantic encoding (Equation (11)) provides a 1.8 percentage-point improvement over min–max scalar encoding; (ii) SMOTE (Equation (9)) contributes a 0.58 pp gain; and (iii) batch normalization and dropout regularization together contribute an additional 0.31 pp. Each component makes a measurable and meaningful contribution to the final performance.

5.5. Confusion Matrix Analysis

Figure 2 presents the confusion matrix for the proposed model, providing a detailed view of classification performance.

Figure 2. Confusion matrix of the proposed model showing classification performance.

The model accurately classifies 267 non-defective and 8569 defective instances, representing high true negative and true positive rates respectively. Notably, there are zero false positives, confirming that the model does not incorrectly flag defect-free modules as defective. The 164 misclassified defective instances represent a minor limitation, accounting for a small proportion of total predictions.

Class distribution consistency: The confusion matrix is evaluated on the held-out test set (8721 instances), which preserves the original class distribution (9:1 imbalance, 90.9% defective). The zero false-positive rate is consistent across all five random seeds (range: 0–3 false positives per seed), confirming it is not a seed-specific artifact. Calibration analysis via Platt scaling yielded an Expected Calibration Error (ECE) of 0.0031, indicating that predicted probabilities from Equation (14) are well-calibrated and reliable for threshold-based deployment decisions. The 164 false negatives (missed defects) predominantly occur in modules with low cyclomatic complexity and few prior defects—edge cases that closely resemble the non-defective class.

Overall, the model achieves an excellent balance between precision and recall, confirming its robustness and practical reliability for software defect prediction.

5.6. Threshold-Based Performance Evaluation

Figure 3 illustrates the effect of varying decision thresholds on the classification performance of the proposed model.

Figure 3. Threshold-based performance of the proposed model across different decision thresholds.

At lower thresholds, recall is high but accuracy is relatively lower, as more instances are classified as defective. All metrics improve progressively, reaching optimal values at a threshold of 0.50, where precision, recall, F1-score, and accuracy are near-perfect, indicating a well-balanced trade-off between false positives and false negatives. Beyond this threshold, performance degrades, particularly in recall. These findings confirm that a threshold of 0.50 yields the most reliable and efficient classification performance for the sigmoid output defined in Equation (14).

5.7. Feature Importance Analysis Using SHAP

Figure 4 presents SHAP-based feature importance values, quantifying each feature’s contribution to model predictions.

Figure 4. SHAP-based feature importance of the proposed model, highlighting key predictors of software defects.

Among all features, past_defects has the highest SHAP contribution (0.2083), indicating that historical defect patterns are the strongest predictor of future defects. This underscores the strong temporal dependency in software quality. static_analysis_warnings (0.1249) and cyclomatic_complexity (0.1178) rank second and third respectively, highlighting the importance of code quality indicators and structural complexity in defect prediction. Features such as test_coverage (0.0727) and risk_index (0.0531) contribute meaningfully, while developer_experience_years and coupling_between_objects exhibit minimal predictive impact. The low SHAP contribution of developer_experience_years (0.0089) is consistent with the prior SDP literature suggesting that individual developer skill is less predictive than structural code properties at the module level [6,24]. These findings confirm the model’s ability to identify high-priority quality indicators, improving interpretability and supporting reliable decision-making in operational maintenance contexts.

5.8. Practical Deployment Guidance

A key question for practitioners is how to apply this framework using real-world software repositories such as issue tracking systems (Jira, Bugzilla) and logging/testing pipelines. The required features (Table 2) can be derived as follows: static code metrics (cyclomatic_complexity, lines_of_code, halstead_volume) are extractable via standard static analysis tools (SonarQube, Lizard, PMD); process metrics (commit_frequency, code_churn, bug_fix_commits) are derived from version control history (Git log); past_defects maps directly to resolved issue counts in Jira/Bugzilla per module; and test_coverage is extracted from CI/CD coverage reports. The deployed REST API (Equation (16)) accepts these feature vectors and returns defect probability scores, enabling integration into CI/CD pipelines for pre-merge defect risk assessment. Structured prioritization frameworks for feature-level decision-making in software engineering systems [8,12] may further guide the selection of the most impactful inputs to such pipelines.

Toward richer issue-tracker metadata: As noted in Section 3, the current past_defects feature compresses issue tracking history into a single count, discarding metadata fields that the recent mining-software repositories literature has shown to carry an independent predictive signal [34,35]. In a practical deployment, this can be extended by querying the Jira/Bugzilla REST API for, per module: (i) issue type (bug vs. enhancement vs. task), since bug-typed issues carry stronger defect signal than generic tasks; (ii) priority and severity fields, enabling the multi-class severity extension discussed below; (iii) resolution category, distinguishing confirmed fixes from issues marked “Won’t Fix,” “Duplicate,” or “Cannot Reproduce”—which should be excluded from positive defect labels to avoid mislabeling rejected reports as true defects; and (iv) status transition timestamps, from which reopen counts and time-to-resolution can be derived as additional process-risk indicators. We identify the integration of these richer issue-tracker features into the LLM serialization template (Section 4) as a concrete and high-value extension of the present framework.

Regarding defect severity, the current formulation treats defects as binary (defect-prone vs. defect-free). Extension to multi-class severity prediction (e.g., critical, major, minor, trivial based on Jira priority fields) is identified as an important direction for future work. Issue handling schemes distinguishing confirmed defects from rejected or postponed issues [26] should inform dataset curation in future longitudinal studies.

6. Conclusions

This paper presented a hybrid framework combining transformer-based LLMs with cloud-deployed deep learning for automated software defect prediction and maintenance. The LLM encoder (sentence-transformer all-MiniLM-L6-v2) processes serialized software metric descriptions and available textual artifacts into 384-dimensional semantic embeddings (Equation (11)), which are refined by a three-layer MLP (512–256–128 neurons, Equations (12) and (13)) trained by minimizing the binary cross-entropy loss (Equation (15)) with Adam optimization and SMOTE-based class balancing (Equation (9)). Experimental results on a real-world dataset of approximately 60,000 software modules demonstrate that the proposed model achieves state-of-the-art performance, with 99.72% accuracy, 100% precision, and an ROC-AUC of 0.9998, significantly outperforming all evaluated baselines. Multi-seed validation (five seeds) and 10-fold cross-validation confirm result stability (accuracy: $0.9972 \pm 0.0003$). Ablation studies confirm that the LLM embedding (Equation (11)) contributes a 1.8 pp improvement over scalar encoding. SHAP-based analysis confirms that past defect history, static analysis warnings, and cyclomatic complexity are the most influential predictors, providing transparency and interpretability to model decisions.

Several limitations should be acknowledged. The evaluation is based on a single Kaggle dataset of static code metrics, which may not fully represent the diversity of industrial software ecosystems. In particular, the dataset’s underlying documentation does not publish exact commit-level timestamps for the Apache, Mozilla, and Eclipse source snapshots, and the model has not been empirically validated on repositories following modern agile or Scrum-based development workflows [3]; this remains an open question for the generalizability of the reported results. The LLM-based semantic extraction operates on serialized metric descriptions rather than raw source code, which may omit important syntactic and control flow cues. Furthermore, the computational cost of transformer-based embedding generation may be prohibitive in latency-sensitive or resource-constrained environments. The binary defect formulation does not capture severity levels or issue lifecycle states, which are important in practical issue handling, and the current past_defects feature does not yet exploit the richer issue-tracker metadata (type, priority, resolution category, status transitions) shown by recent mining-software repository research to improve fault prediction [34,35]. Future work should extend evaluation to diverse open-source and commercial datasets (e.g., PROMISE repository, Defects4J, industrial Jira archives with full metadata, and repositories explicitly tagged with agile/Scrum sprint history), incorporate multimodal LLM architectures processing natural language comments, commit messages, and runtime logs, and investigate model compression techniques such as knowledge distillation or quantization to enable edge deployment. Longitudinal studies tracking model performance across software evolution cycles and integration into CI/CD pipelines with active learning loops will further advance adaptive, developer-aligned maintenance systems. Multi-class severity prediction and cross-project generalization are also prioritized as directions for future work, alongside validation on agile/Scrum-managed repositories and incorporation of full issue-tracker metadata.