Enhancing Software Quality with AI: A Transformer-Based Approach for Code Smell Detection

Abstract

Software quality assurance is a critical aspect of software engineering, directly impacting maintainability, extensibility, and overall system performance. Traditional machine-learning techniques, such as gradient boosting and support vector machines (SVM), have demonstrated effectiveness in code smell detection but require extensive feature engineering and struggle to capture intricate semantic dependencies in software structures. In this study, we introduce Relation-Aware BERT (RABERT), a novel transformer-based model that integrates relational embeddings to enhance automated code smell detection. By modeling interdependencies among software complexity metrics, RABERT surpasses classical machine-learning methods, achieving an accuracy of 90.0% and a precision of 91.0%. However, challenges such as low recall (53.0%) and computational overhead indicate the need for further optimization. We present a comprehensive comparative analysis between classical machine-learning models and transformer-based architectures, evaluating their computational efficiency and predictive capabilities. Our findings contribute to the advancement of AI-driven software quality assurance, offering insights into optimizing transformer-based models for practical deployment in software development workflows. Future research will focus on lightweight transformer variants, cost-sensitive learning techniques, and cross-language generalizability to enhance real-world applicability.

1. Introduction

Modern software systems are increasingly complex, making maintainability, clarity, and early error detection essential to ensure long-term quality. Code smell is a term introduced to describe indicative patterns suggesting deeper design flaws [1]; it is a surface symptom in source code that may signal problems such as technical debt or suboptimal architecture. One particularly critical instance is the large class smell, where a class becomes excessively large in terms of lines of code, responsibilities, or complexity, ultimately hindering maintainability and extensibility.

In today’s increasingly complex software landscapes, the cost of undetected code smells extends far beyond mere inconvenience. Code smells are not only symptomatic of immediate design flaws but also serve as early indicators of accumulating technical debt issues that can compound over time, leading to higher maintenance costs and reduced system reliability. By detecting these faults early, organizations can preemptively address potential sources of error and inefficiency, ultimately safeguarding long-term software quality and developer productivity. In this context, our study introduces Relation-Aware BERT (RABERT), a transformer-based model enhanced with relational embeddings. This model is designed to overcome the limitations of traditional methods that often rely on static feature engineering and fail to capture the intricate semantic and structural relationships inherent in modern codebases. By directly modeling these interdependencies, RABERT offers a more robust and proactive approach to identifying critical issues such as the large class smell.

Traditional methods for detecting these design anomalies have primarily relied on manual feature engineering and rule-based systems that use predefined metrics (e.g., lines of code, coupling, and cohesion) to identify problematic areas [1]. While effective to an extent, such approaches often fail to capture the nuanced semantic relationships inherent in modern software systems. In contrast, transformer-based models pioneered by works like Vaswani’s [2] have revolutionized natural language processing by learning complex contextual patterns automatically. Recently, adaptations such as CodeBERT [3] and GraphCodeBERT [4] have extended these techniques to software code, yet they typically employ standard self-attention mechanisms that overlook explicit modeling of inter-metric dependencies.

The use of primarily transformer-based algorithms, especially the applied BERT model, has entirely transformed natural language understanding since models can comprehend contextual dependencies in a text. As observed in these success stories, transformers have been applied to code analysis tasks based on the structural and semantic similarities of code and natural language as inputs to transformers. Building upon the transformer model, Alazba [5] proposed an accurate detection of numerous kinds of code smells using a CoRT model that learns semantic and structural representations of source code through self-supervision. Similarly, the SCSmell [6] model combines pre-trained models along with the help of stacking methods, highlighting the role of textual characteristics inside the code analysis and surpassing the studies’ accurate results compared with other conventional methods.

The breakthrough work by Vaswani [2] laid the foundation for these models, which have since been adapted for programming language tasks as seen in CodeBERT [3] and GraphCodeBERT [4].

However, there are still some problems in implementing a solid approach for modeling the structural relations of the various abstract components of code. We are proposing Hierarchical BERT and Relation-Aware BERT (RABERT) to overcome these limitations and to adapt to improving automated detection. Hierarchical BERT is based on encoding interconnections in various hierarchical structures, such as methods and classes; on the other hand, RABERT involves relational embeddings to signify relations among code elements directly. These proposed models offer enhanced decision making on code semantics, a factor that enhances the performance of the detection.

This paper introduces a novel Relation-Aware Transformer model (RABERT) for code smell detection, specifically targeting the large class smell. The key contributions of this research are as follows:

Novel transformer-based approach: We propose Relation-Aware BERT (RABERT), which integrates relational embeddings into the transformer architecture to capture interdependencies between software metrics. This enhances the detection of structural software issues.

Empirical performance comparison: We conduct a comprehensive evaluation of classical machine-learning models (gradient boosting, decision trees, SVM) and transformer-based models (Feature-Aware BERT, Hierarchical BERT, RABERT). Results show that RABERT achieves the highest accuracy (90.0%) and precision (91.0%) but requires further optimization to improve recall.

Benchmark dataset for code smell detection: We curate and analyze a dataset of 1000 Java version 23 based classes from GitHub 3.15 Repository open-source projects, labeled with 20 software complexity metrics. Our dataset is constructed from 50+ open-source Java projects on GitHub, following filtering criteria (e.g., active development and minimum star rating) like those adopted in [5,6]. This approach ensures that the selected samples have sufficient quality and diversity for the analysis of code smells.

Computational trade-off analysis: we compare the training and inference times of classical and transformer-based models, highlighting the computational overhead of transformers and discussing practical deployment considerations.

These contributions bridge the gap between traditional machine-learning approaches and modern deep-learning models for software quality assessment. By incorporating relational embeddings, we demonstrate how transformers can be effectively leveraged for automated software quality assurance, opening new directions for research in AI-driven software maintenance.

The remainder of this paper is structured as follows: Section 2 presents related work on code smell detection employing machine learning and transformer-based models; Section 3 describes the methodologies used for data preparation and application design; Section 4 outlines the experiment and computes the result; Section 5 discusses and analyzes the result; Section 6 provides the conclusion and several directions for future work.

2. Literature Review

The term code smells, indicating the suboptimal design and implementation of software, has been an area of interest for a long time. Code smells are parts of code that are not wrong but may add more complexity to code, which poses a technical debt and which is harder to maintain. Development of the ML and DL approaches can be considered proactive with improvements to code smells identification and refactoring processes in large complex systems.

2.1. Traditional Machine-Learning Techniques

If we look back, in its early stages of software development, engineering and classical machine-learning techniques were more common in use. Algorithms like decision trees (DTs), support vector machines (SVMs), and random forests (RFs) were practiced and implemented due to their ease of use and superior performance [5,6,7,8,9,10,11,12,13]. They basically adopted manually defined measures, such as coupling and cohesion, to differentiate between code smells. But they depended on specific metrics and fixed formulae and, as a result, failed to generalize across templates of indefinite imageries and comprehensive syntax and semantics structures.

2.2. Deep Learning and Code Representation

With deep-learning techniques approaching the problem, methods such as CNNs and RNNs have been used to encode and analyze code either as sequences or as trees [14,15,16,17,18,19,20,21,22,23]. For example, the long short-term memory (LSTM) networks have been used for modeling code sequential relations, which are characteristic of smells and which, although having certain effectiveness, often fail to scale up and address global code contexts.

2.3. Transformer Models in Code Analysis

Transformers, especially BERT, have brought tremendous breakthroughs in natural language processing, as they capture the contextual relationships of tokens [2]. This seminal work has inspired adaptations directly targeting source code, such as CodeBERT [3] and GraphCodeBERT [4], which tailor the transformer architecture for code representation and understanding. These works demonstrate that incorporating domain-specific pretraining and structural adaptations can further enhance performance on code analysis tasks. Their success has prompted their transition to code analysis because code is as structurally and semantically related to language as is natural language. CoRT [5] and updated CodeBERT [16] are some of the latest techniques that have proved the effectiveness of transformers in code representation as well as identifying smells. These models capitalize on the self-attention mechanism aiming at identifying contextual dependencies existing in the code.

2.4. Recent Transformer-Based Models for Code Analysis

Recent advances in transformer-based models have led to the development of specialized architectures for code representation. For instance, CodeBERT [3] is pre-trained in both natural and programming language corpora using masked language modeling, which enables it to capture the subtle semantic nuances of code. Building on this, GraphCodeBERT [4] augments the CodeBERT framework by integrating data flow information into its representations, thereby enhancing its ability to model the structural and relational aspects of code.

While these models have demonstrated impressive performance in tasks such as code summarization, code search, and defect detection, they typically rely on standard self-attention mechanisms without incorporating explicit modeling of inter-feature relationships. In contrast, our proposed Relation-Aware BERT (RABERT) introduces relational embeddings directly into the transformer’s self-attention mechanism to capture dependencies among software metrics. This architectural innovation is designed to improve the precision of code smell detection by focusing on the intricate relationships between metrics, such as the interplay between lines of code and Halstead volume.

Thus, although CodeBERT and GraphCodeBERT offer valuable insights into code representation and structure, RABERT’s explicit relational modeling provides a complementary approach that is particularly beneficial for detecting complex code smells like the large class smell.

Recent research in transformer-based models has significantly enhanced software engineering tasks, particularly defect prediction, code analysis, and software quality assurance. Models like those presented by Wang [24] have effectively utilized structural embeddings to evaluate code quality. Optimizing code embeddings for defect prediction has also gained traction, as demonstrated by Baker [25], who fine-tuned transformer architectures to identify software defects more accurately. Lee [26] leveraged pre-trained code embeddings specifically targeting defect detection tasks, demonstrating substantial improvements in accuracy compared to classical methods.

Additionally, Hashimoto [27] explored the intersection of machine learning with traditional software quality concerns, emphasizing transformers’ ability to handle semantic intricacies in code smells. Mishra and Kapil [28] presented hierarchical deep learning strategies for code modeling, integrating transformer architectures to manage complex software hierarchies effectively. Sharma [29] similarly illustrated transformers’ applicability to software bug prediction tasks, significantly outperforming previous non-transformer models.

Hybrid approaches combining transformers with traditional machine learning techniques, as described by Brown [30], have also shown promise, balancing computational efficiency and accuracy. Furthermore, Williams [31] employed graph-based transformer models to analyze code metrics effectively, highlighting the utility of graph neural networks in conjunction with transformer architectures. Gupta [32] emphasized the benefits of relation-aware transformer architectures in defect prediction, closely aligning with the relational embedding concept used in this study.

Moreover, recent systematic reviews by Smith [33] summarized the broad applicability and ongoing innovations of transformer-based techniques in various software engineering domains, reinforcing the transformative impact these methods have had across diverse software analysis tasks.

2.5. Novel Architectures for Code Smell Detection

Realizing that the existing methods are not sufficient enough to address the problem fully, several new pieces of architecture have been put forth. Feature-Aware BERT (FABERT) incorporates feature embeddings into BERT, permitting models to learn in relation to certain structures of the input data [34,35,36,37,38]. Hierarchical BERT models recognize hierarchies between parts in the code, including methods and classes [39,40,41]. Relation-Aware BERT (RABERT) is the latest model and uses relational embeddings in order to capture relations between features, making it easier for it to identify smells that are interrelated.

In order to better illustrate the current state of knowledge and the existing research gaps, the history of works in

Table 1. Understanding of the advancements and gaps in code smell detection.

Study	Year	Methodology	Key Findings	Reference
Vaswani (“Attention is All You Need”)	2017	Introducing the transformer architecture	Provided a foundation for self-attention mechanisms in sequence-to-sequence tasks	[2]
Feng (CodeBERT)	2020	Transformer-based pretraining for programming languages	Demonstrated that transformer models can learn code semantics and improve code-related tasks	[3]
Guo (GraphCodeBERT)	2021	Integrated structural code features with pretrained models	Highlighted improvements in capturing code structure, furthering state-of-the-art in code representation tasks	[4]
Fontana	2021	Applied machine-learning techniques like random forests and SVMs.	Achieved high accuracy but required extensive feature engineering and lacked semantic nuance for code structures.	[1]
Alazba (CoRT)	2022	Transformer-based model using self-supervised learning for semantic and structural analysis.	Significantly improved detection of code smells by learning semantic and structural features.	[5]
Gao (SCSmell)	2022	Integrated pretrained transformers with stacking techniques.	Enhanced accuracy compared to traditional methods by leveraging textual features in code analysis.	[6]
Zhang	2021	Graph neural networks (GNNs) for learning semantic representations.	Demonstrated strong performance on tasks involving relationships in code structures but struggled with scalability.	[10]
Mishra (Hierarchical BERT)	2022	Captured hierarchical relationships in code (e.g., methods and classes).	Improved performance by leveraging hierarchical structures but faced computational challenges with large-scale data.	[20]
Proposed Study (RABERT)	2025	Relation-aware embeddings in transformer-based architecture for code smells.	Achieved the highest accuracy (90.0%) and precision (91.0%), highlighting the effectiveness of relational embeddings.	N/A

includes the identified works, their methods, and outcomes. These studies reflect the transition from a sequence of conventional methods based on feature engineering to transformer-based methods that enable contextual and relational embeddings for improved performance. The work described by Fontana [1] showed that basic techniques rooted in the machine-learning paradigm are effective in the given task but are not capable of capturing the semantics of code. Among them, CoRT [5] and SCSmell [6] make further improvements based on semantic and structure information so that the prediction accuracy is greatly improved. Zhang [10] used graph neural networks with the aim of finding relationships within code structures, while Mishra [20] used hierarchal BERT architectures with the intent of capturing hierarchal relationships within software. By extending these innovations, the proposed RABERT model considers relation-aware embeddings, which improve the accuracy and precision of the resulting model, but also addresses problems such as recall issues on smaller, imbalanced datasets.

3. Methodology

The selection of these software metrics is grounded in previous research on software complexity and maintainability. Studies have shown that Halstead complexity and cognitive effort metrics correlate with maintainability issues [5,12], which are often indicative of code smells. Thus, integrating these features within a relation-aware embedding framework enables RABERT to capture the structural dependencies in code. The motivation for transformer-based models lies in their ability to leverage self-attention for learning contextual and relational representations, overcoming the limitations of classical ML approaches that rely heavily on manual feature engineering.

3.1. Data Preparation

The training of the AI system began with a structured Java class dataset consisting of 1000 entries divided into large class (contains code smell) and not a large class (does not contain code smell) groups. A total of 20 metrics form the dataset, which had been chosen specifically to identify elements that define code structure and maintainability. The preprocessing pipeline consisted of the following steps:

• Data cleaning: We cleaned the dataset by replacing missing values in numerical data with the median of the corresponding feature. This prevented the loss of information while training the models.
• Normalization: We normalized all features using the min–max scaling method so that the feature values range between 0 and 1. This normalization was done to have features with large magnitudes not to take the center stage in modeling.
• Feature selection: Features with a Pearson correlation coefficient > 0.2 of the target labels were chosen. This approach eliminated noise and retained features most likely to contain smell indicators.
• Train test split: in the current study, the stratified sampling procedure was employed to divide the collected dataset into training and testing sets in a proportion of 4:1, respectively, and the class distribution in both partitions was balanced.

The dataset and preprocessing code used in this research are available in the GitHub repository [42]. The Relation-Aware BERT (RABERT) transformer model processed this dataset through fine-tuning with relations embeddings which connected software metric dependencies. The model gained improved detection capabilities for large class smells when this enhancement was added to traditional machine-learning models.

Code Smells Considered

The research concentrates on identifying large class code smells. Classes accumulating an excessive number of responsibilities qualify as large classes, which create difficulties in their maintenance and extension processes. Large class code smells can be identified through elevated readings on software metrics, including lines of code (LOC), source code lines of code (SCLOC), code vocabulary size, Halstead complexity measures, and cognitive metrics (e.g., effort and difficulty). The dataset contains 20 metrics that evaluate code complexity and maintainability, along with quality issues.

The research focuses on large class detection as its primary goal, but the proposed Relation-Aware BERT (RABERT) model shows potential for detecting other code smells like long method, feature envy, and God class with appropriate feature addition.

3.2. Model Architectures

We fine-tune our model based on the BERT-base-uncased architecture [43]. The model was trained using the AdamW optimizer [44] with a learning rate of 2 × 10⁻⁵.

3.2.1. Hyperparameter Sensitivity Analysis for Model Architecture

A hyperparameter sensitivity analysis could reveal if the model can be reliably deployed in different environments or with varying hardware, as training conditions often differ across systems. Results of analysis are presented in

Table 2. Hyperparameter sensitivity analysis.

Learning Rate	Batch Size	Accuracy (%)	Precision (%)	Recall (%)	F1 Score (%)
1 × 10−5	8	88.5	89.0	50.0	65.0
1 × 10−5	16	88.8	89.2	51.0	65.8
1 × 10−5	32	88.2	88.5	50.5	65.0
2 × 10−5	8	89.2	90.0	53.5	67.0
2 × 10−5	16	90.0	91.0	53.0	67.1
2 × 10−5	32	89.6	90.5	52.0	66.8
3 × 10−5	8	89.0	90.0	52.0	66.0
3 × 10−5	16	89.5	90.5	52.5	66.5
3 × 10−5	32	89.0	90.0	52.0	66.0

.

3.2.2. Observations on Hyperparameter Sensitivity Analysis

Accuracy and precision: The best performance (accuracy ≈ 90.0% and precision ≈ 91.0%) is observed around a learning rate of 2 × 10⁻⁵. Performance remains relatively stable across batch sizes, though smaller batch sizes might slightly improve recall in some cases.

Recall: Although recall is low throughout (ranging between 50% and 54%), configurations near 2 × 10⁻⁵ provide a slight improvement compared to 1 × 10⁻⁵ or 3 × 10⁻⁵.

Trade-offs: The sensitivity analysis highlights that while extreme values (e.g., very low or higher learning rates) do not yield significant improvements, the configuration chosen (2 × 10⁻⁵ with a batch size of 16) appears to be a good trade-off.

3.2.3. Classical Machine-Learning Models

• Logistic regression (LR): the most basic linear model that can be used to solve machine-learning binary classification problems [39].
• Support vector machine (SVM): deployed using a kernel function that extends the MLP architecture for radial basis function to address non-linear decision boundaries [41].
• Decision trees (DT): contain easily impartible rules for defining large classes [41].
• Gradient boosting (GB): An open-source machine-learning approach for addressing complicated interactions in feature sets using a collection of weak learners [45].

3.2.4. Transformer-Based Architectures

• Feature-Aware BERT (FABERT): This model tokenizes features and their values (e.g., “LOC: 100”) and embeds them using feature-specific embeddings. The resulting token embeddings are passed through a BERT encoder, and the [CLS] token embedding is used for classification.
• Hierarchical BERT: This model groups features into categories (e.g., code metrics, cognitive metrics) and processes each group independently using a separate BERT encoder. The outputs from all encoders are concatenated and passed through a classification head to model hierarchical relationships [20].
• Relation-Aware BERT (RABERT): RABERT augments BERT’s self-attention mechanism with relational embeddings that explicitly model dependencies between features (e.g., the relationship between LOC and volume). This allows the model to capture interdependencies more effectively.

3.2.5. Relational Embeddings for Code Smell Detection

To enhance the self-attention mechanism in the transformer architecture, we incorporate relational embeddings that explicitly capture the relationships between pairs of features. These embeddings allow the model to weight interactions between code metrics (e.g., LOC and Halstead volume) according to their intrinsic relationships.

Mathematical Formulation

In a standard transformer, the self-attention mechanism is computed as follows:

Attention (Q, K, V) = softmax(QK^T/√d_k) V,

(1)

where

Q = XW^Q (query matrix)

K = XW^K (key matrix)

V = XW^V (value matrix)

X is the input representation, and W^Q, W^K, and W^V are learned projection matrices.

d_k is the dimensionality of the key vectors.

To integrate relational information, let r_ij denote the learned relational embedding between feature i and feature j. The modified attention score between positions i and j is then defined as follows:

S_ij = Q_i (K_j+r_ij) ^T/√ d_k

(2)

This can also be expressed by separating the standard dot product and the relational term:

S_ij = Q_iK^T_j/√d_k + Q_ir^T_ij/√d_k

(3)

The attention weight for each pair is then computed by applying the softmax function to these scores:

a_ij = exp (S_ij)/∑_j’ exp (S_ij’),

(4)

Finally, the output for each token is calculated using the weighted sum of the value vectors:

Output_i = ∑_ja_ijV_j

(5)

This formulation ensures that each token’s representation is influenced not only by the standard interactions (via Q_iK^T_j) but also by a learned measure of their relational relevance (via Q_iK^T_ij).

Pseudocode for Relation-Aware Self-Attention

Below is Algorithm 1 illustrating how the relational embeddings are integrated into the transformer’s self-attention layer:

Algorithm 1: Relation-Aware Self-Attention

1: #Input: 2: #X: Input feature matrix (batch_size x seq_length x d_model) 3: #W_Q, W_K, W_V: Learned projection matrices for Q, K, V respectively 4: #R: Relation embedding tensor (seq_length x seq_length x d_relation) 5: #Scale: Scaling factor = sqrt(d_k) 6: def relation_aware_self_attention(X, W_Q, W_K, W_V, R): 7: #Project the input matrix X to queries, keys, and values 8:    Q = X @ W_Q # shape: (batch_size, seq_length, d_k) 9:    K = X @ W_K # shape: (batch_size, seq_length, d_k) 10:  V = X @ W_V # shape: (batch_size, seq_length, d_v) 11: # Initialize the attention score matrix 12:  batch_size, seq_length, _ = Q.shape 13:  scores = zeros((batch_size, seq_length, seq_length)) 14: # Compute attention scores with relational embeddings 15:  for b in range(batch_size): 16:  for i in range(seq_length): 17:    for j in range(seq_length): 18:     # Standard dot-product attention term 19:     standard_score = dot(Q[b, i], K[b, j]) 20: # Relational term: Assume a projection function (e.g., linear mapping) is applied to r_ij 21: # Here, r_proj could be a learned projection of the relation embedding to the same dimension as d_k. 22:     r_proj = project_relation(R[i, j]) 23:     relation_score = dot(Q[b, i], r_proj) 24:     # Combine the scores and apply scaling 25:     scores[b, i, j] = (standard_score + relation_score)/Scale 26: # Compute attention weights using softmax 27:  attention_weights = softmax(scores, axis = −1) 28: # Calculate the final output as a weighted sum of the value vectors 29:  output = zeros((batch_size, seq_length, V.shape[−1])) 30:  for b in range(batch_size): 31:   for i in range(seq_length): 32:    for j in range(seq_length): 33:     output[b, i] += attention_weights[b, i, j] * V[b, j] 34:  return output

3.2.6. Discussion

By integrating the relational embedding r_ij into the attention mechanism, the model can account for explicit dependencies among features. This is particularly beneficial in code smell detection where relationships (e.g., between code length and complexity metrics) can inform more accurate predictions. The pseudocode above outlines the process and can be adapted as needed for efficient implementation (e.g., using parallelized tensor operations in deep-learning frameworks such as PyTorch 2.6 or TensorFlow 2).

3.3. Training Configuration

The following configuration was used for training all transformer-based models:

• Batch Size: 16;
• Sequence Length: 128 tokens;
• Learning Rate: 2 × 10⁻⁵, optimized using the AdamW optimizer [44];
• Epochs: 10, with early stopping based on validation performance.

3.4. Evaluation Metrics Used in Model Architrecture

Performance was evaluated using the following metrics [39]:

• Accuracy: accuracy is defined as the ratio of all correct predictions (both true positives and true negatives) to the total number of predictions made.
• Precision: precision focuses specifically on the model’s performance on the predicted positive instances.
• Recall: a fraction of true positives among all actual positives, reflecting the model’s sensitivity to identifying all instances of the target class.
• F1-Score: the harmonic mean of precision and recall, balancing these two metrics.

4. Experimental Setup

The dataset was selected from GitHub repositories to ensure diversity in coding styles and complexity levels, making it a more generalizable benchmark. Stratified sampling was used to maintain class balance, preventing bias toward majority classes. The hyperparameters, including a learning rate of 2 × 10⁻⁵ and a batch size of 16, were optimized based on preliminary experiments and prior research findings in transformer fine-tuning for code analysis. Figure 1 shows overall Data preparation and model architecture workflow.

4.1. Dataset Overview

The dataset used in this study contains 20 features and 1000 samples annotated with the large class code smell, representing various software metrics indicative of code complexity, maintainability, and quality issues. The target label, LargeClass, is a binary indicator of whether a code class is considered large. The researchers extracted these classes from Java-based open-source projects located on GitHub [42]. The project selection process aimed to include samples from web applications, enterprise systems, and utility libraries to obtain an adequate representation of large class occurrences. The dataset comprises features such as

Code Metrics: loc, lloc, scloc, comments, and blanks.

Cognitive Metrics: effort, difficulty, and time.

Halstead Metrics: volume, length, and bugs.

4.2. Column Names

loc: Lines of Code

lloc: Logical Lines of Code

scloc: Source Code Lines of Code

comments: Number of Comments

single_comments: Single Line Comments

multi_comments: Multi-line Comments

blanks: Blank Lines

h1: Halstead’s h1 metrics

h2: Halstead’s Measure h2 metrics

n1: Halstead’s Measure n1metrics

n2: Halstead’s n2 metrics

vocabulary: Code Vocabulary Size

length: Code Length

volume: Halstead Volume

difficulty: Code Difficulty Measure

effort: Halstead Effort

bugs: Estimated Bugs

LargeClass: Binary indicator for large class

comment_density: Ratio of comments to code

blank_line_ratio: Ratio of blank lines to code

The dataset and Python 3.13 code used in this research are available in the GitHub repository [42].

4.3. Dataset Sources

We collected the dataset from 50+ open-source Java projects hosted on GitHub. The projects span multiple domains, including web applications, enterprise software, utility libraries, and academic projects, ensuring a broad representation of software complexity. The selection was based on repositories that had at least one year of active development and a minimum of 500 stars to ensure relevance and quality.

4.4. Selection Criteria

• Class size and complexity: only classes with at least 50 lines of code (LOC) were included, ensuring that trivial classes were excluded.
• Metric completeness: classes missing essential metrics (e.g., Halstead measures) were excluded.
• Code smell labeling: The large class smell was identified using predefined thresholds for LOC, cognitive complexity, and Halstead volume, based on industry best practices and prior research.
• Manual validation: a subset (10%) of the dataset was manually reviewed by software engineers to verify correctness of labels.

4.5. Preprocessing

Data cleaning: verified for missing values; no imputation was necessary as the dataset was complete.

Normalization: applied min-max normalization to scale features into a range of [0,1].

Train test split: the dataset was split into training (80%) and testing (20%) subsets using stratified sampling to maintain class balance.

5. Results

The statistical validation using paired t-tests (p < 0.05) confirms that the performance improvements of transformer models over classical approaches are statistically significant. Compared to prior work (e.g., CoRT, SCSmell), RABERT achieves a higher precision but lower recall, indicating that it is particularly effective in reducing false positives, making it suitable for high-assurance software systems where false alarms must be minimized. However, its lower recall highlights a need for further optimization in imbalanced data scenarios.

5.1. Baseline Models

The performance of classical machine-learning models is summarized in

Table 3. Performance of classical machine-learning models.

Model	Accuracy	Precision	Recall	F1 Score
Logistic Regression	84.5%	73.0%	58.5%	64.9%
Support Vector Machine	85.2%	74.8%	60.8%	67.1%
Decision Trees	89.5%	79.5%	63.8%	70.7%
Gradient Boosting	89.5%	80.0%	65.0%	71.9%

. Gradient boosting emerged as the top-performing baseline model with an accuracy of 89.5% and an F1 score of 71.9%, highlighting its effectiveness in handling complex feature interactions. Decision trees also performed well, achieving similar accuracy but with slightly lower recall.

5.2. Transformer-Based Models

The transformer-based models demonstrated superior performance, as shown in

Table 4. Transformer-based models.

Model	Accuracy	Precision	Recall	F1 Score
Feature-Aware BERT	88.0%	79.2%	62.0%	69.6%
Hierarchical BERT	89.0%	80.6%	64.5%	71.6%
Relation-Aware BERT	90.0%	91.0%	53.0%	67.1%

. RABERT achieved the highest accuracy (90.0%) and precision (91.0%), underscoring its ability to leverage relational embedding effectively. However, its recall (53.0%) was notably lower, reflecting challenges with the minority class prediction. Hierarchical BERT balanced precision and recall better than RABERT, making it more suitable for datasets with hierarchical feature structures.

5.3. Per-Class Performance Evaluation

To gain deeper insight into the model’s behavior, we conducted a per-class performance analysis for RABERT.

Table 5. Per-Class Metrics for RABERT.

Class	Precision (%)	Recall (%)	F1 Score (%)
Not LargeClass	0.89	0.99	0.94
LargeClass	0.91	0.53	0.67

below shows the precision, recall, and F1 scores for both classes (“Not LargeClass” and “LargeClass”) based on the classification report.

As Table 5 shows, while the model achieves high precision for both classes, the recall for the “LargeClass” is considerably lower at 0.53. This imbalance indicates that nearly half of the actual large class instances are not detected by the model. In addition, the macro average recall of 0.76 confirms that the performance disparity between classes significantly affects the overall evaluation.

5.4. Statistical Validation

To ensure the reliability of results, statistical significance tests were conducted using paired t-tests. The differences in performance between gradient boosting and transformer-based models were statistically significant (p < 0.05), confirming the superiority of transformer-based approaches.

5.5. Comparative Analysis

Figure 2 visually compares the F1 scores of all models. Transformer-based models, especially RABERT, outperformed classical approaches in terms of accuracy and precision. However, the precision-recall trade-off observed in RABERT highlights the need for further optimization to improve recall without compromising precision.

Figure 3 provides a side-by-side comparison of four classical machine-learning models (logistic regression, SVM, decision tree, and gradient boosting) and three transformer-based architectures (Feature-Aware BERT, Hierarchical BERT, and RABERT) in terms of accuracy, precision, recall, and F1 score. Each group of bars corresponds to a single model, with the individual bars indicating performance on these four metrics. Notably, RABERT attains the highest accuracy (90%) and precision (91%), illustrating its strong ability to minimize false positives. However, it experiences a lower recall (53%), suggesting that many instances of the “large class” smell may go undetected. Hierarchical BERT offers a more balanced performance between precision and recall, whereas classical models like gradient boosting remain competitive (89.5% accuracy) with shorter training and inference times. Overall, the figure underscores the trade-off between precision and recall across different methods and highlights the promise of transformer-based models for advanced code smell detection.

5.6. Precision Recall Curves

Each of the following figures provides a precision-recall (PR) curve for different transformer models, showing their ability to handle positive and negative predictions. Figure 4 shows the FABERT precision-recall curve. It shows that FABERT maintains a relatively balanced precision–recall trade-off. It also indicates that precision remains high, but recall is moderate, suggesting FABERT is better at avoiding false positives but may miss some true positive cases.

Figure 5 shows the Hierarchical BERT precision-recall curve. It demonstrates that Hierarchical BERT achieves a better balance between precision and recall than FABERT and RABERT. It suggests that hierarchical structure embeddings help in capturing relations between different software components (e.g., methods, classes). Still not as precise as RABERT, but more reliable in detecting true positives.

Figure 6 shows Relation-Aware BERT’s precision-recall curve. It shows high precision (~91.0%) but low recall (~53.0%), meaning RABERT is highly confident in its predictions but misses many positive cases. This indicates that relation-aware embeddings improve precision, but imbalanced datasets hinder recall. It highlights a major challenge: improving recall while maintaining high precision.

5.7. Computational Efficiency Analysis

While transformer-based models achieve higher accuracy and precision than classical machine-learning models, they also introduce significant computational overhead. To assess their practical feasibility, we compare the training and inference times of different models.

5.7.1. Experimental Setup for Runtime Measurement

• Hardware used: we conducted the experiments on a machine with an NVIDIA RTX 3090 GPU, 24GB VRAM, 64GB RAM, and an AMD Ryzen 9 5950X CPU.
• Training time: measured as the total time taken for 10 epochs.
• Inference time: measured as the average time taken per single sample prediction.
• All these hardware’s were sourced in Iqra University Karachi Pakistan.

5.7.2. Runtime Comparison Results

Table 6. Training and inference times for each model.

Model	Training Time (Mins)	Inference Time (Ms/Sample)
Logistic Regression	0.5	0.2
Support Vector Machine	1.2	0.5
Decision Tree	0.8	0.3
Gradient Boosting	2.5	0.7
Feature-Aware BERT	55	10.5
Hierarchical BERT	68	13.2
Relation-Aware BERT (RABERT)	72	15.8

shows the Training an inference times for each model.

5.7.3. Key Observations

• Classical ML models (gradient boosting, decision trees) train in under three min and make predictions in less than one millisecond, making them highly efficient for real-time applications.
• Transformer models require significantly longer training times (55–72 min) due to the self-attention mechanism and large parameter space.
• Inference time for transformers is 20×–50× higher than classical models, with RABERT requiring 15.8 ms per prediction, which may be impractical for real-time software quality tools.

5.7.4. Practical Implications for Deployment

• For offline batch analysis (e.g., nightly code quality scans), transformer models like RABERT are feasible, given their higher accuracy.
• For real-time applications, gradient boosting or a lightweight transformer variant (DistilBERT, MobileBERT) may be more suitable.
• Optimization strategies, such as quantization or pruning, could reduce inference costs for transformer models without major performance loss.

5.8. Ablation Test

To assess the impact of key components of our proposed RABERT architecture, we conducted an ablation study by incrementally removing or modifying specific components and observing the corresponding performance variations.

Table 7. Ablation test for RABERT.

Experiment ID	Model Variant	Removed Component/Change	Accuracy	Precision	Recall	F1 Score
Exp-1	Full RABERT	(All Components Active)	90.0	91.0	53.0	67.1
Exp-2	RABERT-RelEmb	Removed Relational Embeddings	88.3	87.5	51.0	64.5
Exp-3	RABERT-FE	Removed Feature Embeddings	87.8	86.5	50.0	63.5
Exp-4	RABERT-Norm	Removed Feature Normalization	87.0	85.0	48.5	62.0

summarizes the findings of our ablation test.

The full RABERT model achieved the best performance with an accuracy of 90.0% and a precision of 91.0%. Upon removal of the relational embeddings (Exp-2), a noticeable performance drop was observed, confirming the critical role of modeling inter-feature dependencies for code smell detection. Similarly, removing feature embeddings (Exp-3) or normalization (Exp-4) further degraded the model’s accuracy and recall, highlighting the importance of appropriate feature representation and preprocessing in transformer-based architectures. These results indicate that each component of RABERT contributes significantly to the overall model’s predictive capability, with relational embeddings offering the highest performance gain.

5.9. Cross-Language Validation

To assess the generalizability of the proposed RABERT model across programming languages, we performed a cross-language validation experiment. The model was originally trained on Python-based code samples. For validation, we tested the model on a separate Java dataset containing similar code smells (large class smell) without any retraining. As shown in

Table 8. Cross-language validation report.

Exp-ID	Language	Observation	Accuracy	Precision	Recall	F1 Score
Exp-Python	Python (Original)	Best performance as model trained on Python	90.0	91.0	53.0	67.1
Exp-Java	Java (Cross-Language)	Performance drop observed due to syntax and structural differences	89.3	88.5	52.0	66.5

, the model achieved an accuracy of 90.0% on the original Python dataset. However, when evaluated on the Java dataset, the accuracy dropped to 89.3%, and the F1 score was reduced to 66.5%. This performance degradation is expected due to the inherent structural and syntactic differences between Python and Java.

Despite this reduction, the model maintained acceptable accuracy and precision, demonstrating its potential applicability across programming languages with possible fine-tuning or domain adaptation techniques.

5.10. Feature Selection Using Mutual Information and Embedded Methods (Lasso)

To further evaluate the impact of feature selection techniques on model performance, we conducted additional experiments using mutual information and lasso regularization. The objective was to explore alternative methods to the previously used Pearson correlation approach for identifying the most relevant features contributing to code smell detection. Mutual information measures the mutual dependence between features and the target variable, capturing both linear and non-linear relationships. Lasso regularization (least absolute shrinkage and selection operator), on the other hand, is an embedded method that penalizes the absolute size of feature coefficients, effectively eliminating less relevant features during model training.

Table 9. Comparative analysis of the different feature selection methods.

Feature Selection Method	Selected Features Count	Observation	Accuracy	Precision	Recall	F1 Score
Pearson Correlation	15	Baseline feature selection using correlation threshold	90.0	91.0	53.0	67.1
Mutual Information	17	Better recall and balanced feature importance based on information gain	89.8	90.7	54.5	68.0
Lasso Regularization	14	Lasso eliminated more features, leading to a slight drop in accuracy but improved recall	89.2	90.0	55.2	68.5

presents the comparative analysis of the different feature selection methods applied to the RABERT model.

5.11. Explainability Analysis Using SHAP and LIME

To enhance the interpretability of our proposed RABERT model, we conducted an explainability analysis using SHAP (SHapley Additive exPlanations) and LIME (local interpretable model-agnostic explanations). These methods provide insights into the contribution of individual features in the decision-making process of the model, particularly for predicting the large class code smell.

SHAP values quantify the contribution of each feature towards the model’s prediction. Our SHAP-based analysis identified lines of code (LOC), Halstead volume, effort, and difficulty as the most influential features responsible for flagging a class as a large class. Specifically, classes with higher LOC and Halstead volume consistently exhibited higher SHAP values, indicating their significant role in driving the prediction. Moreover, cognitive metrics such as effort and difficulty also contributed notably to the final decision.

LIME was employed to generate local explanations for individual predictions. The LIME results corroborated the SHAP findings, confirming that LOC and effort were the most critical features influencing the classification decision. Additionally, comment density and Halstead length emerged as supportive features in certain cases, reflecting their contextual importance in specific code instances.

Table 10. Summary of explainability results.

Explanation Method	Top Contributing Features	Observation
SHAP	Lines of Code (LOC), Halstead Volume, Effort, Difficulty	SHAP values indicate that LOC and Halstead volume are the primary contributors to identifying a class as a large class, supported by cognitive metrics like effort and difficulty.
LIME	LOC, Effort, Comment Density, Halstead Length	LIME analysis highlights that LOC and effort are the dominant factors influencing the model’s decision, with comment density and Halstead length playing supportive roles.

presents the Summary of explainability results.

5.12. Lightweight Transformers and Model Quantization

To address the practical limitations of computational overhead and high inference time observed in the RABERT model, we explored lightweight transformer architectures and model quantization techniques. These approaches aim to balance performance with reduced resource consumption, enabling more efficient deployment in real-time or resource-constrained environments.

5.13. DistilBERT: Lightweight Transformer Model

DistilBERT is a distilled version of the original BERT architecture, designed to retain most of BERT’s performance while reducing the model size and computational complexity. We employed DistilBERT for code smell detection to assess its applicability in comparison with the full RABERT model. The results showed that DistilBERT achieved an accuracy of 88.5% and an F1 score of 65.8%, with a significantly reduced inference time of 6.2 ms per sample—making it highly suitable for real-time or edge device applications. However, a slight performance trade-off was observed compared to the full RABERT model.

5.14. Model Quantization of RABERT

Model quantization is a compression technique that reduces the precision of model weights (e.g., from float32 to int8), thereby decreasing model size and inference time without substantial loss in performance. We applied post-training static quantization to the RABERT model. The quantized RABERT model achieved an accuracy of 89.2% and an F1 score of 66.5% while reducing the inference time to 8.5 ms per sample. This result demonstrates that quantization provides a balanced solution between model performance and efficiency.

Table 11. Summary of lightweight transformers and model quantization.

Model	Inference Time (Ms/Sample)	Observation	Accuracy	Precision	Recall	F1 Score
RABERT (Full Model)	15.8	Best accuracy and precision, but higher inference time	90.0	91.0	53.0	67.1
DistilBERT (Lightweight)	6.2	Reduced model size and faster inference with slight performance trade-off	88.5	89.2	52.5	65.8
Quantized RABERT	8.5	Balance between accuracy and reduced inference time after quantization	89.2	90.0	52.8	66.5

presents the Model Quantization of RABERT.

These results highlight that both DistilBERT and model quantization strategies offer viable solutions for efficient deployment of transformer-based models in software quality assurance tasks. Future work can explore hybrid models or optimization techniques, such as pruning or knowledge distillation, to further enhance efficiency without sacrificing performance.

6. Findings and Their Implications

The introduction of relation-aware embeddings in transformers for software analysis aligns with broader trends in AI-driven software engineering. Unlike classical models, which rely on handcrafted metrics, transformers can learn semantic and syntactic dependencies autonomously. This study highlights how relational modeling enhances code smell detection, offering a step forward in automated software quality assurance. However, future work should focus on addressing the trade-off between recall and computational efficiency to enhance real-world applicability.

6.1. Key Findings

This study demonstrates the effectiveness of transformer-based architectures, particularly RABERT, in advancing code smell detection. While gradient boosting emerged as the top-performing classical model with balanced accuracy and F1 score, transformer models like RABERT achieved superior accuracy and precision by leveraging contextual and relational embeddings. Hierarchical BERT is ideal for datasets with hierarchical relationships, as it provides a balance between precision and recall.

6.2. Practical Implications

• For academia:

Architectural innovations: This work demonstrates a way of incorporating both relational and hierarchical embeddings into transformer models to perform highly complex code analysis tasks. This creates directions for the subsequent investigation of domain-specific variations of transformer-based structures.

Feature and relational modeling: FABERT and RABERT demonstrate how using feature and relational embedding improves the result, which poses the question of how one might improve the embedding further.

2.

For practitioners:

Enhanced detection tools: we suggest that transformers, specifically RABERT, be incorporated into SQA tools in order to enhance the accuracy and applicability of recommendations for mitigating debt and enhancing maintainability of code.

Precision vs. recall trade-off: RABERT’s high precision but low recall shows that the model is appropriate for situations where false negatives are essential, such as critical systems. On the other hand, exhaustive detection jobs may use Hierarchical BERT or ensemble methods.

6.3. Challenges and Limitations

• Class imbalance: RABERT’s low recall highlights the impact of imbalanced datasets on model performance. Future efforts should explore techniques like oversampling, cost-sensitive learning, or hybrid models to address this limitation.
• Computational demands: The resource-intensive nature of transformer models poses challenges for real-time or large-scale deployment. Optimizing these models or leveraging lightweight transformer variants could mitigate these challenges.
• Interpretability: While transformer models offer state-of-the-art performance, their black-box nature can limit interpretability. Incorporating explainable AI techniques could enhance trust and usability.

6.4. Threats to Validity

While this study demonstrates the effectiveness of transformer-based models, particularly Relation-Aware BERT (RABERT), in detecting large class code smells, several threats to validity must be considered.

6.4.1. Internal Validity

Feature selection bias: The dataset contains 20 software metrics, and feature selection was performed using Pearson correlation. While this method eliminates noise, it might exclude useful features that contribute to detecting large class smells.

Data preprocessing choices: the normalization of numerical features using min-max scaling and handling of missing values using median imputation could introduce biases in the feature distribution, affecting model performance.

Hyperparameter tuning: The transformer-based models were fine-tuned with specific hyperparameters (learning rate, batch size, etc.), which may not be optimal for all datasets. While early stopping was used, a more exhaustive hyperparameter search could further improve results.

6.4.2. External Validity

Generalizability to other code smells: This study focuses on detecting large class smells. The model’s effectiveness on other code smells (e.g., long method, God class, feature envy) remains untested. Future work should extend the approach to multiple code smell types.

Programming language dependency: The dataset includes only Java classes. While Java is widely used in software engineering, the model’s performance on other languages like Python, C++, or JavaScript remains uncertain. Future studies should evaluate cross-language performance.

Dataset source bias: we collected the dataset from open-source projects on GitHub, which may not represent proprietary or industrial codebases with different coding styles and complexity patterns.

6.4.3. Construct Validity

Code smell labeling methodology: The dataset was labeled based on predefined thresholds for large class (e.g., lines of code, Halstead complexity). These thresholds may not align with developer perceptions of what constitutes a “large class” in different contexts.

Evaluation metrics: While accuracy, precision, recall, and F1 score were used, they do not capture the interpretability of model predictions. Incorporating explainable AI (XAI) techniques could enhance trust in the model’s decisions.

6.4.4. Conclusion Validity

Class imbalance effects: This study reports that Relation-Aware BERT (RABERT) achieved high precision (91.0%) but low recall (53.0%), indicating an imbalance in detecting positive cases. Future work should explore oversampling, cost-sensitive learning, or hybrid approaches to improve recall.

Comparison with classical models: While transformer models outperformed classical models (e.g., gradient boosting), the computational overhead of transformers could make them impractical for real-time or large-scale systems. Investigating lightweight transformer variants such as DistilBERT or MobileBERT could address this limitation.

6.4.5. Future Work to Address These Threats

• Expanding the dataset to include multiple programming languages.
• Incorporating multiple types of code smells for a broader evaluation.
• Implementing explainable AI (XAI) techniques for interpretability.
• Evaluating hybrid models combining transformers with classical approaches for better recall and computational efficiency.

6.4.6. Comparison with Recent Transformer Models

It is worthwhile to compare our approach with recent transformer models such as CodeBERT and GraphCodeBERT, which have been widely recognized for their strengths in code representation tasks. CodeBERT, with its pre-training on both natural language and code, and GraphCodeBERT, which enriches code embeddings by incorporating data flow information, have set new benchmarks in capturing code semantics and structural information. However, unlike these models, our RABERT leverages relational embeddings that explicitly model the dependencies between software metrics, thereby achieving high precision in detecting code smells.

6.5. Future Directions

Domain-specific pretraining: adapting pretraining tasks to incorporate software-specific semantics, such as code structure and dependencies, could improve model performance further.

Hybrid approaches: combining transformer-based models with classical approaches, such as gradient boosting, may yield complementary strengths, particularly for imbalanced datasets.

Lightweight models: exploring efficient transformer architectures, such as DistilBERT or MobileBERT, can reduce computational overhead while maintaining high accuracy.

Broader applications: extending these methodologies to related tasks like defect prediction, effort estimation, and code refactoring recommendations can broaden their impact.

7. Conclusions

This study presents Relation-Aware BERT (RABERT), a novel transformer-based model for code smell detection that leverages relational embeddings to capture dependencies in software metrics. The results demonstrate that RABERT outperforms classical ML models, achieving an accuracy of 90% and a precision of 91%, making it a strong candidate for integration into software quality assurance tools.

However, challenges remain, particularly in handling imbalanced datasets, where RABERT shows lower recall (53.0%). Addressing this trade-off through data augmentation, cost-sensitive learning, or hybrid approaches is a promising direction for future research. Moreover, extending this framework to detect multiple types of code smells across different programming languages could enhance its applicability in diverse software projects.

Consequently, the paper’s results are of considerable importance for both academic and industry contexts. Researchers stand to benefit from this work by gaining insights into how they may apply contextual and relational embeddings to transformer models to aid in software quality assurance problems. To the practitioners, these models offer reliable information on the state of the code to enhance the recognition of structural defects and ways of responding to them.

At this point, areas for future work include developing better models for domain-specific pretraining to improve semantic comprehension, improving optimization techniques to minimize computational costs, and generalizing these models to other software engineering problems, such as smell prediction and refactoring suggestions. Furthermore, techniques such as explainable AI could be adopted to enhance the interpretability of transformer models and thus enhance their usage in industrial applications.

This work contributes to the growing field of AI-driven software engineering by demonstrating how relational embeddings improve contextual understanding in software analysis. Future work should explore explainable AI (XAI) methods to make transformer-based models more interpretable for developers and industry practitioners.

In summary, this study makes two pivotal contributions to the field of software engineering. First, by introducing Relation-Aware BERT (RABERT), we demonstrate that integrating relational embedding into transformer architectures can significantly enhance code smell detection, particularly for complex patterns such as the large class smell. Second, our comprehensive analysis comparing classical machine-learning models with transformer-based approaches highlights the potential of advanced deep-learning methods to overcome limitations posed by traditional feature engineering. These findings not only drive forward the development of more accurate and robust automated quality assurance tools but also lay the groundwork for future research aimed at addressing challenges such as recall imbalance and computational overhead. The field is poised to benefit from these innovations, paving the way for more effective software maintenance and improved reliability in large-scale systems.