Software Fault Localization Approach with Coverage Matrix Optimization Boosted by LLM-Based Code Naturalness

Abstract

Spectrum-based fault localization (SBFL), one of the typical types of software fault localization techniques, has been widely adopted to assist developers in identifying faulty program elements. However, conventional SBFL techniques rely solely on test coverage statistics and overlook intrinsic characteristics of the source code itself. To fill this gap, this study proposes an enhanced SBFL approach, code-naturalness-based fault localization (CNFL), which incorporates code naturalness evaluated by a large language model (LLM) in the pipeline of SBFL. By weighting program statements according to their naturalness scores, CNFL prioritizes statements that deviate from typical coding patterns and therefore optimizes the coverage matrix for effective fault localization. Comprehensive experiments are conducted on the Defects4J dataset with five representative SBFL formulas and five LLMs for naturalness evaluation. The results demonstrate that CNFL significantly outperforms conventional SBFL techniques. Specifically, it boosts the Top-1 fault localization hit rate by up to 60.8% and 56.8% when applied to classic SBFL formulas like Jaccard and Ochiai, respectively. Moreover, CNFL is further confirmed to consistently surpass both standalone LLM methods and representative fault localization approaches that primarily optimize the coverage matrix.

1. Introduction

Modern software systems are increasingly complex and continuously evolving, making debugging a time-consuming and costly activity in the software development lifecycle. Among debugging tasks, fault localization, which aims to identify the specific program elements responsible for observed failures, plays a critical role in reducing maintenance effort and improving software reliability. However, manually inspecting large codebases to pinpoint defects is often impractical, especially when failures emerge from subtle interactions among program components. To address this challenge, automated fault localization techniques have been extensively studied, with spectrum-based fault localization (SBFL) emerging as one of the most widely adopted approaches. SBFL techniques analyze execution information from passing and failing test cases to compute suspiciousness scores for program elements, thereby guiding developers toward likely fault locations. Owing to its lightweight design, language-agnostic nature, ease of use, and relatively low overhead in test execution time [1,2], SBFL has been widely adopted.

However, SBFL techniques mainly rely on dynamic execution information and overlook the intrinsic properties of the source code itself. Such reliance on coverage statistics alone may lead to inaccurate suspiciousness rankings, particularly in the presence of coincidental correctness, insufficient test coverage, or weak failure discrimination. Therefore, there remains an urgent need to investigate complementary signals and innovative perspectives to further improve the performance of SBFL techniques.

Code naturalness [3] refers to the observation that source code, like natural language, exhibits regular and predictable patterns that can be statistically modeled from large code corpora. Developers tend to follow common idioms, naming conventions, and structural patterns. Therefore, code that deviates from these norms is often less natural and has been empirically associated with a higher likelihood of defects. This insight has been leveraged in various software engineering tasks, including bug detection, code completion, and defect repair [4,5,6]. On the other hand, recent advancements in large language models trained on code have significantly expanded the scope of research on code naturalness. They are demonstrated to be able to more accurately evaluate code naturalness.

This study focuses on enhancing the effectiveness of SBFL by leveraging the strength of code naturalness. To this end, we propose a novel code-naturalness–based fault localization approach, CNFL. The rationale behind CNFL is that code naturalness captures intrinsic properties of the code itself and thus can provide complementary information for optimizing the coverage matrix. Specifically, CNFL leverages LLMs for code to quantify the naturalness scores of individual statements, and then uses these scores to adjust the statements’ weights within the coverage matrix. The refined coverage matrix is further processed by standard risk evaluation formulas.

To evaluate the effectiveness of CNFL, we select five representative SBFL formulas (Ochiai [7], Dstar [8], OP2 [9], Russell_Rao [10], and Jaccard [11]). We also employ five large language models spanning three different architectures to quantify the naturalness of code statements, including InCoder [12], DeepSeekCoder [13], Qwen2.5-Coder [14], CodeBERT [15], and CodeT5Plus [16]. We then conduct extensive experiments on 395 faulty versions across six real-world software projects. The results demonstrate the feasibility as well as the effectiveness of CNFL.

The main contributions of this work can be summarized as follows.

• Novel methodology. We propose CNFL, a fault localization approach that leverages code naturalness quantified by LLMs to optimize the coverage matrix. Unlike conventional SBFL techniques that treat all executed program elements uniformly, CNFL differentiates elements based on their naturalness scores, effectively suppressing noise and highlighting fault-relevant information. CNFL preserves the standard SBFL workflow, ensuring seamless integration with existing fault localization techniques while introducing a fundamentally new optimization dimension.
• Comprehensive empirical evaluation. We conduct extensive experiments on the Defects4J benchmark, comparing CNFL against five representative SBFL baselines and two state-of-the-art fault localization approaches. The experimental design explicitly isolates the impact of code-naturalness-based optimization from other confounding factors, providing a fair and rigorous assessment.
• New insights into the effectiveness and efficiency. We systematically analyze the experimental results to uncover when and why CNFL provides significant gains. Our findings reveal that CNFL not only outperforms traditional SBFL methods, but also exhibits consistent superiority over other fault localization approaches that focus on coverage matrix optimization. Notably, the findings also show the impacts of LLMs on the effectiveness and efficiency of CNFL.

The remainder of this paper is organized as follows. Section 2 provides the necessary background, followed by a review of related work in Section 3. Section 4 details our proposed CNFL approach. The experimental setup and results are presented in Section 5 and Section 6, respectively. Finally, Section 7 discusses potential threats to validity, and Section 8 concludes the paper with future research directions.

2. Background

This section describes background knowledge about this study. We first introduce the spectrum-based fault localization method, and then clarify the essence of code naturalness.

2.1. Spectrum-Based Fault Localization

Spectrum-based fault localization (SBFL) is a widely studied automated debugging technique. It aims to alleviate the high cost and inefficiency of manual fault identification in complex software systems. Its core logic is based on the fundamental heuristic that software defects manifest as differences in execution behavior between passing and failing test cases [17]. By systematically comparing the spectral differences between these two execution states, SBFL quantifies the statistical correlation between program elements and failing test cases, thereby identifying highly suspicious code regions [18,19].

In practice, SBFL techniques collect two primary types of spectra from test executions: code coverage (which elements are executed) and test outcomes (pass/fail). Based on this information, the following key statistics are summarized:

-

$e_f$: The number of failing test cases that execute the program element.

-

$e_p$: The number of passing test cases that execute the program element.

-

$n_f$: The number of failing test cases that do not execute program element.

-

$n_p$: The number of passing test cases that do not execute program element.

-

$T_f$: The total number of failing test cases in the test suite.

-

$T_p$: The total number of passing test cases in the test suite.

These statistics form the basis for computing a suspiciousness score for each element using a SBFL formula. SBFL techniques leverage various formulas to quantify the association strength between executionbehavior and observed failures. While these formulas generally follow the heuristic that a statement ismore likely to be faulty if it is executed frequently in failing tests and rarely in passing tests, their respective aims differ in their statistical emphasis. The SBFL formulas investigated in this study are described as follows:

• Jaccard [11] aims to quantify the suspiciousness of a program element by measuring the proportion of failing test cases in which the element is executed among all relevant test cases. The formula is defined as follows.

  $$\mathrm{Jaccard}={\displaystyle \frac{e_f}{T_f+e_p}}$$

  (1)
• Ochiai [7] adopts a non-linear formulation to balance the influence of passing and failing test cases, which helps mitigate the coincidental correctness problem, where faulty statements are executed but tests still pass. The formula is as follows.

  $$\mathrm{Ochiai}={\displaystyle \frac{e_f}{\sqrt{T_f·(e_f+e_p)}}}$$

  (2)
• Dstar [8] exponentially increases the contribution of elements executed in failing tests (typically $\alpha =2$), thereby amplifying fault signals and improving the distinguishability of faulty elements. The formula is defined as follows.

  $$\mathrm{Dstar}={\displaystyle \frac{e_f^{\alpha }}{e_p+n_f}}$$

  (3)
• Op2 [9] ranks program elements by computing the difference between execution frequencies in failing and passing test cases, thereby enhancing the distinguishability between faulty and non-faulty elements. The formula is as follows.

  $$\mathrm{Op}2=e_f-{\displaystyle \frac{e_p}{T_p+1}}$$

  (4)
• Russell_Rao [10] computes the absolute probability of a program element appearing in failing tests. Its objective is to identify entities that exhibit high coverage consistency in failing tests. The formula is defined as follows.

  $$\mathrm{Russell}\_\mathrm{Rao}={\displaystyle \frac{e_f}{T_f+T_p}}$$

  (5)

Based on these suspiciousness scores, SBFL provides developers with a prioritized list of candidates for inspection. Compared to manual code examination, this approach dramatically narrows the fault search space. The key characteristics of SBFL include its excellent scalability, simplicity, and independence from specific fault types or program semantics, making it a cornerstone technology in the field of automated fault localization.

2.2. Naturalness of Source Code

Despite the inherent complexity and expressive power of natural languages, their usage in practice tends to follow relatively simple patterns, resulting in a high degree of repetitiveness and predictability [20]. This property, commonly referred to as naturalness, has been widely exploited in the field of natural language processing. Previous research has shown that programming languages also exhibit statistical repetitiveness and predictability, which is commonly referred to as code naturalness [21]. This concept is easy to understand. For instance, when a developer types “$for(i=0;i<10$” in an integrated development environment, the AI assistant can automatically suggest the completion “$;i++)$”. This is because such patterns are frequently used in practice and constitute a significant part of assistant training data. Consequently, Li et al. [22] define code naturalness as the degree of fluency and adherence to expressive habits that code exhibits in a specific context.

Existing studies employ various language models to learn statistical regularities in source code by modeling the probability distribution over token sequences. Early approaches typically rely on n-gram language models to capture local lexical and syntactic regularities in source code. These models assign higher naturalness scores to code fragments that are more likely to occur under the learned distributions [23,24]. More recent methods [25,26] extend these models with neural networks, such as recurrent or convolutional architectures, to incorporate longer-range dependencies and richer contextual information. However, these approaches often require task-specific feature engineering and are limited in their ability to generalize across diverse programming contexts. In contrast, LLMs trained on diverse code corpora are able to capture semantic and syntactic regularities with minimal reliance on manually engineered features. This enables LLMs to provide more expressive and context-aware estimations of code naturalness.

To quantify statistical regularities in source code, existing studies [22,27] typically employ cross-entropy as a measure of code naturalness. Cross-entropy reflects the average uncertainty of a language model when predicting a code sequence. Lower values indicate that the code aligns well with the statistical patterns learned by the model, while higher values may suggest the presence of defects or unusual patterns.

3. Related Work

SBFL is a widely utilized fault localization technique. In recent decades, numerous SBFL formulas have been developed. One of the earliest and most representative formulas is Tarantula. Subsequently, several similarity measures initially developed in the field of molecular biology were adapted for fault localization, including Ochiai. Although these formulas differ in their specific statistical formulations, they generally follow the same underlying paradigm, namely characterizing the correlation between program elements and test failures based on execution spectra.

As the foundational framework of SBFL progresses, several studies have improved its effectiveness by integrating additional program information. For example, Baah et al. [28] introduced causal inference to discard non-faulty statements that are executed due to dependency relationships. Similarly, Mao et al. [29] employed program slicing to remove logically irrelevant statements, further refining the suspiciousness calculation. Beyond statement coverage, Santelices et al. [30] incorporated branch coverage and data-flow coverage, which not only enhanced localization accuracy but also mitigated the tie issue. Furthermore, Le et al. [31] introduced a multimodal framework that combines information retrieval techniques with SBFL. By leveraging the semantic similarity between bug reports and source code, their hybrid approach effectively integrates dynamic spectrum analysis with static semantic information to achieve more precise ranking outcomes.

Beyond integrating external information, several studies have focused on improving the performance of SBFL by optimizing the coverage matrix. For instance, Li et al. [32] proposed a strategy to optimize the coverage matrix by assigning different weights to test cases. Specifically, failed test cases that cover fewer statements are assigned higher weights because they provide more precise information for fault localization. Similarly, passing test cases whose execution paths differ significantly from those of failing test cases are also assigned higher weights, as they offer additional evidence for ruling out non-faulty statements. These weights are then incorporated into the coverage matrix, thereby reflecting the relative importance of each test case in the computation of suspiciousness scores. Building upon Li et al.’s approach of assigning weights to test cases, Dutta et al. [33] further incorporated statement execution frequency to optimize the coverage matrix. Specifically, each statement is assigned a weight based on its frequency of execution across failing test cases, while each test case is also assigned a contribution score reflecting its significance in distinguishing faulty statements from non-faulty ones. These weights and contribution scores are then integrated into the coverage matrix, thereby enabling the computation of suspiciousness scores to simultaneously account for the relative importance of both statements and test cases.

In addition, some studies explore learning-based approaches that integrate execution spectra with various static and dynamic features to further improve the performance of SBFL. For example, Xuan Le et al. [34] framed fault localization as a Learning-to-Rank problem and utilized neural networks to automatically derive an optimal nonlinear combination of over 30 SBFL formulas. In a similar vein, Sohn and Papadakis [35] introduced the Fluccs framework, which uses machine learning models to combine execution spectrum data with evolutionary features such as code complexity and code change history. This approach improves robustness when handling complex faults.

Different from these strategies, this study introduces code naturalness as a complementary statistical information to optimize the SBFL framework at the coverage matrix level. While current approaches predominantly focus on execution behavior or structural dependencies, they seldom account for the statistical regularities inherent in the source code itself. Code naturalness quantifies the probability distribution of code within large-scale corpora and indicates the degree of irregularity present in program statements. Rather than altering suspiciousness formulas or developing new learning models, our method modifies execution entries in the coverage matrix through naturalness-based weighting. Consequently, statistically uncommon statements executed by failing tests acquire greater discriminative power. This matrix-level optimization offers an alternative pathway for improvement that maintains the generality of SBFL while enhancing its fault localization capabilities.

4. Methodology

This section presents an innovative SBFL approach, CNFL, which optimizes the coverage matrix by leveraging code naturalness. The key novelty of CNFL lies in the use of code naturalness to assign different weights to program statements, thereby constructing a weighted coverage matrix for more effective fault localization.

4.1. Overview

Figure 1 presents the overall workflow of the CNFL approach. Given a program and its test suite, CNFL generates a ranked list of program statements with their corresponding suspiciousness scores. The whole procedure consists of the following stages.

• Constructing the original coverage matrix. CNFL firstly constructs the original coverage matrix by executing the program with the given test suite. Each element in the coverage matrix is either 1 or 0, denoting whether a statement is covered by the relevant test case.
• Code naturalness calculation. CNFL collects statements that are covered by failing test cases. For each statement, it further performs code naturalness evaluation to obtain the naturalness score.
• Naturalness-aware statement weight optimization. CNFL integrates the naturalness scores of statements into the original coverage matrix. Specifically, the raw naturalness scores are first converted into standardized weights through a normalization process. These weights are then assigned to the corresponding statements, and the original binary coverage states (0 or 1) are multiplied by their respective weights to construct a weighted coverage matrix. In the weighted coverage matrix, executed statements are no longer treated as equivalent; instead, they are assigned distinct suspiciousness contributions according to their naturalness scores.
• Calculating suspicious scores. The statistical formulas are then applied to the weighted coverage matrix to deliver the list of statements and their suspicious scores.

4.2. Code Naturalness Calculation

In this study, we adopt two strategies for quantifying code naturalness, categorized by their model architectures and pre-training objectives: autoregressive modeling and mask-based modeling.

For decoder-only models, the primary pre-training objective is causal language modeling (CLM) [12,13,14], which models the probability distribution of subsequent tokens conditioned on preceding context. Encoder–decoder models, although trained with a span prediction objective, reconstruct masked token sequences through the decoder in an inherently autoregressive manner. Therefore, for both architectures, we employ autoregressive conditional likelihood to measure code naturalness.

Formally, a target code line can be described as a sequence of tokens with length n:

$$L=(t_1,t_2,\dots ,t_n)$$

(6)

Given the preceding context Pre, the model predicts each token sequentially. The conditional probability of the i-th token $t_i$ is defined as follows:

$$P\left(t_i\right)=P(t_i\mid \mathrm{Pre},t_1,\dots ,t_{i-1})$$

(7)

For encoder-only models, pre-training is based on masked language modeling (MLM) [15], which reconstructs masked tokens using bidirectional context. To quantify the naturalness of L, each token $t_i$ is masked in turn and its prediction probability is calculated under the bidirectional context consisting of Pre, the succeeding context Suf, and the remaining tokens in the same line:

$$P\left(t_i\right)=P(t_i\mid \mathrm{Pre},t_1,\dots ,t_{i-1},,t_{i+1},\dots ,t_n,\mathrm{Suf})$$

(8)

Although different architectures obtain conditional probabilities through distinct mechanisms, the naturalness score of the entire code line is uniformly measured using Average Cross-Entropy. To account for the effect of line length, we normalize the negative log-likelihood by the total number of tokens:

$$H\left(L\right)=-{\displaystyle \frac{1}{n}}\sum _{i=1}^{n}logP\left(t_i\right)$$

(9)

Here, n denotes the total number of tokens in the target line. From an information-theoretic perspective, $H\left(L\right)$ represents the average uncertainty of the model when predicting the token sequence. A lower cross-entropy value indicates that the code closely follows the statistical patterns learned by the model, reflecting higher naturalness. In contrast, higher values suggest deviations from common coding patterns, which may correspond to structural anomalies or logical defects.

4.3. Naturalness-Aware Statement Weight Optimization

Existing SBFL methods typically represent the execution spectrum using a binary coverage matrix, with entries reflecting the execution status of statements under test cases. However, intrinsic code properties, such as semantic complexity and statistical regularity, are overlooked in the existing coverage matrix.

Models trained on large-scale, high-quality code corpora provide naturalness estimates that capture both statistical and semantic regularities of source code. These estimates offer valuable signals for fault localization. Motivated by this observation, we propose to use code naturalness to enrich the information of the coverage matrix. Specifically, CNFL employs LLMs for code to evaluate code naturalness, which is then used to optimize the coverage matrix.

Given a test suite T and a faulty program P, CNFL first executes P with each test case in T to obtain the original coverage matrix X. For each statement $s_j$ executed by at least one failing test case, we calculate a raw naturalness score $H\left(s_j\right)$ using cross-entropy values obtained from the LLMs for code. To ensure numerical consistency and emphasize unnatural code, we apply min–max normalization to map these scores into a normalized weight $W_j\in [0,1]$:

$$W_j={\displaystyle \frac{H\left(s_j\right)-min\left(H_{fail}\right)}{max\left(H_{fail}\right)-min\left(H_{fail}\right)}}$$

(10)

$H_{fail}$ denotes the set of raw entropy scores for all statements present in the failing execution traces of the faulty program. Subsequently, these normalized scores are used to refine the original coverage matrix X into a weighted spectrum matrix $X^{\prime }$ according to the following formula:

$$x_{ij}^{\prime }=\left\{\begin{array}{cc}{x}_{ij}·W_j,\hfill & \mathrm{if}i\in T_F\hfill \\ x_{ij},\hfill & \mathrm{if}i\notin T_F\hfill \end{array}\right.$$

(11)

$T_F$ represents the set of indices for failing test cases. This optimization strategy incorporates semantic information into the coverage matrix by modifying the execution indicators. Statements executed by failing test cases are assigned their statistical naturalness scores, while unexecuted statements and those covered by passing traces remain unchanged. Finally, the existing SBFL suspiciousness formulas are applied to the weighted matrix $X^{\prime }$ to compute the fault ranking. This process effectively emphasizes statements that deviate from common programming patterns within the failure traces.

4.4. A Motivating Example

Table 1. A running example of CNFL.

Code Snippet									Test Suite						Fault Information
L1: x1 = 0;		L8: else: {x1 = y + 1;			L15: {output (x1);}				The test suite contains two passing test cases and two failing test cases.						Fault statement is L10; its correct version is if (n > 6).
L2: x2 = 0;		L9: x2 = z + 1;			L16: else: {output (x2);}
L3: x3 = 0;		L10: if (n > 0):			L17: output (x3);}
L4: if (y < 0):		L11: {n = n + z;}
L5: {x1 = y;		L12: else: n = n + y}
L6: x2 = z;		L13: x3 = n + 1;
L7: x3 = n;}		L14: if (z > 0):
Detailed Score Annotation (L9–L11)
L9: x2 = z + 1;              // Ochiai = 0.816, natural_score = 0.32       L10: if (n > 0):              // Ochiai = 0.816, natural_score = 0.80       L11: {n = n + z;}              // Ochiai = 0.816, natural_score = 0.34
test	n, y, z	L1	L2	L3	L4	L5	L6	L7	L8	L9	L10	L11	L12	L13	L14	L15	L16	L17	result
t1	9, 2, 6	1	1	1	1	0	0	0	1	1	1	1	0	1	1	1	0	0	0
t2	8, −2, 6	1	1	1	1	1	1	1	0	0	0	0	0	1	1	1	0	0	0
t3	2, 2, −3	1	1	1	1	0	0	0	1	1	1	1	0	1	1	0	1	1	1
t4	1, 8, 5	1	1	1	1	0	0	0	1	1	1	1	0	1	1	1	0	0	1
Ochiai	susp	0.707	0.707	0.707	0.707	0.00	0.00	0.00	0.816	0.816	0.816	0.816	0.00	0.707	0.707	0.408	0.707	0.707	-
	rank	5	6	7	8	14	15	16	1	2	3	4	17	9	10	13	11	12	-
t1	9, 2, 6	0.12	0.15	0.13	0.20	0	0	0	0.30	0.32	0.80	0.34	0	0.28	0.36	0.32	0	0	0
t2	8, −2, 6	0.12	0.15	0.13	0.20	0	0	0	0	0	0	0	0	0.28	0.36	0.32	0	0	0
t3	2, 2, −3	0.12	0.15	0.13	0.20	0	0	0	0.30	0.32	0.80	0.34	0	0.28	0.36	0	0.40	0.22	1
t4	1, 8, 5	0.12	0.15	0.13	0.20	0	0	0	0.30	0.32	0.80	0.34	0	0.28	0.36	0.32	0	0	1
CNFL	selected	✓	✓	✓	✓	×	×	×	✓	✓	✓	✓	×	✓	✓	✓	✓	✓	-
	susp	0.244	0.273	0.254	0.316	0	0	0	0.447	0.461	0.730	0.336	0	0.374	0.424	0.231	0.447	0.331	-
	rank	12	10	11	9	14	15	16	3	2	1	7	17	6	5	13	4	8	-

The symbols ✓ and × indicate whether a statement is selected or not selected by the CNFL method, respectively.

shows an example of applying CNFL to a program with 17 statements (that is, statements L1 to L17), where the faulty statement is L10 (the correct code should be if (n > 6)). The column labeled “n, y, z” corresponds to input parameters of the program, and the data in this column represent test cases. There are four test cases, including two passing (t1 and t2) and two failing test cases (t3 and t4). In the column “result”, the outcome of each test case is reported, where 0 and 1 denote passing and failing, respectively. The entries under “L1–L17” indicate statement coverage, with 1 representing a covered statement and 0 indicating otherwise. As the traditional SBFL, Ochiai is applied in this example, with the suspicious score of each statement being reported in the row “Ochiai”.

CNFL first applies Ochiai to obtain the original suspicious scores of statements. Then, it focuses on statements executed by failing test cases. That is, statements that are not executed by any failing test case, including L5, L6, L7, and L12, are ignored. For the other statements, that is, L1, L2, L3, L4, L8, L9, L10, and L11, CNFL evaluates their naturalness scores using LLMs and assigns weights accordingly. Table 1 shows the original execution matrix (row 5) and the optimized matrix based on code naturalness (row 7). Specifically, the optimized matrix is constructed by replacing the execution status “1” in the original matrix with the statement’s corresponding naturalness score. For instance, as shown in the Detailed Score Annotation, the naturalness score of L10 is 0.80; thus, its execution states in $t_1$, $t_3$, and $t_4$ are transformed from 1 to 0.80 in the optimized matrix, while the unexecuted state “0” in $t_2$ remains unchanged. The suspiciousness scores calculated by using the optimized matrix are displayed in the “CNFL” row of Table 1. It can be observed that these scores are quite different from the original suspiciousness scores calculated based on the original matrix. Taking statements L9, L10, and L11 as an example, their original suspiciousness scores are all 0.816, while the suspiciousness scores calculated by CNFL are 0.461, 0.730, and 0.336, respectively. Because of this adjustment, it can be found that CNFL ranks the faulty statement, L10, in the prior position as compared to Ochiai. For Ochiai, the suspiciousness ranking of statements is {L8, L9, L10, L11, L1, L2, L3, L4, L13, L14, L16, L17, L15, L5, L6, L7, L12}, with the faulty statement L10 appearing in the 3rd position. However, the resulting suspiciousness ranking of CNFL is {L10, L9, L8, L16, L14, L13, L11, L17, L4, L2, L3, L1, L15}, where the faulty statement L10 is now in the 1st position, indicating an improvement in fault localization effectiveness.

5. Experimental Setup

This section describes the experimental configurations. We first clarify our research questions, and then report the setup of the experiments, including the baselines, dataset, LLMs employed for evaluating code naturalness, and the evaluation metrics.

5.1. Research Questions

RQ1: How much does CNFL improve on existing SBFL methods and the standalone LLM method? This research question investigates the extent to which CNFL improves both traditional SBFL methods and standalone LLM approaches. To answer this question, we compare the fault localization effectiveness of CNFL with both SBFL methods and LLMs.

RQ2: What effect do the model capacities have on the effectiveness of CNFL? In this research question, one of the key factors, the capability of the LLM for code naturalness, is analyzed to reveal its impact on the effectiveness of CNFL.

RQ3: How effective is CNFL compared with other weight-based fault localization methods? In this research question, we compare CNFL with some of the state-of-the-art fault localization methods that adopt matrix weight optimization strategies in order to demonstrate the effectiveness of CNFL.

5.2. Baselines

Since CNFL is built upon SBFL methods, some SBFL methods are employed in the comparison analysis. To ensure the representativeness of the selected SBFL methods, we follow the classification framework proposed in a previous study [9] and choose five representative techniques from different categories, including Ochiai [7], Dstar [8], OP2 [9], Russell_Rao [10] and Jaccard [11]. In the experiments, we employ GZoltar [36] to implement these methods. Furthermore, we also apply LLMs as standalone fault localization approaches and employ them as a comparison baseline.

To further demonstrate the effectiveness of CNFL, we also select two existing weight-optimization-based fault localization methods, WTCFL [32] and PFL [33], as baselines for comparison. These methods primarily focus on optimizing the coverage matrix by exploiting internal execution patterns or statistical frequencies, which improves the discriminative power of SBFL.

5.3. Dataset

We conduct experiments on Defects4J v1.2 [37], which is a widely used benchmark dataset in the field of fault localization [35,38,39]. The dataset contains 395 real-world faults from six open-source Java projects, including Chart, Lang, Math, Time, Mockito, and Closure.

Table 2. Overview of the benchmark used in the experiments.

ID	Description	Faults	LoC (K)	Tests
Chart	JFreeChart	26	96	2205
Lang	Commons Lang	65	22	2245
Math	Commons Math	106	85	3602
Time	Joda-Time	27	28	4130
Mockito	Unit tests Framework	38	67	1075
Closure	Closure compiler	133	90	7927

summarizes key information for each subject program, including its name, the number of fault versions, the executable statements, and the number of test cases.

5.4. LLMs for Code Naturalness Evaluation

To mitigate the potential bias introduced by any single model, we utilize five representative LLMs that are specifically pre-trained on code. These models, summarized in

Table 3. Large language model code summary.

Year	Model	Architecture	Size
2020	CodeBERT	Encoder-only	125 M
2022	InCoder	Decoder-only	6.7 B
2024	DeepSeekCoder	Decoder-only	6.7 B
2024	Qwen2.5-Coder	Decoder-only	7 B
2023	CodeT5Plus	Encoder–decoder	6 B

, span a range of architectures and parameter scales, ensuring a more comprehensive and reliable evaluation of the method we propose. Specifically, these models can be categorized into three primary architectural types.

• Decoder-only models. Three models, InCoder (6.7 B, Meta AI, Menlo Park, CA, USA) [12], DeepSeekCoder (6.7 B Base, DeepSeek-AI, Hangzhou, China) [13], and Qwen2.5-Coder (7 B Base, Alibaba Cloud, Hangzhou, China) [14] are selected. All of them adopt a Transformer decoder architecture and are trained using the CLM objective. Although they share the same training objective, each model differs in its approach: InCoder uses a causal infilling strategy, DeepSeekCoder adheres to the standard CLM framework, and Qwen2.5-Coder integrates instruction tuning to address complex programming and debugging tasks.
• Encoder-only model. CodeBERT (125 M Base, Microsoft, Redmond, WA, USA) [15] is selected to represent this architecture. It leverages MLM and replaced token detection to capture bidirectional semantic correspondences between source code and natural language.
• Encoder-decoder model. CodeT5Plus (6 B, Salesforce AI Research, Palo Alto, CA, USA) [16] is selected as the representative model of this category. It uses a span-denoising objective to reconstruct masked segments from a bidirectional context, providing strong capabilities in both code understanding and generation.

We utilize a unified 200-line context window to estimate the naturalness of target statements. When the context length exceeds the processing limit of an LLM, we apply a symmetric truncation scheme to meet the model’s input constraints.

5.5. Evaluation Metrics

To comprehensively evaluate CNFL, we follow the experimental settings of existing studies [40,41] and adopt the following evaluation metrics.

Number of Top-K [42]: This metric measures the number of buggy versions in which at least one faulty statement appears within the top-K positions of the fault localization ranking. Given that prior studies indicate that most respondents typically inspect only the top five fault localization results [42], we set the value of K to 1, 3, and 5 for evaluation.

Mean First Rank (MFR) [38]: This metric measures the average rank of the first faulty statement identified by an FL method across all faulty versions.

Mean Average Rank (MAR) [38]: This metric computes the average rank of all faulty statements within each faulty version and then averages these values across all faulty versions, providing an overall assessment of the localization accuracy of the fault localization method on all faults.

6. Results and Analysis

6.1. RQ1: Comparison of CNFL with SBFL Methods as Well as LLM-Based Methods

We first investigate the effectiveness of CNFL. To this end, we focus on five baseline SBFL techniques (Ochiai, Dstar, Opt2, Jaccard, and Russell_rao) and five LLMs used for code naturalness estimation (as displayed in Table 3), resulting in 25 variants of CNFL. For ease of presentation, each variant is named as SBFLNameCNFL_ModelName. For instance, OchiaiCNFL_InCoder denotes the CNFL method that employs the InCoder model for code naturalness evaluation and applies the Ochiai formula to calculate the suspicious scores.

We compare CNFL with the relevant SBFL techniques using multiple evaluation metrics, including Top-N, MFR, and MAR. The results are reported in

Table 4. Comparison of CNFL with SBFL baselines.

SBFL Technique	Scenario/Model	Top-1	Top-3	Top-5	MFR	MAR	p-Value	A-Test
Ochiai	Ochiai	44	95	126	225.69	570.37	-	-
	OchiaiCNFLInCoder	66	110	127	175.21	519.31	7.24 × 10−8	0.601
	OchiaiCNFLCodeBERT	60	109	134	213.58	551.16	5.09 × 10−4	0.574
	OchiaiCNFLDeepSeekCoder	60	112	133	193.83	533.08	7.44 × 10−8	0.582
	OchiaiCNFLQwen2.5-Coder	69	112	130	183.12	527.92	2.52 × 10−7	0.593
	OchiaiCNFLCodeT5Plus	43	76	98	250.24	654.50	0.99	-
Dstar	Dstar	45	95	121	226.07	571.80	-	-
	DstarCNFLInCoder	59	111	129	179.19	516.71	2.56 × 10−6	0.566
	DstarCNFLCodeBERT	50	98	122	222.93	561.51	0.051	-
	${\mathrm{DstarCNFL}}_{DeepSeekCoder}$	58	108	128	196.49	534.51	3.89 × 10−4	0.538
	DstarCNFLQwen2.5-Coder	65	112	130	187.61	523.65	4.03 × 10−3	0.582
	DstarCNFLCodeT5Plus	36	72	94	251.06	651.83	0.99	-
Opt2	Opt2	41	87	109	396.60	783.74	-	-
	Opt2CNFLInCoder	43	76	98	303.17	736.13	2.37 × 10−3	0.517
	Opt2CNFLCodeBERT	38	73	91	383.76	773.43	0.057	-
	Opt2CNFLDeepSeekCoder	38	75	94	317.44	756.0	0.063	-
	Opt2CNFLQwen2.5-Coder	32	79	94	360.58	807.56	0.14	-
	Opt2CNFLCodeT5Plus	13	34	48	451.05	874.94	0.99	-
Jaccard	Jaccard	46	92	124	212.21	557.10	-	-
	JaccardCNFLInCoder	65	118	140	204.13	532.85	1.82 × 10−20	0.611
	JaccardCNFLCodeBERT	56	104	128	205.17	543.23	0.053	-
	JaccardCNFLDeepSeekCoder	66	122	143	199.54	525.08	6.59 × 10−22	0.622
	JaccardCNFLQwen2.5-Coder	74	117	145	204.46	538.34	1.14 × 10−11	0.591
	JaccardCNFLCodeT5Plus	46	88	115	228.05	611.85	0.694	-
Russell_rao	Russell_rao	1	11	21	649.25	940.78	-	-
	Russell_raoCNFLInCoder	22	53	71	402.03	820.70	2.06 × 10−26	0.664
	Russell_raoCNFLCodeBERT	14	29	37	620.19	920.30	4.02 × 10−3	0.539
	Russell_raoCNFLDeepSeekCoder	20	42	63	437.19	862.90	2.35 × 10−24	0.608
	Russell_raoCNFLQwen2.5-Coder	23	53	65	526.05	889.36	1.64 × 10−11	0.583
	Russell_raoCNFLCodeT5Plus	6	18	25	641.94	934.12	0.31	-

Bold values indicate the best performance among the CNFL variants for each SBFL technique.

. To improve readability, the best experimental results are presented in bold. The results indicate that CNFL consistently demonstrates better effectiveness. For instance, the Top-1, Top-3, and Top-5 scores of Ochiai are 44, 95, and 126, respectively, while they are 69, 112, and 130 for the CNFL involving Qwen2.5-Coder (as shown in the 6th row of Table 4), gaining improvements of 56.8%, 18.9%, and 3.2% in terms of the Top-N metric. Meanwhile, the MFR decreases from 225.69 to 183.12, and the MAR decreases from 570.37 to 527.92, reflecting a clear improvement in the overall fault ranking. Similar performance gains are also observed for the DStar, Jaccard, and Russell_Rao baseline methods. Although the Opt2 baseline exhibits only marginal improvements in Top-1, Top-3, and Top-5 scores across all models, its integration with InCoder results in a substantial decrease in MFR (from 396.60 to 303.17) and in MAR (from 783.74 to 736.13), demonstrating that CNFL improves the ranking of faulty statements even when Top-N gains are limited. These results indicate that using code naturalness to optimize the coverage matrix can effectively enhance fault localization performance.

We then performed a statistical analysis to assess whether CNFL significantly improves effectiveness of fault localization compared to the corresponding SBFL methods. We perform a Wilcoxon Signed-Rank (WSR) test [43], which is a non-parametric method for evaluating the differences between paired measurements. The ranks of faulty statements are used to construct paired samples for statistical comparison. Specifically, each paired sample consists of a list of ranks obtained by an SBFL method across all subject programs and the corresponding ranks produced by CNFL. The test is performed using a one-tailed p-value at a significance level of 0.05. If the resulting $p<0.05$, it indicates that CNFL produces significantly lower ranks for faulty statements compared to the baseline, confirming its improved fault localization performance. Otherwise, when $p\ge 0.05$, the difference is not statistically significant, and CNFL does not exhibit a clear advantage over the compared baseline. A total of 25 WSR tests are conducted, and the corresponding p-values are presented in Table 4. Statistically significant differences are found in 15 of these tests. Furthermore, for each baseline method, at least one CNFL variant that shows statistically superior performance.

We further quantify the magnitude of the differences using the Vargha–Delaney A-test [44], for the 15 cases showing statistically significant differences in the WSR test. The value of A above 0.56 (or below 0.44) is regarded as a small effect, values exceeding 0.64 (or below 0.36) correspond to a medium effect and values above 0.71 (or below 0.29) indicate a large effect. The results are reported in the last column of Table 4, where - indicates that the A-test was not performed because the WSR test did not reach statistical significance. Overall, 11 comparisons exhibit small effect sizes, one comparison demonstrates a medium effect size, and three comparisons show negligible effect sizes.

The above results suggest that CNFL generally demonstrates better fault localization capability compared to the relevant SBFL approach. Nevertheless, we also observe performance variations among CNFL variants constructed with different LLMs for code under the same baseline method. This further suggests that LLMs for code under investigation exhibit varying performance for providing information to optimize the coverage matrix. Among all the evaluated models, the use of Qwen2.5-Coder consistently delivers significant improvements across all SBFL methods, while DeepSeekCoder, InCoder, and CodeBERT improve the performance of most SBFL methods.

We further compare CNFL with LLM-based methods. That is, we report the Top-N fault localization effectiveness by ranking statements with respect to their naturalness scores calculated by LLMs.

Table 5. Overall Top-N performance using code naturalness (calculated by LLMs evaluating code) as a standalone fault localization approach.

LLMs for Code	Top-1	Top-3	Top-5
CodeBERT	15	35	59
InCoder	11	19	28
DeepSeekCoder	9	23	34
CodeT5Plus	10	23	32
Qwen2.5-Coder	8	24	36

Bold values indicate the best performance for each metric among the evaluated LLMs.

reports the Top-1, Top-3, and Top-5 hit counts of the five LLMs for code across six projects of the Defects4J benchmark. It can be generally observed that code naturalness, when used as a standalone fault localization technique, fails to effectively pinpoint buggy statements. The best Top-1 performance is achieved by CodeBERT, yet it only assigns the highest score to faulty statement in 15 out of 395 programs. Even when considering the Top 5, the best-performing model only identifies buggy statements in 59 programs. These results indicate that while buggy statements are sometimes assigned the highest naturalness scores, they are often overshadowed by other non-faulty statements that the model also perceives as unnatural. By comparing the performance of CNFL with that LLM-based methods, it can be found that CNFL always have higher effectiveness in terms of Top 1, Top 3, and Top 5.

Answer to RQ1: CNFL has been demonstrated to be of better fault localization effectiveness than the baseline SBFL methods. Moreover, it is also superior to LLM-based methods that use code naturalness as a standalone indicator for fault localization.

6.2. RQ2: Effect of Model Capacities

We further analyze the impacts of the LLMs used for code naturalness evaluation. Specifically, we analyze whether the parameter scale of LLMs evaluating code significantly affects the performance improvement of CNFL relative to its baseline methods. In the experiment, CNFL is configured with Qwen2.5-Coder, since it achieves the best performance (as reported in Table 4).

To investigate whether different parameter scales of LLMs evaluating code significantly influence the performance of CNFL, we select four variants (0.5 B, 1.5 B, 3 B, and 7 B parameters) of Qwen2.5-Coder. Figure 2 presents a comprehensive evaluation of CNFL across different model scales and ranking formulas. Specifically, subfigures (a)–(c) report the Top-1, Top-3, and Top-5 localization performance achieved by CNFL using various parameter sizes of Qwen2.5-Coder in combination with different suspiciousness formulas, revealing the relationship between model scale and localization performance. Figure 3 further reports the computational overhead, presenting the time required by each Qwen2.5-Coder variant for code naturalness evaluation.

Figure 2 reports that Top-1, Top-3, and Top-5 values are generally improved with the increase in model scale. For instance, under the DStar formula, Top 1 rises from 59 with the 0.5 B model to 61 with the 1.5 B model, then to 63 with the 3 B model, and 65 with the 7 B model; Top 3 increases from 106 to 107 and then to 112; Top 5 grows from 124 to 125, 127, and 130. Similar trends are observed for the Ochiai, Jaccard, and Russell_Rao formulas. It is also notable that the total Top-1 improvement across the five formulas from 0.5 B to 3 B is 24, while the improvement from 3 B to 7 B is only 3. This indicates that the performance gains for CNFL from scaling the model from 0.5 B to 3 B are substantially larger than those from 3 B to 7 B.

It can be observed from Figure 3 that the computational overhead for calculating code naturalness increases substantially with the scale of the Qwen2.5-Coder model, approximately doubling as the model size increases. Taking the Closure project as an example, the time required to compute code naturalness for the 0.5 B model is approximately 8037 s (134.0 min), which rises to 14,404 s (240.1 min) for the 1.5 B model, 25,927 s (432.1 min) for the 3 B model, and reaches 45,731 s (762.2 min) for the 7 B model. These results indicate that excessively large model scales substantially increase the cost of code naturalness computation, thereby raising the overall fault localization overhead.

From an efficiency perspective, the 7 B model incurs a high computational cost, 954.3 min, nearly 1.75 times that of the 3 B model. Moreover, as shown in Figure 2, the 7 B model does not consistently improve fault localization effectiveness across all cases. Even when improvements are observed, they are only marginal. Therefore, when considering both effectiveness and efficiency, the 3 B model is a preferable choice to boost CNFL.

Answer to RQ2: Increasing the scale of model is beneficial to more accuracy code naturalness, and thus can help CNFL deliver better fault localization effectiveness. However, the scale of the model also affects the cost. These suggests that CNFL should be configured with the consideration of both fault localization effectiveness and time cost.

6.3. RQ3: Comparsion of CNFL with Other Weight-Based Fault Localization Methods

We then conduct a comparison analysis between CNFL and the other weight-optimization-based approaches. To this end, we choose ${\mathrm{CNFL}}_{Qwen2.5-Coder}$, which achieved the best performance in RQ1, as the representative CNFL method. We then compare it with two existing weight-optimized-based fault localization methods, WTCFL [32] and PFL [33], to systematically evaluate its relative effectiveness. To assess performance, we focus on the ranks of faulty statements, and conduct WSR tests at a 0.05 significance level. Specifically, for each group of data under investigation, we conduct three types of WSR tests, a one-tailed (right) test, a one-tailed (left) test, and a two-tailed test. For these tests, the p-values are interpreted based on the directional relationship of the rankings. The p-values of the one-tailed (left) test evaluates whether the ranks produced by CNFL are significantly lower than those of the baseline, where $p<0.05$ indicates that CNFL is statistically superior in localization effectiveness. Conversely, the p-values of the one-tailed (right) test assesses whether CNFL yields significantly higher ranks, implying a performance degradation. Finally, the p-values of the two-tailed test measures whether a significant discrepancy exists between the two sets of rankings regardless of the direction, where $p<0.05$ suggests that the two compared methods are not statistically equivalent.

Table 6. WSR tests for the proposed $CNF{L}_{Qwen2.5-Coder}$ versus $WTCFL$ across five SBFL techniques.

Wilcoxon Tests		Right-Tailed	Left-Tailed	Two-Tailed	Conclusion
CNFL vs. WTCFL	Ochiai	0.967	0.031	0.036	better
	Dstar	0.985	0.012	0.008	better
	Opt2	0.954	0.042	0.045	better
	Jaccard	0.992	6.41 × 10−3	1.20 × 10−4	better
	Russell_rao	0.998	1.12 × 10−4	2.24 × 10−5	better

and

Table 7. Wilcoxon tests for the $CNF{L}_{Qwen2.5-Coder}$ versus $PFL$ across five SBFL techniques.

Wilcoxon Tests		Right-Tailed	Left-Tailed	Two-Tailed	Conclusion
CNFL vs. PFL	Ochiai	0.991	0.038	0.042	better
	Dstar	0.982	0.034	0.041	better
	Opt2	0.951	0.049	0.048	better
	Jaccard	0.997	3.25 × 10−4	6.50 × 10−4	better
	Russell_rao	0.998	5.12 × 10−7	1.02 × 10−6	better

report the comparative statistical results of ${\mathrm{CNFL}}_{Qwen2.5-Coder}$ with WTCFL and PFL in five baseline SBFL methods. The comparison results show that CNFL consistently achieves better performance across all baselines. This indicates that, compared to WTCFL and PFL, CNFL ranks faulty statements higher. Taking WTCFL as an example, the p-values for Ochiai are 0.967, 0.031, and 0.036 for the right-tailed, left-tailed, and two-tailed tests, respectively. The rejection of the null hypothesis in the left-tailed and two-tailed tests ($p<0.05$) provides strong evidence that CNFL produces significantly more accurate rankings (i.e., lower ranks) than WTCFL. At the same time, the high p-value in the right-tailed test confirms that the naturalness-based refinement does not result in any statistically significant deterioration in performance. Therefore, from a statistical standpoint, CNFL demonstrates superior performance compared to WTCFL across all baseline methods.

Answer to RQ3: Our results show that the proposed approach, CNFL, is more effective than the other two baseline approaches, which also adopt matrix weight optimization strategies. These results further confirm the positive support from LLM-based code naturalness evaluation in software fault localization.

7. Threats to Validity

Potential threats to internal validity mainly stem from the implementation details of CNFL. Since CNFL enhances fault localization performance by adjusting the weights of statements in the coverage matrix, any logical errors or implementation biases may affect the experimental results. To mitigate this risk, we implemented CNFL as an extension of an established SBFL framework and carefully verified the correctness of key computational procedures. In addition, we consistently employed officially released pre-trained models in our experiments to ensure a standardized experimental environment and the reliability of the results.

A threats to external validity in this study primarily arise from the selection of LLMs for code and SBFL methods. Different model architectures, parameter scales, and risk evaluation formulas may influence the generalizability of the experimental findings. To mitigate this threat, we included multiple LLMs for code with diverse architectures and parameter sizes, along with five representative SBFL methods, to ensure the robustness and representativeness of our conclusions.

Another external validity threat of this study lies in the use of the Defects4J dataset exclusively. Since most state-of-the-art fault localization techniques have been evaluated and compared on this benchmark, we selected Defects4J to ensure the comparability and reproducibility of our results. However, this choice may restrict the generalizability of our findings to other datasets. Moreover, code naturalness is a phenomenon observed on diverse programming languages. The optimal configuration of CNFL (e.g., naturalness threshold, model size) may vary across different languages. Future work will replicate our experiments on C/C++ benchmarks (e.g., ManyBugs) and Python benchmarks (e.g., BugsInPy) to confirm the generalizability of CNFL. In future work, we plan to further assess the applicability and robustness of our approach on more diverse benchmarks, such as Bugs.jar and ManyBugs.

8. Conclusions

In this paper, we introduced CNFL, a novel fault localization approach based on code naturalness. This method leverages code LLMs to quantify the naturalness of code statements and uses these scores to adjust the weights of statements within the coverage matrix. The suspiciousness of statements is then computed using standard risk evaluation formulas. To evaluate the effectiveness of CNFL, we select five representative SBFL formulas and employ multiple LLMs covering three different architectural types to quantify the naturalness of code statements. In addition, we also introduce a standalone LLM-based code naturalness fault localization method and two state-of-the-art methods adopting coverage matrix optimization strategies as comparison baselines. We then conduct extensive experiments on 395 faulty versions across six real-world software projects. The experimental results demonstrate that CNFL significantly outperforms traditional SBFL baselines as well as the standalone LLM-based fault localization approach. Specifically, when applied to classic SBFL formulas such as Jaccard and Ochiai, CNFL improves the Top-1 hit rate by 60.8% and 56.8%, respectively. In terms of absolute performance, the best Top-1 result (74 cases) achieved by CNFL is approximately 4.9 times higher than that of the LLM-based method (15 cases). Furthermore, CNFL is also more effective than the state-of-the-art fault localization methods adopting weight matrix optimization. We also investigate the impact of model parameter scale on the localization performance of CNFL. We conduct comparative experiments using four Qwen2.5-Coder variants with different parameter sizes. The results indicate that CNFL’s localization performance generally improves as the model scale increases; however, the computational overhead also rises substantially with larger model sizes.

In the future, we plan to expand the scope of our study by incorporating diverse LLMs and SBFL formulas to further validate the generalizability of the proposed approach. Moreover, we will also explore the capability of the proposed approach for fault localization in concurrent and cooperative programs. Another promising direction for future work is to integrate the proposed approach with existing matrix-optimization-based techniques, thereby establishing a hybrid paradigm for coverage matrix optimization that can further improve the effectiveness of fault localization.