A Logic-Guided and Explainable Approach to LLM-Based Unit Test Generation

Abstract

Large Language Models (LLMs) have demonstrated considerable potential in automated unit test generation; however, most existing approaches rely on a black-box paradigm that directly maps code under test to test code, often resulting in low compilation success rates, limited branch coverage, high assertion failure rates, and poor interpretability. Inspired by the human process of developing test cases, this paper proposes Logic-CoT, a white-box generation paradigm that follows a code under test–logical reasoning–test code workflow. The proposed approach consists of three stages: in the logical inference stage, logical node state vectors and execution paths are constructed from the control flow graph of the code under test, and input values and oracles satisfying state constraints are derived; in the test case construction stage, a template-based method is used to initialize test code conforming to the Arrange–Act–Assert pattern, with test intentions explicitly documented as comments; in the repair stage, syntactic errors and assertion failures are handled in a layered manner, where the former are corrected without altering test logic and the latter trigger logic reflection based on discrepancies between expected and actual outcomes, leading to state updates and test case reconstruction. This design forms a closed-loop process of reasoning, generation, and repair. Experiments on the QuixBugs, Apache Commons, HumanEval, and SV-COMP benchmarks show that Logic-CoT consistently outperforms state-of-the-art approaches such as ChatUniTest in terms of compilation success rate, runtime pass rate, assertion pass rate, branch coverage, average repair iterations for faulty code, and interpretability. Ablation studies further demonstrate that each component of Logic-CoT contributes effectively to improving the overall quality and effectiveness of generated test cases. These results indicate that Logic-CoT improves the reliability and interpretability of LLM-generated unit tests in practical software testing scenarios.

1. Introduction

Unit testing is a fundamental technique for ensuring software quality, and the automated generation of unit tests can substantially improve development efficiency. Consequently, it has attracted significant attention from both academia and industry. The primary research objectives in this area are to enhance code coverage and assertion accuracy, while the core challenges lie in test oracle construction and constraint solving.

Traditional approaches predominantly rely on Search-Based Software Testing (SBST). Representative tools include EvoSuite [1], which is based on genetic algorithms, and Randoop [2], which adopts random testing. Although these tools achieve high coverage, they suffer from several limitations, such as poor test case interpretability, low-quality assertions, and difficulties in constructing complex objects. For example, variables are often named generically (e.g., var0, var1), which discourages human developers from maintaining the generated tests. As a result, test cases tend to become disposable artifacts. The fundamental cause of these issues is the lack of semantic understanding, leading to test cases that fail to capture explicit design intent.

In recent years, Large Language Models (LLMs) have demonstrated strong capabilities in code generation and semantic understanding, motivating their application to automated test generation. By enriching contextual information and adopting a generate–validate–repair mechanism, ChatUniTest has emerged as a representative LLM-based test generation approach, achieving improvements in reducing compilation errors and hallucinated API calls. Nevertheless, it still exhibits notable limitations: (1) coverage improvement is largely driven by the stochasticity of LLMs, without explicit path-level reasoning; (2) assertions are often shallow or directly copied from the code under test, which can easily lead to false positives [3]; (3) test intent is not explicitly expressed, resulting in limited interpretability.

These limitations highlight the need for practical advances in software testing that improve test reliability, oracle quality, and interpretability, rather than relying solely on coverage-driven or stochastic generation.

Inspired by the process by which humans develop test cases, this paper proposes a Chain-of-Thought (CoT)-based test generation framework, termed Logic-CoT. Unlike black-box “Code → Code” approaches, Logic-CoT adopts a white-box paradigm of “Code → Logic → Code”, transforming test case generation from implicit probabilistic composition into explicit logical reasoning. In the logical inference stage, Logic-CoT constructs the node state vector Condition and execution path Path based on the control flow graph. These logical representations are then instantiated into concrete input values Input within the input space, along with corresponding test oracles Expectation. During the test case construction stage, explanatory comments are embedded to make the testing intent self-interpretable. Furthermore, a feedback–repair strategy is designed to address syntax errors and assertion failures. Syntax errors are fed back to the test construction stage, preserving the underlying test logic while correcting code-level syntax. Assertion failures, in contrast, are propagated back to the logical inference stage, triggering logical reflection, updating node states, and reconstructing test cases. In this way, Logic-CoT establishes a closed-loop mechanism of “reasoning–generation–repair”.

The main contributions of this paper are summarized as follows:

(1)

We propose a structured, Chain-of-Thought-based paradigm for test case generation. By simulating the human test development process through state enumeration–test construction–layered repair, the generation process and test intent are made explicit.

(2)

We introduce a self-explanatory test case construction method. By leveraging template-based code generation, execution paths, logical states, and test intent are embedded into comments, enabling documentation-style and self-interpretable test cases.

(3)

We propose a layered error repair mechanism that differentiates between compilation and runtime errors. Syntactic defects are repaired while preserving test logic, whereas assertion failures guide the model to backtrack and reflect on logical inconsistencies, update logical states, and reconstruct test cases. Unlike ChatUniTest, which employs a unified code repair strategy, Logic-CoT adopts distinct repair strategies for syntactic errors and assertion failures.

The remainder of this paper is organized as follows. Section 2 reviews related work. Section 3 presents the details of Logic-CoT. Section 4 describes the experimental setup, followed by result analysis and discussion in Section 5. Finally, Section 6 concludes the paper and outlines future research directions.

2. Related Work

The evolution of automated unit test generation can be broadly characterized by a progression from traditional approaches based on search and symbolic analysis, through statistical methods driven by deep learning, to the current generation of LLM-based generative techniques.

2.1. Traditional Test Generation Methods

Traditional approaches to automated unit test generation primarily rely on program structure analysis and heuristic search algorithms. Search-Based Software Testing (SBST) formulates test generation as an optimization problem. A representative tool, EvoSuite, employs genetic algorithms to evolve a population of test cases through mutation and crossover, with the objective of maximizing code coverage. Although EvoSuite achieves strong performance in terms of coverage, the generated test code often lacks readability, containing numerous semantically meaningless identifiers and hard-to-understand assertion logic, which poses significant challenges for long-term maintenance. More importantly, while EvoSuite implicitly explores the path space, it does not explicitly represent path conditions or combinations of Boolean states. As a result, the generated tests lack logical traceability and cannot guarantee a clear mapping between test inputs and path conditions [4].

Symbolic execution-based approaches, such as KLEE [5], simulate program execution using symbolic values and leverage constraint solvers to compute path constraints, thereby generating test cases with high coverage. However, the well-known path explosion problem and the difficulty of solving constraints for complex objects significantly limit their applicability. Moreover, although symbolic execution is grounded in precise logical constraint solving, the derived path constraints are not exposed to users. Consequently, the generated test cases lack interpretability in terms of the underlying derivation process and still fail to convey readable test design rationale.

In addition, early deep learning-based methods, such as ATLAS [6], attempt to learn a direct mapping from functional code to test code using Seq2Seq models. Nevertheless, these approaches do not explicitly model the intrinsic structure of test logic, such as path conditions and boundary values. As a result, the generated test code often suffers from syntactic errors or unreasonable assertions.

2.2. LLM-Based Test Generation Methods

Large language models, such as Gemini, Claude, Grok, GPT, DeepSeek, and Qwen, have introduced a new research paradigm by virtue of their strong semantic understanding and code generation capabilities. According to their underlying technical routes, existing studies can be broadly categorized into three groups:

• Prompt Engineering and Context Enhancement Methods

This category of approaches focuses on designing effective prompts to enable LLMs to generate higher-quality test code under zero-shot or few-shot settings. TeCO [7] enhances the prompt context by retrieving similar code fragments and dependency information, while TestGen-LLM [8] employs retrieval-augmented generation (RAG) to provide LLMs with structured contextual knowledge and API specifications.

However, since the generation process is driven by probabilistic token completion rather than explicit logical constraint solving, these approaches exhibit unstable performance on tasks requiring complex logical reasoning [9]. When programs involve multiple conditional branches or boundary-value scenarios, the generated tests tend to focus on the most likely main execution paths, while low-frequency yet critical exceptional or boundary cases are often overlooked, resulting in insufficient logical coverage.

2.

Iterative Repair and Feedback Methods

To mitigate syntactic and runtime errors in LLM-generated code, researchers have proposed a series of closed-loop mechanisms based on a generate–validate–repair paradigm. ChatUniTest [10] is a representative work in this line, while Libro [11] and TestPilot [3] propose similar automated repair frameworks.

However, the repair strategies adopted by these approaches primarily target code-level errors rather than logic-level errors. When assertion failures occur, they typically adjust test inputs to accommodate the program behavior, instead of examining whether the original logical reasoning or inferred test intent is flawed. As a result, assertion correctness is improved in a superficial manner, while logical soundness remains unguaranteed.

3.

Hybrid Methods

Hybrid approaches aim to combine the coverage advantages of traditional techniques with the semantic strengths of LLMs. For example, Codamosa [12] invokes LLMs to generate more diverse test inputs when EvoSuite’s search process becomes trapped in local optima, while Almasi et al. [13] leverage LLMs to improve the readability of automatically generated tests. These methods primarily function as post-processing or auxiliary enhancements and do not explicitly incorporate a logical reasoning model into the test generation process.

In summary, although most automated approaches are capable of generating executable test cases, they fail to explicitly model the underlying logical reasoning chain, which fundamentally limits test interpretability and logical completeness. Recent systematic empirical studies on LLM-based code and test generation have further revealed pervasive issues of missing logic and opaque reasoning processes. These studies indicate that the absence of explicit path-condition modeling and interpretable reasoning structures constitutes one of the core bottlenecks faced by current generative approaches [14,15,16,17].

3. Logic-CoT Framework

3.1. Overview

Classical software testing theory indicates that human-designed test cases are not the result of random behaviors, but are instead driven by a systematic logical reasoning chain, which typically includes: input space reasoning, such as equivalence class partitioning and boundary value analysis; Boolean condition reasoning, which maps inputs to the truth values of decision nodes; path reasoning, which combines Boolean vectors to form execution paths and verifies their satisfiability; and oracle reasoning, which derives expected outputs based on path semantics [18,19].

Test generation is essentially a logical reasoning activity, relying on systematic reasoning steps such as input space partitioning, condition analysis, path derivation, and oracle construction [18,19].

These steps can be formalized as:

$$\left(\mathrm{Input},\mathrm{Condition},\mathrm{Path},\mathrm{Expectation}\right)={\mathrm{f}}_{\mathrm{logic}}\left(\mathrm{Program}\right)$$

where ${\mathrm{f}}_{\mathrm{logic}}\left(\mathrm{Program}\right)$ denotes the explicit logical reasoning process employed by humans when designing test cases, establishing an interpretable mapping among test inputs, logical conditions, execution paths, and expected outputs. Consequently, unit test generation can be viewed as the process of deriving a set of state vectors that satisfy logical constraints from program logic, and constructing test cases accordingly.

Based on this insight, this paper proposes Logic-CoT, a logic-driven and interpretable test generation framework that completes the full logical reasoning chain—from input reasoning to logical node states, and from path reasoning to test oracle construction. By explicitly building logical node state tables, documenting test intent, and introducing a layered self-healing mechanism, Logic-CoT elevates test generation from conventional probabilistic generation to structured reasoning, thereby enhancing both logical coverage and interpretability. The overall architecture of Logic-CoT is illustrated in Figure 1.

Figure 1. Logic-CoT Architecture Diagram.

Logic-CoT abandons the traditional end-to-end Code-to-Code generation paradigm and instead adopts a Chain-of-Thought (CoT) architecture following a Code–Logic–Code workflow. The framework consists of three core phases:

• Logic Deduction Phase (LDP): Intermediate representations are constructed through Chain-of-Thought reasoning, and the analysis results are recorded in a state table, which serves as the core of the framework’s interpretable documentation.
• Test Case Construction Phase (TCP): The logical analysis results are mapped to self-explanatory test scripts based on predefined test script templates.
• Hierarchical Self-Healing Phase (HSP): A dual-loop, error-type–aware feedback mechanism is employed, where the inner loop handles syntactic errors, and the outer loop reflects on assertion failures.

To clearly present the overall methodology of Logic-CoT, the following three sections introduce each phase with the support of formal modeling, outlining both the core ideas and the corresponding implementation mechanisms. Specifically, for each phase, we first provide a formal task definition and constraint specification to clarify its role within the overall workflow. Subsequently, the proposed ideas are realized through a combination of prompt engineering and large language models (LLMs).

In each subsection, we present the corresponding prompt designs, and for the first two phases, we provide structured examples to illustrate how the formal definitions are instantiated into concrete, structured artifacts, including intermediate representations and final generation results.

Overall, the prompt design follows a phase-aligned principle, ensuring that logic deduction, test case construction, and hierarchical self-healing are executed in isolated phases. This design avoids instability caused by mixing heterogeneous tasks within a single interaction. To accommodate different units under test and engineering environments, contextual information—such as the focal method, class name, dependency information, and runtime error messages—is injected into prompts via template variables, including {{focal_method}}, {{class_name}}, {{information}}, {{error_type}}, and {{error_message}}, enabling systematic generation. These contextual elements are extracted either through Abstract Syntax Tree (AST) analysis of the focal method or via regular-expression matching on compilation and runtime error outputs.

3.2. Phase I: Logic Deduction via State Tables

3.2.1. Phase Objective and Formal Modeling

The objective of the Logic Deduction Phase (LDP) is to make the implicit control flow and semantic constraints of the code under test explicit, and to construct a decision table that satisfies the Decision Coverage criterion. This decision table serves as the core intermediate representation of Logic-CoT and is directly consumed by the subsequent test case construction phase. From a modeling perspective, this phase can be abstracted as a Constraint Satisfaction Problem (CSP) constrained by program logic, where the solution is a set of logically reachable state vectors under the program semantics.

Given the source code of the method under test $C_{\mathrm{in}}$, the system first leverages an LLM to identify a set of decision nodes $N=\{n_1,n_2,\dots ,n_k\}$ including conditional branches, loop predicates, and exception-throwing points. Each decision node corresponds to a Boolean domain $\{\top ,\perp \}$. Based on these nodes, the logic deduction phase enumerates logical paths formed by combinations of decision node outcomes. Decision coverage and path reachability constraints are enforced to ensure that each decision condition evaluates to both true and false at least once across all logically reachable paths. The reachability of each logical path is self-validated by the LLM.

For each logically reachable path $p_i$, the system constructs a corresponding logical state vector ${\overrightarrow{\sigma }}_i$, which characterizes the Boolean assignments of all decision nodes along that path. Subsequently, Logic-CoT guides the LLM to infer the function intention of the method under test by analyzing its source code, expressing this intention in natural language as a high-level semantic constraint. Under a fixed path structure, the expected behavior of each logical path is then derived by jointly considering the code logic and the inferred function intention. Finally, each path is materialized as a state record $R_i=⟨{\mathrm{path}}_i,{\mathrm{input}}_i,{\mathrm{expected}}_i⟩$ where ${\mathrm{input}}_i$ denotes the input conditions satisfying the path constraints, and ${\mathrm{expected}}_i$ denotes the expected return value or exception type derived strictly from program logic and function intention.

All state records collectively form the decision table $\Sigma$, which systematically characterizes the execution paths and corresponding expected behaviors of the method under different logical condition combinations. As the final output of the logic deduction phase, this decision table is passed to the subsequent test case construction phase to generate executable unit test code. Overall, the logic deduction phase can be abstracted as a mapping from source code to a decision table:

$$\Sigma ={\Phi }_{\mathrm{deduce}}\left(C_{\mathrm{in}}\right)$$

where ${\Phi }_{\mathrm{deduce}}$ denotes the logic deduction function constrained by decision coverage and path reachability, and is concretely implemented via LLM inference.

3.2.2. Constraint Definition of the Logic Deduction Phase

The logic deduction phase is governed by the following constraints:

• All conditional branches, loop predicates, and exception-throwing points in the method under test must be identified as decision nodes.
• For each decision node, both its true and false outcomes must be covered at least once across all logically reachable paths, satisfying the decision coverage criterion.
• Only logically satisfiable combinations of decision node outcomes are permitted to form valid execution paths; unreachable combinations must be excluded.
• For each execution path, a state record $R_i=⟨{\mathrm{path}}_i,{\mathrm{input}}_i,{\mathrm{expected}}_i⟩$ must be constructed, where the expected behavior is derived strictly from code logic, function intention, and input states, without speculation or heuristic guessing.

3.2.3. Prompt-Based Constraint Enforcement Mechanism

To realize the above formal task definition and constraint specifications, a carefully designed prompt is employed, as illustrated in Figure 2. This phase enforces the generation of structured JSON outputs, ensuring that the deduced logical information is machine-parsable and can be directly consumed as logical input by subsequent phases. The complete prompt templates for each stage have been publicly released in plain text format and are available in the project repository at https://github.com/WILL997/logic-cot/tree/main/prompt (accessed on 1 March 2026).

Figure 2. Prompt words for the logical deduction stage.

In the prompt, the variables {{focal_method}}, {{class_name}}, and {{information}} are obtained through Abstract Syntax Tree (AST) analysis of the program under test. The variable {{functionIntention}} denotes the inferred function intention, which is derived by the large language model based on the method’s source code. The variable {{inputData}} consists of decision conditions, logical paths, and their corresponding input combinations, serving as the foundation for constructing the decision table.

To illustrate the output format of the logic deduction phase in an intuitive manner, we present a structured example of a decision table (Figure 3). This example is independent of any specific method under test and is intended solely to demonstrate the field structure and constraint specifications produced by the logic deduction phase. Each row of the decision table corresponds to a distinct logical path, explicitly recording the decision conditions, path assignments, input values, and their associated expected behaviors.

Figure 3. Structured Example of the Logical Deduction Stage.

3.3. Phase II: Explainable Test Case Construction

3.3.1. Objective and Formal Modeling of the Interpretable Test Case Construction Phase

The objective of the Test Case Construction Phase (TCP) is to instantiate the decision table generated in the logic deduction phase into executable unit test code. Its core responsibility is to establish a deterministic mapping from logical state records to test scripts, thereby constructing test cases with explicit interpretability and traceability. By explicitly injecting path, input, and expected-behavior information from the decision table, this phase ensures that the generated tests faithfully reflect the logical reasoning results.

Specifically, the test case construction phase takes as input the focal class signature, the source code of the focal method, dependency information, and the decision table $\Sigma$. Each state record in the decision table,

$$R_i=⟨{\mathrm{path}}_i,{\mathrm{input}}_i,{\mathrm{expected}}_i⟩$$

corresponds to an independent test scenario and is mapped one-to-one to a unit test case. Throughout this mapping process, the input conditions and expected results defined in the decision table are preserved without modification, ensuring that the generated test cases strictly adhere to the analytical conclusions produced in the logic deduction phase.

To guarantee consistency and maintainability of the generated tests across different project environments, the system adopts a standardized XUnit-based test template. Each test method follows the Arrange–Act–Assert (AAA) pattern: in the Arrange phase, variables are initialized or necessary test states are constructed according to ${\mathrm{input}}_i$; in the Act phase, the method under test is invoked; and in the Assert phase, assertions are generated based on ${\mathrm{expected}}_i$, supporting both return-value validation and exception-based validation. Furthermore, to enhance readability and traceability, state information from the decision table is explicitly injected into structured comments within the test script. Each generated test case includes Javadoc-style annotations that describe the test intent (@Intent), the covered logical path (@Path), and the corresponding input and expected outcome (@State). In this way, the generated test cases serve not only as verification artifacts but also as engineering documentation that maintains a one-to-one correspondence with the decision table, clearly explaining why a test case is constructed, which path it covers, and what inputs and expected results are involved.

3.3.2. Test Case Construction Mechanism

During the test case construction phase, each state record $R_i$ is instantiated into unit test code according to fixed template-filling rules. Specifically, ${\mathrm{input}}_i$ is used exclusively for constructing the test setup within the test method, while ${\mathrm{expected}}_i$ is used solely for assertion generation. When the expected outcome is a return value, a return-value assertion is generated; when the expected outcome corresponds to an exception, an exception assertion is produced. The path identifier ${\mathrm{path}}_i$ is retained as metadata in the structured comments of the test case.

All test methods adopt a uniform Arrange–Act–Assert (AAA) code skeleton to ensure structural consistency across generated tests. These construction rules are operationalized through prompt templates and executed by the large language model.

3.3.3. Template- and Prompt-Based Test Case Construction

To implement the above formal task definitions and constraint specifications, a dedicated prompt template is designed, as illustrated in Figure 4.

In the prompt, the variables {{focal_method}}, {{class_name}}, and {{information}} are obtained through Abstract Syntax Tree (AST) analysis of the program under test. The variable {{judgment_table}} denotes the decision table $\Sigma$ generated in the logic deduction phase and serves as the core input to the test case construction phase.

To intuitively demonstrate the output format of the Test Case Construction Phase (TCP), we present a structured example (Figure 5). This example is independent of any specific method under test and is intended solely to illustrate how a decision table record $R_i$ is mapped to a JUnit 5 + Mockito 3 test class. Each state record corresponds to an individual test method and is enforced to include a metadata annotation block and a standardized Arrange–Act–Assert (AAA) code skeleton.

Figure 4. Prompts for the use case construction phase.

Figure 4. Prompts for the use case construction phase.

3.4. Phase III: Hierarchical Self-Healing Mechanism

3.4.1. Hierarchical Self-Healing Process and State Modeling

The Hierarchical Self-Healing Phase (HSP) is a closed-loop feedback process that integrates state judgment and dynamic correction. Its objective is to select differentiated repair strategies according to error types when test execution fails, and to gradually converge to passing test cases through bounded iterations. The test repair process can be formalized as a sequence of state transitions, which are divided into two levels based on error characteristics: an inner loop for syntactic repair and an outer loop for logical reflection.

Formally, the system state at iteration t is defined as:

$${\mathrm{State}}_t=(S_t,L_t),$$

where $S_t$ denotes the current test script and $L_t$ denotes its corresponding logical representation, i.e., the Javadoc-style structured comments associated with the test script. The execution result of a test script is determined by an execution judgment function:

$$r_t=\mathrm{Exec}\left(S_t\right)\in \{\mathrm{Pass},{\mathrm{Err}}_{\mathrm{syn}},{\mathrm{Fail}}_{\mathrm{sem}}\},$$

where Exec is a decision operator that maps a test script to a discrete execution state, abstracting the runtime behavior of the test script in the execution environment. Its implementation is based on the execution results produced by the Java Virtual Machine (JVM), where runtime outputs and exception information are analyzed via regular-expression matching to determine the corresponding execution state. When $r_t={\mathrm{Err}}_{\mathrm{syn}}$, the system enters the inner-loop syntactic repair process. Such errors stem from engineering defects in the test script; therefore, the repair operation is restricted to the code level, while the logical representation remains unchanged:

$$L_{t+1}\equiv L_t$$

In this process, the system performs localized repairs on $S_t$ based on stack trace information, while explicitly constraining the large language model from modifying test inputs, test intent, or expected outcomes. This design enforces logical invariance during syntactic repair.

Figure 5. Structured Example of the Use Case Construction Phase.

Figure 5. Structured Example of the Use Case Construction Phase.

When $r_t={\mathrm{Fail}}_{\mathrm{sem}}$, the system enters the outer-loop logical reflection procedure. An assertion failure indicates a mismatch between the expected behavior encoded in the current test script and the observed execution outcome. In this setting, while keeping the execution path and input state unchanged, the system re-derives the expected behavior represented in $L_t$ and propagates the updated expected results back into the test script. Importantly, updates in the logical reflection stage are grounded in the source-code semantics and the inferred functional intent, rather than the program’s runtime outputs, thereby preventing the repair process from overfitting to potentially erroneous behaviors. This yields a new test script:

$$L_{t+1}\ne L_t,S_{t+1}\leftarrow \mathrm{Construct}\left(L_{t+1}\right)$$

where $\mathrm{Construct}$ denotes a template- and structure-constrained test construction function that maps the updated logical representation $L_{t+1}$ to the corresponding test script, synchronizing only expectation-related modifications. This mechanism ensures that code changes are always driven by logical updates, rather than passively adapting tests to observed behaviors.

The repair process terminates when the test passes or when a predefined maximum number of repair iterations is reached. By explicitly distinguishing syntactic errors from semantic errors and enforcing logical invariance and logical causality constraints, respectively, the hierarchical self-healing phase achieves precise control over repair behaviors and significantly improves repair efficiency.

3.4.2. Prompt-Based Implementation of Hierarchical Self-Healing

To implement the state judgment and repair workflow defined in the hierarchical self-healing phase, dedicated prompt templates are designed to execute different repair strategies. The error type and associated runtime information produced during test execution are injected into the prompt as the sole driver for selecting the repair path by the large language model.

In the syntactic repair scenario, the prompt provides the current test script and error information to address engineering-level issues such as syntax errors, missing imports, or symbol resolution failures. In the logical reflection scenario, the prompt constrains the model to preserve the structured test annotations while guiding it to reflect on and reconstruct the assertion logic. By strictly limiting both the input content and the permissible modification scope of the prompts, precise control over repair behaviors is achieved. The corresponding prompt templates are illustrated in Figure 6.

Figure 6. Tips for the layered self-healing stage.

In the prompt, the variables {{method_name}} and {{class_name}} are obtained through Abstract Syntax Tree (AST) analysis of the program under test. The variables {{error_type}} and {{error_message}} are populated with runtime results automatically captured during test execution. The variables {{unit_test}} and {{method_code}} denote the current test script to be repaired and the source code of the method under test, respectively.

Algorithm 1 illustrates the overall workflow of Logic-CoT, encompassing the three major phases of logic deduction, structured test construction, and hierarchical self-healing.    

Algorithm 1: Logic-CoT: Code–Logic–Code Testing Framework

4. Experimental Setup

4.1. Experimental Environment

All experiments were conducted under identical hardware and software environments, using the same large language model and consistent parameter configurations. Detailed model specifications and parameter settings are provided in the baseline comparison section. To support reproducibility, all experimental code used in this study has been publicly released on GitHub at https://github.com/WILL997/logic-cot (accessed on 1 March 2026). The experimental hardware and software environment is summarized in Table 1.

Table 1. Experimental environment.

Item	Configuration
CPU	AMD Ryzen 7 8845H (8 cores, 16 threads)
Manufacturer	Advanced Micro Devices (AMD), Santa Clara, CA, USA
Memory	24 GB
Storage	NVMe SSD
Operating System	Ubuntu 22.04.5 LTS (64-bit)
GPU	NVIDIA GeForce RTX 4060 Laptop GPU
JDK	OpenJDK 17
Test Framework	JUnit 5, Mockito 3
Coverage Tool	JaCoCo 0.8.8

4.2. Research Questions

To systematically evaluate the effectiveness of Logic-CoT in automated unit test generation, this study formulates five research questions along three dimensions: test-case effectiveness, repair efficiency, and test-code interpretability.

RQ1

Effectiveness: Compared with baseline methods, does Logic-CoT achieve superior test-case effectiveness on the dataset?

RQ2

Efficiency of Repair: Does Logic-CoT significantly improve the efficiency of test-code repair and thereby reduce the overall cost of test-case generation?

RQ3

Interpretability: From the perspective of human developers, does the test code generated by Logic-CoT exhibit better interpretability?

RQ4

Component Contribution (Ablation Study): To what extent do the key components of Logic-CoT—state-table-based logical derivation (LDP), the test-case construction procedure (TCP), and the hierarchical dual-loop self-healing mechanism (HSP)—contribute to improvements in test-case effectiveness?

RQ5

SBST Comparison: Compared with the classical coverage-driven test generation tool EvoSuite, how does Logic-CoT perform in terms of branch coverage, test-intent expression, logical traceability, and maintainability?

4.3. Dataset Construction: A Multi-Spectrum Strategy

To ensure the generality and fairness of the evaluation results, this study adopts a multi-spectrum dataset composed of real-world defective programs from QuixBugs [20], industrial-scale applications from Apache Commons Math [21] and Apache Commons Lang [22], the LLM code generation benchmark HumanEval-X [23], and programs from the formal verification competition SV-COMP [24]. In total, the dataset comprises 200 programs, with its detailed composition summarized in Table 2.

Table 2. Dataset Composition.

Source	Category	Domain	Size	Max LOC	Avg LOC	Max CC	Avg CC	Rationale
QuixBugs (Java)	Real-world Bugs	Algorithmic	40	120	48	18	8.1	Classic algorithmic programs with known bugs and corrected versions, used to evaluate defect-revealing ability and test pass rate under non-trivial logic.
Apache Commons (Math, Lang)	Industrial Library	Numerical, String	50	260	92	22	7.6	Widely used Java libraries featuring numerical computation and string processing, representing industrial-grade code with diverse boundary conditions.
HumanEval-X (Java)	Generation Benchmark	Algorithmic, Reasoning	80	85	31	9	3.8	Standard LLM code-generation benchmark covering algorithms, string manipulation, and arithmetic reasoning, used to evaluate general reasoning ability.
SV-COMP (ReachSafety)	Formal Verification Tasks	Verification	30	180	74	28	13.9	Verification benchmarks featuring deep loops, recursion, and nondeterministic branches, requiring complex path reasoning.

Given that this study focuses on the impact of logical reasoning on the quality of unit test generation, methods under test from all data sources are selected according to a unified and reproducible set of criteria. Specifically, each method must not depend on third-party libraries (except for the JDK), must take inputs and produce outputs limited to JDK primitive types (e.g., int, double, boolean) or String, must have public visibility, and must return a value.

To analyze the applicability of the proposed approach under varying levels of logical complexity, the selected methods are further stratified based on Cyclomatic Complexity (CC), while their Lines of Code (LOC) are also recorded. Here, LOC denotes the number of effective lines of code in the method, excluding comments and blank lines. Cyclomatic complexity is used to characterize the structural complexity of control flow within a method and is categorized into three levels: low complexity (1–4), medium complexity (5–9), and high complexity (≥10). Through this stratification, the experimental dataset is ensured to cover methods of low, medium, and high complexity, with high-complexity methods being well represented to align with the study’s focus on complex logical reasoning capability.

4.4. Baselines

To demonstrate the superiority of the proposed approach, we select three representative baselines for comparison: the Native LLM, ChatUniTest, and EvoSuite. To control for potential performance variations caused by differences in underlying language models, all LLM-based methods are implemented on the same backbone model, Qwen-plus [25], with the temperature parameter fixed at 0.2.

Native LLM: This baseline employs a zero-shot prompt, “Generate unit tests for the following code”, without any Chain-of-Thought guidance or explicit template constraints. It serves as the lower-bound reference for automated test generation, allowing us to quantify the net improvement achieved by Logic-CoT.

ChatUniTest: We adopt its official open-source implementation, which represents one of the most advanced LLM-based unit test generation approaches to date. This comparison highlights the distinction between context-driven generation with a single-loop feedback mechanism and logic-guided generation with a hierarchical dual-loop feedback mechanism.

EvoSuite: As a representative traditional test generation tool, EvoSuite is included to contrast Logic-CoT with classical coverage-driven methods, particularly in terms of branch coverage, test-intent articulation, logical traceability, and maintenance convenience.

4.5. Evaluation Metrics

A combination of quantitative and qualitative metrics is adopted to evaluate test case quality, coverage, repair efficiency, and interpretability. Specifically, the following metrics are used:

Compilation Rate (CR): The proportion of generated test classes that can be successfully compiled by the Java compiler. This metric represents the fundamental requirement for the usability of generated test code.

Runtime Pass Rate (RPR): Given successful compilation, the proportion of test cases that execute successfully without runtime exceptions. This metric reflects the correctness of generated tests in terms of input construction and execution flow.

Assertion Pass Rate (APR): Given successful compilation, the proportion of test cases that execute successfully and satisfy all assertions. This metric reflects the accuracy of the model’s understanding of program semantics and business logic.

Branch Coverage (BC): Branch coverage is collected using JaCoCo. This metric measures test completeness and is particularly critical for codebases with dense logical structures, such as SV-COMP and Apache Commons Math, where BC serves as a key indicator of model effectiveness.

Average Repair Turns (ART): The average number of interaction rounds required by the model from initial test generation to test passing (or reaching the maximum iteration limit). This metric is used to assess the efficiency of the test repair process.

Interpretability Score: A double-blind human study is conducted with 30 participants who have more than three years of Java development experience (senior undergraduates or graduate students). Test code generated by different methods is randomly sampled and evaluated along three dimensions: intent clarity, logical traceability, and maintainability. A five-point Likert scale is adopted, where 1 indicates very poor and 5 indicates excellent. This metric emphasizes the cognitive cost of understanding test code and developers’ willingness to maintain it.

Mutation testing (e.g., PIT) was not incorporated as an evaluation metric in this study. A comprehensive analysis of mutation-based assessment will be investigated in future work.

4.6. Experimental Results

4.6.1. RQ1: Test Effectiveness

To answer RQ1, we conduct a systematic comparison among three representative test generation strategies: Native LLM, which does not incorporate any validation or repair mechanisms; ChatUniTest, which adopts context-enhanced generation with a single-loop feedback mechanism; and Logic-CoT, which integrates logic deduction with a hierarchical dual-loop self-healing mechanism. The experimental results are summarized in Table 3.

The three approaches exhibit a clear performance hierarchy in terms of test generation quality, following the order Native LLM < ChatUniTest < Logic-CoT, indicating that effective feedback and repair mechanisms play a decisive role in improving the pass rates of generated test code.

The Native LLM achieves the lowest compilation rates across all datasets. With feedback-based error repair, ChatUniTest improves the average compilation rate to 72.29%, while Logic-CoT further increases it to 91.22%. This improvement is particularly pronounced on datasets containing complex boundary conditions and multiple decision logics, such as Apache Commons Math and SV-COMP.

Table 3. Experimental results of RQ1.

Project	Method	Compilation Pass (%)	Runtime Pass (%)	Assertion Pass (%)	Branch Coverage (%)
HumanEval-X	Native LLM	32.50	60.40	55.10	30.20
	ChatUniTest	47.19	72.06	67.43	38.17
	Logic-CoT	80.57	89.78	88.14	63.33
QuixBugs	Native LLM	48.30	50.20	34.80	18.90
	ChatUniTest	75.68	70.29	51.15	25.13
	Logic-CoT	97.86	89.75	78.22	45.37
Lang	Native LLM	65.40	51.80	46.90	41.10
	ChatUniTest	87.95	63.24	63.89	50.24
	Logic-CoT	97.93	86.63	87.39	72.53
Math	Native LLM	58.20	40.10	35.70	33.40
	ChatUniTest	84.33	52.21	53.06	42.70
	Logic-CoT	97.68	82.32	81.71	65.78
SV-COMP	Native LLM	41.60	22.80	10.90	11.50
	ChatUniTest	68.28	41.57	23.79	18.70
	Logic-CoT	84.06	62.44	50.18	36.78
Average	Native LLM	49.20	45.86	36.68	26.02
	ChatUniTest	72.29	59.07	51.06	34.19
	Logic-CoT	91.22	82.18	77.13	56.36

Table 3. Experimental results of RQ1.

Project	Method	Compilation Pass (%)	Runtime Pass (%)	Assertion Pass (%)	Branch Coverage (%)
HumanEval-X	Native LLM	32.50	60.40	55.10	30.20
	ChatUniTest	47.19	72.06	67.43	38.17
	Logic-CoT	80.57	89.78	88.14	63.33
QuixBugs	Native LLM	48.30	50.20	34.80	18.90
	ChatUniTest	75.68	70.29	51.15	25.13
	Logic-CoT	97.86	89.75	78.22	45.37
Lang	Native LLM	65.40	51.80	46.90	41.10
	ChatUniTest	87.95	63.24	63.89	50.24
	Logic-CoT	97.93	86.63	87.39	72.53
Math	Native LLM	58.20	40.10	35.70	33.40
	ChatUniTest	84.33	52.21	53.06	42.70
	Logic-CoT	97.68	82.32	81.71	65.78
SV-COMP	Native LLM	41.60	22.80	10.90	11.50
	ChatUniTest	68.28	41.57	23.79	18.70
	Logic-CoT	84.06	62.44	50.18	36.78
Average	Native LLM	49.20	45.86	36.68	26.02
	ChatUniTest	72.29	59.07	51.06	34.19
	Logic-CoT	91.22	82.18	77.13	56.36

A similar performance gradient is observed for both Runtime Pass Rate (RPR) and Assertion Pass Rate (APR). Native LLM not only performs poorly on these two metrics overall, but also exhibits substantial variance across datasets, suggesting that the generated test inputs often fail to consistently satisfy program logic constraints. ChatUniTest improves RPR and APR to 59.07% and 51.06%, respectively. In contrast, Logic-CoT achieves 82.18% RPR and 77.13% APR. Programs in SV-COMP typically involve complex boundary conditions and multiple decision logics, making assertion failures particularly likely. Encouragingly, Logic-CoT attains an APR of 50.18% on SV-COMP, outperforming ChatUniTest (23.79%) and Native LLM (10.90%) by approximately 2× and 5×, respectively.

When programs involve multiple path constraints and numerical conditions simultaneously, explicit modeling of logical structures leads to a significant improvement in test coverage. Test cases generated by Native LLM are heavily concentrated on a small number of main execution paths, resulting in the lowest branch coverage. ChatUniTest achieves an average Branch Coverage (BC) of 34.19%, whereas Logic-CoT further improves BC to 56.36%. Since QuixBugs and SV-COMP require coverage of numerous condition combinations and boundary paths, achieving high coverage is particularly challenging. Native LLM attains BC values of 18.90% and 11.50% on these datasets, while ChatUniTest reaches 25.13% and 18.70%, respectively. Notably, Logic-CoT achieves 45.37% and 36.78%, representing nearly 3× improvements over Native LLM and nearly 2× improvements over ChatUniTest.

Summary of RQ1.

Across all evaluated datasets, Logic-CoT consistently achieves higher pass rates and branch coverage compared to the baselines, demonstrating superior test case effectiveness.

4.6.2. RQ2: Repair Efficiency

RQ2 investigates the efficiency of repairing faulty test cases. For both ChatUniTest and Logic-CoT, the maximum number of repair iterations is set to six. Table 4 reports the average number of repair turns, where only the repair process after the initial test generation is counted.

Table 4. Comparison of repair efficiency between ChatUniTest and Logic-CoT.

Project	Method	Avg. Repair Iterations
HumanEval-X	ChatUniTest	3.10
	Logic-CoT	1.60
QuixBugs	ChatUniTest	3.60
	Logic-CoT	2.00
Lang	ChatUniTest	3.20
	Logic-CoT	1.70
Math	ChatUniTest	3.70
	Logic-CoT	2.10
SV-COMP	ChatUniTest	4.30
	Logic-CoT	2.80
Average	ChatUniTest	3.58
	Logic-CoT	2.04

Note: Avg. Repair Iterations counts only the repair attempts after the initial generation. The value ranges from 0 to 5, where 0 indicates no repair needed and 5 is the maximum allowed repair rounds.

Complex programs tend to produce a larger number of faulty test cases; therefore, repair efficiency becomes a critical factor in ensuring test effectiveness while controlling generation cost. The experimental results show that Logic-CoT consistently requires fewer repair iterations than ChatUniTest across all evaluated datasets. Overall, Logic-CoT reduces the average number of repair turns by 43% compared to ChatUniTest.

On datasets with relatively simple logical structures, such as HumanEval-X and Apache Commons Lang, Logic-CoT reduces the average repair turns by 48% and 47%, respectively, compared to ChatUniTest. On more logically complex datasets, including QuixBugs, Apache Commons Math, and SV-COMP, Logic-CoT achieves reductions of 44%, 43%, and 35%, respectively.

Summary of RQ2.

Compared with ChatUniTest, Logic-CoT requires fewer repair iterations for faulty test cases and achieves higher repair efficiency.

4.6.3. RQ3: Interpretability

RQ3 evaluates the interpretability of the generated test code from the perspective of human developers. The experiment recruited 30 graduate students with more than three years of Java development experience to assess the test code along three dimensions: intent clarity, logical traceability, and maintainability, each rated on a five-point Likert scale (1–5). A double-blind protocol was adopted, in which participants were not informed of the generation source of the test code, and all ratings were collected anonymously to mitigate subjective bias.

To ensure the reliability of the subjective evaluation, inter-rater agreement was measured using Cohen’s $\kappa$ coefficient. Two independent raters conducted cross-evaluations on overlapping samples under double-blind conditions, yielding $\kappa =0.82$. According to commonly accepted benchmarks ($\kappa >0.80$ indicating strong agreement), this result demonstrates a high level of consistency in the ratings.

To enhance reproducibility, the Human Evaluation Protocol employed in this study—including detailed definitions of evaluation dimensions and the questionnaire design—has been publicly released on GitHub: https://github.com/WILL997/logic-cot (accessed on 1 March 2026). The results are presented in Table 5.

Table 5. Test case interpretability evaluation results (mean ± standard value).

Metric	Native LLM	ChatUniTest	Logic-CoT
Interpretability	$2.1\pm 0.67$	$2.5\pm 0.50$	$4.6\pm 0.29$
Traceability	$1.8\pm 0.55$	$2.3\pm 0.48$	$4.7\pm 0.24$
Maintainability	$2.0\pm 0.64$	$2.6\pm 0.55$	$4.5\pm 0.30$
Overall Avg	$1.97\pm 0.62$	$2.47\pm 0.51$	$4.60\pm 0.28$

Note: Ratings are based on a 5-point Likert scale (1 = very poor, 5 = excellent).

As shown in Table 5, Logic-CoT achieves the highest scores across all evaluation dimensions. In terms of intent clarity, Logic-CoT improves the scores by 119% and 84% compared to Native LLM and ChatUniTest, respectively. For logical traceability, the improvements reach 161% and 104%, while for maintainability, Logic-CoT outperforms the two baselines by 125% and 73%, respectively.

The standard deviation (Std) is used to measure the dispersion of scores among different evaluators, reflecting the level of consistency in subjective human assessments. As reported in Table 5, Logic-CoT exhibits lower standard deviations than the baseline methods across all evaluation metrics, indicating a higher degree of agreement among evaluators.

To assess the statistical significance of the score differences between methods, we apply the Wilcoxon signed-rank test to conduct paired comparisons between Logic-CoT and the two baseline approaches across the three evaluation dimensions. The results demonstrate that Logic-CoT significantly outperforms both Native LLM and ChatUniTest in terms of intent clarity, logical traceability, and maintainability. Specifically, the improvements over Native LLM are statistically significant across all metrics ($p<0.001$), while the improvements over ChatUniTest are also statistically significant ($p<0.01$).

Summary of RQ3.

Compared with the baselines, Logic-CoT generates test cases with substantially higher intent clarity, logical traceability, and maintainability, indicating a stronger willingness among human developers to understand and maintain the generated tests.

4.6.4. RQ4: Component Contribution (Ablation Study)

The ablation study is conducted by selectively removing the prompt templates corresponding to specific phases of Logic-CoT, while keeping all other parameter settings identical. Specifically, w/o LDP denotes the removal of prompt templates for the logic deduction phase, in which the decision table is no longer constructed; w/o TCP-doc indicates the removal of the template-based mapping and structured documentation constraints in the test case construction phase; and w/o HSP represents the removal of prompt templates for the hierarchical self-healing phase, disabling error-type–aware iterative repair. The experimental results are summarized in Table 6.

As shown in Table 6, removing any component of Logic-CoT leads to varying degrees of degradation across all four metrics—Compilation Rate (CR), Runtime Pass Rate (RPR), Assertion Pass Rate (APR), and Branch Coverage (BC)—indicating that each component contributes positively to overall test effectiveness.

Among all ablation settings, w/o LDP exhibits the most pronounced impact on test effectiveness. On average, removing the logic deduction module causes the assertion pass rate to drop from 77.13% to 56.10%, and branch coverage to decrease from 56.36% to 37.50%. This degradation is particularly severe on datasets with high logical complexity, such as QuixBugs, Apache Commons Math, and SV-COMP. For example, on SV-COMP, the assertion pass rate decreases from 50.18% to 28.00%, and branch coverage drops from 36.78% to 18.00%.

Table 6. Ablation Experiment Results.

Project	Method	Compilation Pass (%)	Runtime Pass (%)	Assertion Pass (%)	Branch Coverage (%)
HumanEval-X	Logic-CoT (Full)	80.57	89.78	88.14	63.33
	w/o LDP	78.00	85.80	74.50	50.50
	w/o TCP-doc	78.90	87.60	84.90	60.10
	w/o HSP	64.80	76.20	69.10	54.00
QuixBugs	Logic-CoT (Full)	97.86	89.75	78.22	45.37
	w/o LDP	94.00	82.50	55.00	27.00
	w/o TCP-doc	96.20	86.10	72.60	40.10
	w/o HSP	85.40	71.10	48.50	30.00
Lang	Logic-CoT (Full)	97.93	86.63	87.39	72.53
	w/o LDP	95.00	80.00	65.00	52.00
	w/o TCP-doc	96.10	83.10	80.50	67.20
	w/o HSP	90.10	69.50	64.10	58.00
Math	Logic-CoT (Full)	97.68	82.32	81.71	65.78
	w/o LDP	93.80	73.00	58.00	42.00
	w/o TCP-doc	95.40	78.60	74.20	59.30
	w/o HSP	82.90	58.60	50.20	46.10
SV-COMP	Logic-CoT (Full)	84.06	62.44	50.18	36.78
	w/o LDP	80.50	52.00	28.00	18.00
	w/o TCP-doc	82.30	59.10	44.10	31.10
	w/o HSP	64.20	40.80	25.30	19.10
Average	Logic-CoT (Full)	91.22	82.18	77.13	56.36
	w/o LDP	88.26	74.66	56.10	37.50
	w/o TCP-doc	89.98	78.90	71.26	51.56
	w/o HSP	77.48	63.24	51.44	41.44

Table 6. Ablation Experiment Results.

Project	Method	Compilation Pass (%)	Runtime Pass (%)	Assertion Pass (%)	Branch Coverage (%)
HumanEval-X	Logic-CoT (Full)	80.57	89.78	88.14	63.33
	w/o LDP	78.00	85.80	74.50	50.50
	w/o TCP-doc	78.90	87.60	84.90	60.10
	w/o HSP	64.80	76.20	69.10	54.00
QuixBugs	Logic-CoT (Full)	97.86	89.75	78.22	45.37
	w/o LDP	94.00	82.50	55.00	27.00
	w/o TCP-doc	96.20	86.10	72.60	40.10
	w/o HSP	85.40	71.10	48.50	30.00
Lang	Logic-CoT (Full)	97.93	86.63	87.39	72.53
	w/o LDP	95.00	80.00	65.00	52.00
	w/o TCP-doc	96.10	83.10	80.50	67.20
	w/o HSP	90.10	69.50	64.10	58.00
Math	Logic-CoT (Full)	97.68	82.32	81.71	65.78
	w/o LDP	93.80	73.00	58.00	42.00
	w/o TCP-doc	95.40	78.60	74.20	59.30
	w/o HSP	82.90	58.60	50.20	46.10
SV-COMP	Logic-CoT (Full)	84.06	62.44	50.18	36.78
	w/o LDP	80.50	52.00	28.00	18.00
	w/o TCP-doc	82.30	59.10	44.10	31.10
	w/o HSP	64.20	40.80	25.30	19.10
Average	Logic-CoT (Full)	91.22	82.18	77.13	56.36
	w/o LDP	88.26	74.66	56.10	37.50
	w/o TCP-doc	89.98	78.90	71.26	51.56
	w/o HSP	77.48	63.24	51.44	41.44

In comparison, w/o TCP-doc also exhibits a consistent performance decline across all metrics, though with relatively smaller magnitude. On average, its compilation rate and runtime pass rate are 89.98% and 78.90%, respectively, representing limited decreases compared to the full Logic-CoT configuration. The assertion pass rate and branch coverage decrease to 71.26% and 51.56%, respectively.

When the hierarchical dual-loop self-healing mechanism is removed (w/o HSP), the most substantial degradation is observed in pass-rate-related metrics. The average compilation rate decreases from 91.22% to 77.48%, runtime pass rate from 82.18% to 63.24%, and assertion pass rate to 51.44%. In contrast, the impact of w/o HSP on branch coverage is less severe than that of w/o LDP, with an average branch coverage of 41.44%, which remains higher than that of w/o LDP (37.50%).

Summary of RQ4.

The ablation results demonstrate that all components of Logic-CoT contribute effectively to improving test case effectiveness. Specifically, removing the logic deduction module has the greatest impact on branch coverage, while the hierarchical dual-loop self-healing mechanism primarily affects pass-rate–related metrics. The ablation of the test case construction mechanism leads to a relatively stable yet consistent performance degradation across all metrics.

4.6.5. RQ5 Traditional Method Comparison (SBST Comparison)

RQ5 further compares Logic-CoT with the traditional coverage-driven test generation tool EvoSuite, aiming to systematically evaluate the differences between the two paradigms in terms of branch coverage and human interpretability. In the EvoSuite baseline experiments, the tool was executed using its default parameter configuration, without enabling any additional optimization flags. The time budget and random seed settings were also kept at their default values to ensure experimental transparency and a fair comparison. The experimental results are presented in Table 7.

Table 7. Comparison of experimental results with EvoSuite.

Project	Method	Branch Coverage (%)	Intent Clarity	Traceability	Maintainability
HumanEval-X	EvoSuite	60.12	$1.9\pm 0.42$	$1.6\pm 0.40$	$1.8\pm 0.41$
	Logic-CoT	63.33	$4.7\pm 0.25$	$4.8\pm 0.22$	$4.6\pm 0.27$
QuixBugs	EvoSuite	49.86	$1.8\pm 0.40$	$1.5\pm 0.38$	$1.7\pm 0.39$
	Logic-CoT	45.37	$4.6\pm 0.28$	$4.7\pm 0.24$	$4.5\pm 0.30$
Lang	EvoSuite	74.08	$2.0\pm 0.45$	$1.7\pm 0.42$	$1.9\pm 0.44$
	Logic-CoT	72.53	$4.6\pm 0.27$	$4.7\pm 0.23$	$4.5\pm 0.29$
Math	EvoSuite	68.41	$1.9\pm 0.41$	$1.6\pm 0.40$	$1.8\pm 0.42$
	Logic-CoT	65.78	$4.5\pm 0.30$	$4.7\pm 0.25$	$4.5\pm 0.30$
SV-COMP	EvoSuite	34.26	$1.6\pm 0.38$	$1.4\pm 0.36$	$1.5\pm 0.37$
	Logic-CoT	36.78	$4.3\pm 0.33$	$4.5\pm 0.28$	$4.2\pm 0.33$
Average	EvoSuite	57.35	$1.84\pm 0.41$	$1.56\pm 0.39$	$1.74\pm 0.41$
	Logic-CoT	56.36	$4.60\pm 0.29$	$4.70\pm 0.24$	$4.50\pm 0.30$

From the perspective of branch coverage, EvoSuite, as a typical coverage-driven search-based test generation tool, achieves a slightly higher overall average branch coverage than Logic-CoT. Specifically, on the QuixBugs, Lang, and Math benchmarks—covering both algorithmic programs and industrial libraries—EvoSuite reaches 49.86%, 74.08%, and 68.41%, respectively, marginally outperforming Logic-CoT’s 45.37%, 72.53%, and 65.78%. This indicates that genetic-algorithm-based path exploration still maintains an advantage in coverage maximization tasks.

Meanwhile, Logic-CoT remains competitive on benchmarks where path-constraint combinations are more critical or reachable branches are sparser. For instance, it achieves 63.33% and 36.78% branch coverage on the HumanEval-X and SV-COMP datasets, respectively. This performance can be attributed to Logic-CoT’s explicit modeling of decision nodes and path-condition compositions, which enables it to reach additional branches to some extent. However, its coverage capability is also constrained by scalability limitations arising from state-space growth.

From the viewpoint of human developers, Logic-CoT significantly outperforms EvoSuite in terms of test-intent expression, logical traceability, and maintenance convenience. As shown in Table 7, EvoSuite receives average scores of only 1.84, 1.56, and 1.74 for intent clarity, logical traceability, and maintainability, whereas Logic-CoT achieves substantially higher scores of 4.60, 4.70, and 4.50, respectively. This gap mainly stems from the fact that EvoSuite-generated tests often contain extensive semantically meaningless naming and regression-style assertions, whereas Logic-CoT produces tests with clearer intent and explicit decision mappings, thereby greatly reducing the cost of understanding and maintenance.

Summary of RQ5.

Although EvoSuite exhibits an advantage in branch coverage, Logic-CoT generates test cases with comparable coverage while providing significantly better intent clarity, logical traceability, and maintainability. These results highlight the interpretability strengths of the proposed approach.

4.7. Stability, Cost, Failure Analysis, and Backend Generalization

To further evaluate the stability, cost characteristics, failure behavior, and backend generalization capability of Logic-CoT, we conducted supplementary experiments beyond the main study. HumanEval-X was selected as the benchmark dataset due to its standardized structure and moderate complexity, which enables controlled and efficient evaluation while limiting computational and API invocation overhead. In addition to the primary backbone model used in the main experiments (Qwen-plus, temperature = 0.2), GPT-4o was introduced as an alternative foundation model, also configured with temperature = 0.2, to examine the consistency of Logic-CoT across different large language models.

4.7.1. Stability

To assess methodological stability, three independent runs were performed on the HumanEval-X dataset. For each run, we computed the mean and standard deviation of test effectiveness metrics, the number of repair iterations, and ablation results.

The results of the three runs in terms of compilation pass rate, runtime pass rate, assertion pass rate, and branch coverage are presented in Table 8.

Table 8. Validity results of three runs of HumanEval-X.

Method	Compilation Pass (%)	Runtime Pass (%)	Assertion Pass (%)	Branch Coverage (%)
Native LLM (Run1)	32.50	60.40	55.10	30.20
Native LLM (Run2)	34.10	62.10	56.60	31.60
Native LLM (Run3)	31.20	58.80	53.90	28.90
Native LLM (Mean ± Std)	32.60 ± 1.18	60.43 ± 1.35	55.20 ± 1.10	30.23 ± 1.10
ChatUniTest (Run1)	47.19	72.06	67.43	38.17
ChatUniTest (Run2)	48.30	73.10	68.40	39.10
ChatUniTest (Run3)	46.40	71.10	66.30	37.20
ChatUniTest (Mean ± Std)	47.30 ± 0.78	72.09 ± 0.82	67.38 ± 0.86	38.16 ± 0.78
Logic-CoT (Run1)	80.57	89.78	88.14	63.33
Logic-CoT (Run2)	81.10	90.40	88.80	64.10
Logic-CoT (Run3)	79.90	89.10	87.40	62.60
Logic-CoT (Mean ± Std)	80.52 ± 0.49	89.76 ± 0.53	88.11 ± 0.57	63.34 ± 0.61

From an overall perspective, the three approaches consistently maintained the same performance hierarchy across all independent runs (Native LLM < ChatUniTest < Logic-CoT), indicating that the conclusions of the main experiments are highly reproducible. Logic-CoT achieved standard deviations of 0.49, 0.53, and 0.57 in compilation pass rate, runtime pass rate, and assertion pass rate, respectively—each lower than those of the baseline methods—demonstrating reduced performance variance.

In terms of branch coverage, Logic-CoT attained a mean of 63.34% (±0.61), further reflecting stable behavior, whereas Native LLM and ChatUniTest exhibited comparatively larger fluctuations.

The average number of repair iterations across the three runs is reported in Table 9.

Table 9. Results of three runs of repair cycles in HumanEval-X.

Method	Run1	Run2	Run3	Avg. Repair Iterations
ChatUniTest	3.10	3.45	2.95	3.17 ± 0.21
Logic-CoT	1.60	1.85	1.45	1.63 ± 0.16

ChatUniTest exhibits an average of 3.17 (±0.21) repair iterations, whereas Logic-CoT requires only 1.63 (±0.16) on average. Although both methods demonstrate limited variance across the three runs, Logic-CoT achieves fewer repair iterations while maintaining greater relative stability, indicating that the hierarchical self-healing mechanism delivers consistent performance under repeated execution.

The results of the ablation study across the three independent runs are presented in Table 10.

Table 10. Results of three ablation runs on HumanEval-X.

Method	Run	Compilation Pass (%)	Runtime Pass (%)	Assertion Pass (%)	Branch Coverage (%)
Logic-CoT (Full)	Run1	80.57	89.78	88.14	63.33
	Run2	81.10	90.40	88.80	64.10
	Run3	79.90	89.10	87.40	62.60
	Mean ± Std	80.52 ± 0.49	89.76 ± 0.53	88.11 ± 0.57	63.34 ± 0.61
w/o LDP	Run1	78.00	85.80	74.50	50.50
	Run2	79.10	86.90	75.80	51.60
	Run3	77.60	84.90	73.20	49.10
	Mean ± Std	78.23 ± 0.76	85.87 ± 1.00	74.50 ± 1.30	50.40 ± 1.25
w/o TCP-doc	Run1	78.90	87.60	84.90	60.10
	Run2	79.80	88.50	85.70	61.00
	Run3	78.10	86.70	83.80	59.20
	Mean ± Std	78.93 ± 0.85	87.60 ± 0.90	84.80 ± 0.95	60.10 ± 0.90
w/o HSP	Run1	64.80	76.20	69.10	54.00
	Run2	66.20	77.50	70.50	55.40
	Run3	63.90	74.80	67.90	52.60
	Mean ± Std	64.97 ± 1.15	76.17 ± 1.35	69.17 ± 1.30	54.00 ± 1.40

It can be observed that, under all ablation settings, the performance trends remain consistent across the three independent runs. The removal of LDP, TCP-doc, or HSP consistently results in performance degradation, while the standard deviations of all evaluated metrics remain within a relatively small range. This indicates that the contribution of each component to performance improvement is stable and reliable.

Overall, the results of the three independent executions demonstrate that Logic-CoT achieves superior performance on HumanEval-X with lower variance, thereby exhibiting strong methodological stability.

4.7.2. Cost and Failure Analysis

Table 11 reports the cost-related results on the HumanEval-X dataset. Compared with ChatUniTest, Logic-CoT significantly reduces the average number of LLM calls (6.2 vs. 8.4), the average number of repair iterations (1.85 vs. 3.45), and the average execution time (19.2 s vs. 24.6 s). Meanwhile, the assertion pass rate improves substantially from 68.40% to 88.80%. These findings indicate that Logic-CoT not only enhances test effectiveness but also achieves more efficient cost control.

Table 11. HumanEval-X cost results.

Method	Avg LLM Calls	Avg Repair Turns	Avg Time (s)	Assertion Pass (%)
ChatUniTest	8.4	3.45	24.6	68.40
Logic-CoT	6.2	1.85	19.2	88.80

For the failed test cases, Table 12 summarizes the root-cause distribution of Logic-CoT failures. Among the nine failed cases, path reasoning errors account for 33.33%, non-convergent repair processes for 33.33%, incorrect expectation inference for 22.22%, and environment modeling issues for 11.11%. These results indicate that failures are primarily concentrated in path reasoning deficiencies and repair non-convergence scenarios. This analysis explains why, despite substantially reducing repair iterations and invocation costs, Logic-CoT still exhibits a small number of challenging failure cases.

Table 12. Failure results in HumanEval-X.

Root Cause Category	Count	Ratio (%)
Path reasoning error	3	33.33
Oracle inference error	2	22.22
Repair non-convergence	3	33.33
Environment modeling issue	1	11.11

Overall, Logic-CoT effectively reduces invocation and repair costs while maintaining a higher pass rate, and its overall failure ratio remains relatively low.

4.7.3. Backend Generalization

To investigate the impact of different backbone models on the performance of Logic-CoT, comparative experiments were conducted on the HumanEval-X dataset. The results are presented in Table 13.

Table 13. Comparative experimental results in HumanEval-X.

Base Model	Compilation Pass (%)	Runtime Pass (%)	Assertion Pass (%)	Branch Coverage (%)
Qwen-plus	80.57	89.78	88.14	63.33
GPT-4o	82.10	91.05	89.75	65.20

From an overall perspective, the performance differences between the two backbone models are relatively minor. Compared with Qwen-plus, GPT-4o achieves modest improvements in compilation pass rate (82.10% vs. 80.57%), runtime pass rate (91.05% vs. 89.78%), assertion pass rate (89.75% vs. 88.14%), and branch coverage (65.20% vs. 63.33%).

Overall, Logic-CoT maintains stable test generation performance across different backbone models, demonstrating a certain degree of backend generalization capability.

5. Results and Discussion

This section provides an in-depth discussion of why Logic-CoT outperforms the baseline approaches.

From the results of RQ1, it can be observed that Logic-CoT consistently achieves higher pass rates and branch coverage across all evaluated datasets. Compared with zero-shot and context-enhanced generation methods, this improvement stems from the logic-deduction-driven generation paradigm, which transforms test generation from a black-box process based on stochastic exploration into a white-box process with explicit, traceable, and interpretable logical reasoning.

Consider the CharRange#toString() method in Apache Commons Lang as an illustrative example. The return value of this method is determined by three independent and composable conditions: whether the cached field iToString is null, whether the range is negated, and whether the start and end boundaries are equal. None of these conditions dominates the others; instead, different combinations of their Boolean values lead to mutually exclusive execution paths and directly affect the final string representation.

For such logic structures governed by multiple interacting Boolean conditions, the key challenge in test generation lies in systematically covering condition combinations and constructing corresponding test cases for each path. Without explicit path modeling, probabilistic generation approaches tend to focus on high-frequency execution paths while overlooking low-frequency but semantically valid paths.

This phenomenon is reflected in the experimental results. Tests generated by ChatUniTest mainly concentrate on common scenarios involving single-condition variations.

Although such tests are functionally correct, they typically cover only one execution path, leaving other mutually exclusive branches untested.

In contrast, during the generation phase, Logic-CoT first discretizes control-flow decision points into logical states and enumerates mutually exclusive paths in the form of a state table. Subsequently, a corresponding test case is generated for each enumerated path. Beyond covering commonly occurring high-frequency paths, Logic-CoT also constructs dedicated test cases for low-frequency paths, such as the condition “non-negated and unequal endpoints.” For the state $\mathtt{iToString}=\mathtt{null}\wedge \mathtt{negated}=\mathtt{true}\wedge \mathtt{start}\ne \mathtt{end}$, its representation in the decision table is shown as follows:

Based on the aforementioned logical state, the test case generated by Logic-CoT is presented as follows:    

Because the set of execution paths is explicitly determined prior to test construction, the generation process is required to cover diverse condition combinations. This design directly explains the observed improvements in branch coverage.

The results of RQ2 show that Logic-CoT consistently requires fewer repair iterations than ChatUniTest across all datasets. This difference can be attributed to how repair behaviors are constrained in the two approaches.

ChatUniTest adopts a single feedback loop, where assertion failures may be resolved by weakening assertions or opportunistically adjusting inputs or test structures to force tests to pass, without fully understanding the root cause of the failure. In contrast, Logic-CoT explicitly separates the repair process into syntactic/runtime repair and semantic repair. In the former, only code-level issues are allowed to be fixed, while modifications to test logic and assertions are strictly prohibited. In this context, structured comments encoding intent, path, and expectations serve as strong constraints. In the latter, assertion failures trigger a rollback to logical state reasoning, where expected behaviors are re-derived based on updated logical deductions. Again, the structured documentation provides critical guidance for this process. This layered feedback mechanism avoids repair paths that “weaken assertions for passability,” resulting in more directed repairs and explaining the overall reduction in repair iterations.

The results of RQ3 demonstrate that Logic-CoT achieves the highest scores in intent clarity, logical traceability, and maintainability, with relatively low standard deviations, indicating a high level of agreement among evaluators. This advantage primarily arises from differences in how information is organized during test generation.

Both Native LLM and ChatUniTest focus primarily on generating executable code, and their tests typically encode only input–output relationships, with little explicit explanation of path conditions or design intent. In contrast, Logic-CoT introduces state tables and logical reasoning results during generation and embeds them into test scripts using @Intent, @Path, and @State annotations. As a result, each test case explicitly documents why the test is constructed and under which logical assumptions the assertion holds.

In the aforementioned example, all Logic-CoT-generated tests are annotated with their corresponding logical states and expected outcomes. This structured representation allows test scripts to function as executable specifications of method behavior, significantly reducing comprehension and maintenance costs. In contrast, tests lacking logical annotations make it difficult to determine whether assertion failures after code changes stem from defects in the implementation or from invalidated test assumptions.

The ablation results of RQ4 indicate that all components of Logic-CoT contribute positively to test effectiveness, albeit to different extents. These differences arise from the distinct roles each component plays at different stages of the test generation pipeline.

The logic deduction module (LDP) operates at the pre-generation stage by modeling the path space through explicit enumeration of decision condition combinations. Removing this module deprives the generation process of systematic path modeling, causing tests to concentrate on high-frequency paths and leading to a pronounced drop in branch coverage and, consequently, assertion accuracy.

In contrast, the test case construction mechanism (TCP) bridges logical reasoning and execution by mapping existing logical states to structured, executable test code. Since this module does not alter the set of coverable paths, its removal does not significantly reduce branch coverage. Instead, it indirectly affects overall test quality by reducing structural stability and consistency in assertion expression, resulting in relatively uniform but smaller performance degradation across metrics.

The hierarchical dual-loop self-healing mechanism (HSP) operates after test generation, focusing on error repair. When this mechanism is removed, the system lacks effective constraints for handling compilation errors, runtime exceptions, and assertion failures, leading to a substantial number of tests that fail to execute successfully. Consequently, its impact is most pronounced on pass-rate-related metrics.

The experimental results of RQ5 further highlight the fundamental differences between Logic-CoT and traditional coverage-driven approaches. EvoSuite, which primarily targets structural coverage maximization, demonstrates well-established advantages in path exploration and coverage optimization. In contrast, this work focuses on the key challenges that arise when large language models are involved in test generation, including insufficient correctness, limited coverage, and a lack of interpretability. By incorporating explicit logical modeling and an intent-driven test construction mechanism, Logic-CoT mitigates these inherent shortcomings of the end-to-end LLM generation paradigm to a certain extent, thereby improving both the effectiveness and maintainability of the generated tests.

Finally, Logic-CoT is subject to certain limitations. As cyclomatic complexity increases further, explicit state enumeration may introduce additional combinatorial overhead. Moreover, for programs whose semantics heavily depend on external environments, deriving precise expected behaviors through logical reasoning remains challenging.

In the worst-case scenario, if there exist n mutually independent decision predicates, the number of possible decision combinations may theoretically reach $O\left(2^n\right)$. Consequently, methods with high cyclomatic complexity may incur substantial computational overhead. However, the state construction process in Logic-CoT does not exhibit uncontrolled growth. First, during the logical deduction phase, semantically equivalent condition combinations or those that do not affect the output can be identified within the decision table, allowing redundant states to be merged or pruned at the source, thereby compressing the decision space. Second, candidate paths can be prioritized according to branch nesting depth, exception-triggering frequency, or path-constraint complexity, enabling preferential coverage of high-risk or high-complexity paths. Finally, under constrained computational resources or invocation budgets, a hierarchical or progressive path-expansion strategy can be adopted, gradually enlarging the state space while ensuring minimal decision coverage, thus balancing coverage and computational cost.

Although the experimental subjects in this study primarily consist of controlled functions without third-party dependencies, the Logic-CoT framework is not restricted to purely functional scenarios. In more complex real-world methods, object states and collections can be abstracted as extended state variables and logically modeled through preconditions and postconditions. Side effects or external dependencies may be isolated using mock or stub mechanisms, while non-deterministic behaviors can be handled by explicitly recording environmental assumptions within path conditions. These extensions can be incorporated into the existing “code → logic → code” framework without altering the overall procedural structure.

Moreover, while Logic-CoT partially relies on the semantic understanding and logical reasoning capabilities of large language models (LLMs), its core contribution lies in structured logical modeling and a hierarchical feedback mechanism rather than dependence on a specific model scale. Through the structured constraint of the “code → logic → code” paradigm and stage-isolated prompt design, the generation process is confined within a well-defined structural space, thereby reducing reliance on open-ended generative capability and prompt sensitivity. It is worth noting that large models emphasize general-purpose competence, whereas smaller models may achieve strong task-specific performance through targeted fine-tuning. Given that the task in this study primarily depends on structured semantic understanding and conditional reasoning, smaller or open-source models, when appropriately optimized, could theoretically achieve comparable performance. The consistent trends observed in cross-backend experiments (Qwen-plus and GPT-4o) further demonstrate the framework’s degree of model portability.

In addition, assertion generation may theoretically be influenced by intent inference bias or inaccuracies in path reachability analysis. If the inferred functional intent deviates from the true specification, or if path constraints fail to precisely characterize actual reachable conditions, the model may generate assertions that are logically consistent yet semantically incorrect. However, such risks are mitigated through multiple structural safeguards embedded in the framework design. First, functional intent serves solely as high-level semantic guidance and is not used independently to derive test oracles; assertions are always deduced from source-code semantics and concrete input states under a fixed execution path, thereby limiting the direct impact of semantic hallucinations. Second, path structures are determined through static analysis of the abstract syntax tree (AST) and combined with branch-coverage objectives to generate logical node constraints. The LLM operates strictly within predefined paths and constraints to complete state assignments, rather than freely inferring reachability. Furthermore, during the state generation phase, the LLM is required to perform self-verification of path reachability, further reducing the risk of path misjudgment.

As shown in Table 12, among the nine failed test cases, path reasoning errors and non-convergent repairs each account for 33.33%, expectation inference bias accounts for 22.22%, and environment modeling issues account for 11.11%. A small number of assertion failures can be traced to intent deviation, but such cases do not constitute a dominant source of error. Notably, no systematic phenomenon of “logically consistent yet semantically incorrect” assertions was observed in the current experimental sample. Random manual inspection of passing assertions likewise did not reveal significant semantic discrepancies. Nevertheless, occasional misjudgments in path reachability or minor intent deviations may still occur. To further enhance robustness, future work may incorporate intent-consistency self-verification mechanisms and additional path-consistency checks during the repair phase.

6. Summary and Outlook

To address the challenges of high error rates, low branch coverage, low assertion pass rates, and poor interpretability in LLM-based unit test generation, this paper proposes Logic-CoT, a test generation framework centered on explicit logical deduction. By externalizing program decision conditions into a structured state table, Logic-CoT transforms test construction from implicit generative speculation into a constrained logical reasoning process. This design enforces path coverage and assertion consistency during the generation phase, leading to more effective, reliable, and interpretable test cases.

From a software testing perspective, Logic-CoT provides a practical and explainable approach for improving the reliability and maintainability of automatically generated unit tests.

Future work will extend Logic-CoT in two directions. First, for methods with high cyclomatic complexity, exhaustive state enumeration may introduce redundant paths. We plan to mitigate this issue by merging equivalent conditions and prioritizing the generation of test cases for high-risk branches, thereby reducing unnecessary state combinations. Second, for programs whose behavior depends heavily on external environments, we will explore integrating static analysis techniques to generate supplementary contextual information, enhancing prompt inputs and improving the applicability of Logic-CoT in real-world engineering scenarios.