LLM4ATS: Applying Large Language Models for Auto-Testing Scripts in Automobiles

Abstract

This paper introduces LLM4ATS, a framework integrating large language models, RAG, and closed-loop verification to automatically generate highly reliable automotive automated test scripts from natural language descriptions. Addressing the complex linguistic structure, strict rules, and strong dependency on the in-vehicle communication database inherent in ATS scripts, LLM4ATS innovatively employs fine-grained line-level generation and a rule-guided iterative refinement mechanism. The framework first enhances prompt context by retrieving relevant information from constructed syntax and case knowledge bases via RAG. Subsequently, each generated script line undergoes rigorous verification through a two-stage validator: initial syntax validation followed by semantic compliance checks against the communication database for signal paths and value domains. Any errors trigger structured feedback, driving iterative refinement by the large language model until fully compliant scripts are produced. This paper evaluated the framework’s effectiveness on real ATS datasets, testing models including GPT-3.5, GPT-4, Qwen2.5-7B, and Qwen2.5-72B-Instruct. Experimental results demonstrate that compared to zero-shot and few-shot baseline methods, the LLM4ATS framework significantly improves generation quality and pass rates across all models. Notably, the strongest GPT-4 model achieved a script pass rate of 91% with LLM4ATS, up from 42% in zero-shot mode, and validated functional effectiveness on a specified in-vehicle hardware platform (Chery Fengyun T28 dashboard). At the same time, expert manual evaluations confirmed the superior performance of the generated scripts in correctness, readability, and compliance with industry standards.

1. Introduction

Intelligent in-vehicle systems are critical to modern automotive technology, with a global market valued at approximately USD 2.3 billion in 2023 [1]. While automated test scripts (ATSs) are pivotal for verifying these systems, their traditional manual development is time-consuming, error-prone, and requires deep domain expertise [2]. Recent advances in large language models (LLMs) present a promising alternative to conventional ATS generation methods [3,4,5,6]. However, it is crucial to note that the naive application of LLMs in the engineering workflow, characterized by inefficient prompting and unverified output execution, can lead to unsafe operations [7].

Automated test scripts (ATSs) in automotive systems represent a domain-specific language engineered for the automated validation and verification of vehicle electronic systems and components. These scripts function as a structured, imperative interface between test specifications and a heterogeneous suite of test bench hardware, including programmable power supplies, communication bus interfaces (CAN, LIN, and Ethernet), relay modules, programmable resistor loads, robotic manipulators, and data acquisition units. Their syntax, defined by formal specification documents, is designed for precision, enabling both the actuation of vehicle components and the assertion of expected system responses.

The command structure is highly variable and context-dependent, extending far beyond a single template. While a core format for direct signal manipulation is Bus-Type-ECU-Message-Value (e.g., CAN-Sig-CCU_1-UMM_UsageModeSt-1), as shown in Figure 1, the language encompasses a wide array of specialized formats. For hardware control, commands follow patterns such as Pwr-1-SupplyPower-13.5 to set a power supply channel to 13.5 V or Res-2-Random-10-100 to set a resistor channel to a random value between 10 and 100 ohms. Actuation commands include sequences like Robot-Touch-A-B-C for coordinated robotic arm movements. For bus systems, complex instructions are supported, such as CAN-1-Sig-PEPS-PEPS2-SysPowerMods-Step-1-20-1 to ramp a CAN signal value from 1 to 20 in increments of 1.

Crucially, the ATS language incorporates constructs for sophisticated test logic. This includes flow control with loops (e.g., Repeat-1-3-100 to execute steps 1 through 3 one hundred times), timed delays (Wait-500ms or Wait-Random-10ms-10s), and conditional result checking integrated into actions (e.g., …-check-all). The assertion or validation side employs a parallel syntax for verifying system states, such as CAN-GetSig-PEPS-PEPS2-SysPowerMods-1 to confirm a signal value or Vision-OCR-Gear-P to validate a displayed character via optical recognition.

Given its role in validating safety-critical vehicle functions, the correctness of an ATS script is paramount. Each command must be syntactically valid according to the formal grammar and semantically compliant with hardware constraints and DBC file definitions (e.g., DBC files for valid message–signal–value mappings). This necessitates rigorous, automated verification of both syntax and semantics prior to execution on physical test systems.

To address these challenges, this paper proposed an automated framework for generating and verifying ATSs based on LLMs, to improve flexibility, accuracy, and consistency in script generation, ultimately improving test coverage and reliability in the automotive testing process. This framework, as shown in Figure 2, is divided into two key stages: the Retrieval-Augmented Generation (RAG) stage [8] and the correction stage [9]. During the RAG stage, the system takes natural language specifications that describe vehicle system tests as input and uses RAG technology to retrieve relevant information from knowledge bases and case examples, which helps to generate scripts that adhere to syntactic norms [10]. In the correction stage, this paper designs retrieval-enhanced prompts to generate ATSs and then validates the generated scripts against syntax checkers and DBC files for both grammatical and functional accuracy. Any detected error messages are fed back to further optimize the scripts until they pass the corrections.

Furthermore, this paper has designed a specialized dataset and employed Parameter-Efficient Fine-Tuning (PEFT) techniques [11], particularly Low-Rank Adaptation (LoRA) [12], to enhance training in specific domains. This approach significantly improves the efficiency and reliability of automotive testing processes, reducing manual effort while maintaining high standards for modern vehicle systems.

This paper’s key contributions include the following:

• Development of an automated language-driven system that generates verified testing scripts for intelligent vehicle systems from natural language descriptions.
• Integration of LLMs with automated verification mechanisms to ensure script reliability through iterative refinement.
• Comprehensive evaluation of various LLM configurations (GPT-3.5, GPT-4, Qwen2.5-7B, and Qwen2.5-72B-Instruct) for the generation of ATSs, measuring success rates and quality scores assessed by experts.

This paper is structured as follows. Section 2 reviews related work. Section 3 details the LLM4ATS method, which has a RAG stage and a correction phase using DBC files. Section 4 explains the experiments, including the dataset and fine-tuning approach. Section 5 presents the results and compares different LLMs. Section 6 discusses the implications for low-resource code generation and compares model types, before Section 7 concludes.

2. Related Work

This paper is primarily related to two fields of prior research: ATS code programming and verification and LLM augmentation.

2.1. Large Language Models for Code Generation

The application of LLMs to specialized programming domains has garnered significant research interest, particularly in safety-critical systems where code correctness is paramount. Existing approaches can be broadly categorized into those focusing on general-purpose code generation and those targeting domain-specific verification.

The LLM4PLC framework represents a notable advancement in applying LLMs to Industrial Control System (ICS) programming for Programmable Logic Controllers (PLCs) [13]. This work shows that state-of-the-art LLMs such as GPT-4 and Code Llama fail to produce valid programs for PLCs, primarily because ladder logic and structured text languages used in industrial automation are niche. To address this, LLM4PLC proposes a user-guided iterative pipeline incorporating external verification tools, including grammar checkers, compilers, and SMVs (Symbolic Model Verifiers). The framework further employs PEFT techniques and prompt engineering to enhance the quality of generated output. Experimental evaluations demonstrate that this pipeline improves the generation success rate from 47% to 72% and expert-assessed code quality from 2.25/10 to 7.75/10. A closely related approach, LLM4CVE, extends similar principles to software vulnerability repair, achieving a 20% increase in ground-truth code similarity for real-world vulnerabilities [14].

In contrast, automotive ATSs consist of a sequence of independent, single-line commands, each adhering to a likelihood “Bus-Type-ECU-MSG-Value” format. The correctness of an ATS depends on the precision of each individual line rather than overarching program logic or structure. The proposed LLM4ATS framework addresses these shortcomings through the following key innovations, representing a substantive advancement:

1.

Fine-Grained, Line-by-Line Generation Mechanism: Unlike methods that generate complete script files, LLM4ATS decomposes natural language instructions and targets the generation and validation of each independent command line. This design enables the model to focus exclusively on the format correctness and semantic accuracy of a single command, significantly reducing the complexity of each generation step. An iterative correction loop allows independent refinement of each line, thereby ensuring the overall reliability of the final script.

2.

Rule-Guided Generation via Structured Domain Knowledge: Beyond retrieving examples from a case library, LLM4ATS critically extracts formalized domain rule descriptions from a knowledge base. With the correction of the DBC file, the generated descriptions (e.g., “The valid value range for message UMM_UsageModeSt is [0, 1]”) are structurally embedded into the prompt, directly instructing the LLMs to comprehend and apply domain-specific constraints. This “rule-guided generation” capability allows the model to produce novel script variants that, while potentially absent from its training data, fully comply with formal specifications.

2.2. Software Defect Prediction and Verification

Software defect prediction has evolved as a critical field to improve software quality proactively. Traditional methods often rely on software metrics, but contemporary approaches increasingly employ sophisticated machine learning and hybrid models to predict defect-prone modules. A notable example is the two-tier deep and machine learning approach optimized by an adaptive multi-population firefly algorithm for software defect prediction [15]. This approach represents the current state of the art in predictive analytics for software quality.

The mentioned two-tier framework ingeniously combines deep learning for feature extraction with ensemble machine learning classifiers (e.g., XGBoost) for final prediction [16]. Its key innovation lies in employing an adaptive multi-population firefly algorithm (FA) to optimize the hyperparameters of the entire pipeline. This metaheuristic optimization addresses the “no-free-lunch” theorem in search algorithms, attempting to find a superior configuration for the specific problem domain of defect prediction on datasets like GHPR (GitHub Pull Request) [17]. The method demonstrates advanced capability in identifying which software modules or files are statistically likely to contain defects based on historical and metric data.

However, this paradigm and similar defect prediction models share a fundamental characteristic: they are proactive yet indirect. They analyze code or development artifacts after they are written to assess the probability of defects, serving as a risk assessment tool to guide manual review efforts. The verification step, if any, remains separate and often manual. The actual presence and precise location of a functional defect—such as an incorrect API call, a wrong variable value, or a violation of a domain-specific protocol—are not automatically identified or corrected by these predictive models [18].

In contrast, the verification mechanism within the LLM4ATS framework operates on a fundamentally different principle: deterministic, rule-based verification at the point of creation. This approach eliminates the probabilistic gap between defect prediction and actual defect detection. It is achieved through a two-stage, automated verification pipeline:

1.

Syntax and Format Validation: Immediately after generation, each line of the ATS is subjected to a rigid syntax checker. This validator ensures strict compliance with the required rules. Any deviation in structure (e.g., a missing field, an extra delimiter, or an incorrect separator) is flagged as an error. This step guarantees basic syntactic integrity, a form of verification that general defect prediction models cannot provide.

2.

Semantic and Domain-Specific Validation via DBC: This constitutes the core of functional correctness verification. Valid scripts from the first stage are cross-referenced against the automotive Database Communication (DBC) files. The system performs a precise lookup to verify the following: (1) Every ECU-CCU-MSG pathway referenced in the script is a valid and defined signal path within the DBC files. (2) The value assigned to each signal falls strictly within its predefined and allowed value domain (e.g., a Boolean 0/1, an enumerated range [0–3], or a specific physical scale).

This process does not predict potential errors; it conclusively identifies them. An error message such as “Row 3 Error, Invalid value, Valid value is [0, 1].” provides direct, actionable feedback. This feedback is then fed back into the generation loop for correction, creating a closed-loop system that ensures the final output is not just statistically sound but guaranteed to be syntactically and semantically correct according to the authoritative domain specification (the DBC file).

3. Methodology

LLM4ATS is a user-guided iterative workflow designed to assist LLMs in generating ATSs for intelligent automotive systems. The purpose of this paper is to overcome the limitations of existing state-of-the-art LLMs in this area. The workflow incorporates real machine testing and utilizes a syntax checker to enhance the quality and reliability of the generated scripts. Optimization is achieved through RAG, Chain of Thought (COT), and model fine-tuning techniques that employ LoRAs. The implementation follows a top-down perspective of the typical workflow in automotive testing environments. The subsequent sections elaborate on the operational flow of the workflow.

3.1. RAG Stage

The development of effective prompts is critical to guide LLMs in generating accurate and coherent ATS for automotive systems. This paper designs structured prompts that provide clear context, facilitating the translation of natural language specifications into executable ATS code. These prompts are carefully crafted to incorporate relevant examples and grammatical rules retrieved from the case knowledge base and the syntax knowledge base. This ensures that the LLMs comprehend the specific syntax patterns and functional requirements of ATS generation.

As shown in Figure 3, this paper utilizes RAG to retrieve relevant information from knowledge bases and case examples, integrating them into the designed prompt templates, as depicted in Figure 4. The syntax knowledge base contains the grammatical rules and usage guidelines for ATSs, thereby reducing the likelihood of syntactical errors. The case knowledge base includes examples that associate natural language commands with corresponding serialized ATSs, providing practical instances that the LLMs can learn from and emulate. Additionally, COT is incorporated into the prompts, such as “Let us think step by step,” to encourage the model to articulate its reasoning process, thereby enhancing the accuracy and coherence of its output [7].

3.2. Correction Phase

ATSs are subjected to syntax checks on the host system to verify their grammatical and functional accuracy. The syntax checker evaluates whether the scripts conform to established syntax rules based on the DBC files [19]. DBC files are critical tools for the definition of ATSs that define the types of messages (MSGs) that ECUs can send or receive, including signal values, ranges, and data encoding formats. If any tests fail, error messages are generated and fed back into the correction phase, as illustrated in Figure 2.

During this process, the syntax checker enforces two key conditions:

1.

The script must strictly follow the syntax format.

2.

Each CCU or ECU must contain only legitimate MSGs as defined in the DBC file, with each message containing valid values for each parameter.

In the syntax validation process, this paper employs two distinct validation methods:

1.

Syntax Validation: This stage first checks whether the generated ATS adheres to the syntax knowledge format. This paper then uses the DBC file to verify the legitimacy of the values associated with ECU or CCU, MSG, and value fields. If discrepancies such as invalid formats or out-of-bound values are detected, an error message is generated. For example, the checker might flag an error like “Row 3 Error, Invalid value, Valid value is [0, 1].”

2.

Real Machine Testing: In this phase, the host system validates the generated ATS to ensure it is fully functional and compliant with all specifications. If the script fails, the system produces a specific error message that identifies the functional flaw or non-compliance.

Both syntax and functional error feedback are then fed back into the correction phase. Each error is processed individually to formulate a targeted correction prompt for the LLM. This sequential error-by-error correction strategy enables the LLM to make precise adjustments, effectively reducing the required number of iterative correction cycles. A key benefit of this approach is the mitigation of cascading errors (e.g., where an initial syntax error causes subsequent issues). By prioritizing the correction of root causes, the system can resolve multiple downstream errors in a single iteration. This dual-layered validation and correction mechanism ensures that the final ATSs are robust and accurate and meet the stringent requirements of automotive testing systems.

3.3. Workflow Operation

The entire workflow works is illustrated in Figure 5. For each natural language instruction provided by the user, the system retrieves the corresponding ATS from the case and syntax knowledge base. The LLMs then refine and restructure this initial match to ensure accuracy and coherence. Subsequently, for each natural language instruction, the system retrieves the relevant syntax rules from the syntax knowledge base. Given the one-to-one mapping characteristic of ATSs, each natural language instruction is decomposed into individual components. Each component is searched against the case knowledge base to find the most relevant ATS code. In the Search Phase, for each natural language component, the system retrieves the three most relevant ATS codes from the case knowledge base.

If the generated ATS violates any of these rules, the syntax checker identifies the issue and returns a detailed error message, such as “Row 3 Error, Invalid value, Valid value is [0, 1].” This error information is incorporated into the correction phase’s prompt template, allowing the model to reconsider and correct the script’s syntax and format by referencing the syntax knowledge base. The corrected ATS is then outputted, ensuring compliance with the structural and specific value requirements.

4. Experiments

4.1. Research Questions

To evaluate the effectiveness of the proposed LLM4ATS framework in generating test scripts and to investigate the performance differences among various models in generating ATSs, this paper explores the following research questions (RQs):

Question 1: How is the quality of test scripts generated using LLM4ATS? The purpose of this question is to confirm the quality of the test scripts generated by LLM4ATS.

Question 2: Do different LLMs exhibit performance differences in ATS generation tasks? This question examines whether model type and parameter size affect ATS generation tasks. For this purpose, this paper evaluates representative closed-source models and open-source models with different parameter sizes.

Question 3: Does fine-tuning with proprietary data improve performance in ATS generation tasks? The purpose of this research question is to explore whether fine-tuning models with proprietary data can enhance their performance in the generation of automotive system test scripts.

4.2. Metrics

The evaluation methodology of this paper includes two key metrics: the pass rate and human quality assessment.

Pass Rate: Scripts passing the structural check are then subjected to a specialized validator that assesses compliance with actual automotive testing rules and constraints. This validation process is conducted on a specific target hardware platform: the Chery Fengyun T28 instrument cluster display (Part Number: 7003001609AAABK, Software Version: 01.15.00, Hardware Version: 0.0.0). The pass rate is evaluated using the pass@k metric, indicating the method’s accuracy in producing scripts that are both structurally correct and functionally valid on this designated hardware. A high pass rate signifies the reliability of the proposed approach in generating scripts that meet industry-specific compatibility and operational standards [20].

Human Quality Assessment: For scripts that pass both structural and validation checks, a human evaluation is conducted by a panel of domain experts to assess the practical usability of the generated scripts. Experts are blinded to the model source of each script. The evaluation employs three criteria, each rated on a scale of 1–10, with detailed rubrics provided in

Table 1. Rubric for human quality assessment of generated ATSs.

Score Range	Correctness	Readability	Conformance to Industry Standards
0–2	fundamentally non-executable; completely contradictory; critical logical or semantic errors	completely unstructured and incomprehensible	no awareness; violates all major standards
2–4	largely incorrect outcomes; major functional deviations	very difficult to understand	few industry conventions; frequent violations
4–6	partial correctness; notable functional errors or omissions	significant effort	some basic industry conventions but inconsistently
6–8	primary requirements; minor edge-case errors	clear and reasonably easy to follow	largely conforms
8–10	flawlessly according to all specifications	exceptionally clear; self-documenting; easy to maintain	fully exemplifies industry best practices

.

The final score for each criterion is the average of the ratings provided by all expert evaluators. This qualitative assessment provides critical insights into the functional reliability, maintainability, and professional suitability of the generated scripts in real-world automotive testing environments.

4.3. Dataset

This paper utilizes the DS-ATS dataset for model training and evaluation, provided by Huizhou Desay SV Automotive Co., Ltd., Huizhou, China. The construction process of the dataset is shown in the Figure 6. This dataset comprises 4329 ATSs, covering a wide range of automated testing scenarios [21].

4.4. LLM Fine-Tuning

To address Question 3, this paper selects two open-source models with different parameter sizes: Qwen2.5-7B and Qwen2.5-72B-Instruct. These models are fine-tuned using the DS-ATS dataset, allocating 90% for training and 10% for evaluation. To better supervise the fine-tuning process, this paper designs three distinct training strategies:

1.

Direct Generation: Utilizing original scripts for end-to-end generation tasks.

2.

Split Generation: Dividing each script into individual commands to create a one-to-one mapping for more granular generation tasks.

3.

Completion Task: Randomly remove sections from the middle of scripts to evaluate the model’s code completion capabilities.

This paper employs the QLoRA method for fine-tuning [11]. For Qwen2.5-72B-Instruct, 4-bit quantization is used during the fine-tuning process, enabling efficient training on four NVIDIA A100 GPUs with 320 GB total VRAM. This quantization approach significantly reduces memory requirements while maintaining model performance. The parameter settings are detailed in

Table 2. Training parameters.

Parameter	Value
Learning Rate	5 × 10−5
LoRA Rank	8
LoRA Alpha	16
Batch Size	256
Epoch	10

.

These parameters were carefully selected on the basis of empirical testing and best practices in the field of large language model fine-tuning, optimized for the specific task of automotive testing script generation and manipulation.

5. Results

In this section, this paper presents the results of empirical evaluations, emphasizing the effectiveness of the approach in automated script generation for automotive systems. This paper assesses the method using two core metrics: pass rate and human-assessed script quality.

5.1. Question 1: How Is the Quality of Test Scripts Generated Using LLM4ATS?

To address this question, this paper compares the performance of zero-shot, few-shot, and LLM4ATS methods using the pass rate and human quality assessment metrics. The evaluation includes four different language models—GPT-3.5, GPT-4, Qwen2.5-7B, and Qwen2.5-72B-Instruct—and applies four distinct methodologies: zero-shot, LoRA, three-shot, and LLM4ATS.

The results are presented in

Table 3. LLM performance comparison across different evaluation metrics. Values that outperform others are highlighted in boldface.

Model	Method	PASS@1	PASS@10	Readability	Correctness	Best-Practice
GPT-3.5	Raw	5%	8%	1.5	1.8	1.4
	Zero-Shot	24%	42%	2.0	2.5	1.4
	LoRA	-	-	-	-	-
	Three-Shot	54%	60%	3.5	4.5	3.6
	LLM4ATS without Correction Stage	68%	72%	6.5	4.5	4.1
	LLM4ATS	79%	83%	7.0	7.5	6.0
GPT-4	Raw	8%	23%	5.5	0.2	4.8
	Zero-Shot	42%	58%	5.2	4.0	5.1
	LoRA	-	-	-	-	-
	Three-Shot	65%	79%	6.5	6.2	6.4
	LLM4ATS without Correction Stage	85%	90%	8.5	7.5	6.8
	LLM4ATS	91%	92%	8.5	9.0	7.0
Qwen2.5-7B	Raw	3%	12%	1.5	0.2	1.4
	Zero-Shot	18%	35%	1.8	1.0	1.9
	LoRA	25%	36%	2.5	2.4	2.3
	Three-Shot	34%	65%	3.3	3.5	3.4
	LLM4ATS without Correction Stage	70%	89%	7.0	6.5	6.7
	LLM4ATS	83%	86%	7.2	8.0	7.0
Qwen2.5-72B-Instruct	Raw	6%	18%	2.0	3.0	2.5
	Zero-Shot	28%	51%	2.8	2.9	2.7
	LoRA	33%	57%	3.3	3.2	3.1
	Three-Shot	61%	86%	6.0	6.2	6.1
	LLM4ATS without Correction Stage	85%	89%	9.0	7.7	7.6
	LLM4ATS	89%	91%	9.0	8.5	7.0

. It is evident that regardless of the backbone model used, the LLM4ATS method proposed in this paper achieves higher pass rates. Specifically, when the base model is GPT-4, the performance improves from 42% with zero-shot to 91% using this paper’s method. Additionally, in few-shot configurations, this paper’s method also exhibits superior performance, increasing from 65% to 91%. The generated test scripts exhibit higher accuracy and readability, as further detailed in the human quality assessment metrics, as shown in Table 1.

5.2. Question 2: Do Different LLMs Exhibit Performance Differences in ATS Generation Tasks?

Different large language models (LLMs) exhibit significant performance variations in ATS generation tasks, reflecting both the inherent capabilities of the models and the influence of configuration methods. As shown in Figure 7, GPT-4 consistently outperforms other models in all configurations, especially in zero-shot and LLM4ATS configurations, demonstrating exceptional accuracy and reliability. Qwen2.5-72B-Instruct also achieves high performance levels, approaching those of GPT-4 after optimization through LLM4ATS. In contrast, Qwen2.5-7B shows weaker baseline capabilities but experiences notable performance improvements through fine-tuning and optimization.

In general, the LLM4ATS configuration significantly enhances the pass rates of all models, with the most pronounced effects observed in more complex models. This underscores the advantages of optimized prompting, LoRA tuning, and syntax checking in improving model performance.

5.3. Question 3: Does Fine-Tuning with Proprietary Data Improve Performance in ATS Generation Tasks?

Introducing LoRA tuning resulted in slight pass rate improvements for certain models. However, the most significant improvements were observed in human quality metrics, where scripts generated using LLM4ATS demonstrated substantial improvements in correctness, readability, and adherence to industry coding standards. This indicates that fine-tuning with proprietary data effectively enhances the models’ capabilities in generating high-quality ATSs.

6. Conclusions

In this paper, we introduced LLM4ATS, an innovative framework that uses LLMs for the automated generation of ATSs within intelligent vehicle systems. This paper addresses the limitations of traditional script development by integrating user feedback and external verification tools, such as syntax checkers and real machine testing, into an iterative refinement process. This ensures that the generated scripts not only adhere to the strict syntax and functional requirements of ATSs but also significantly reduce the manual effort and expertise traditionally required.

The varying performance scores across different models and configurations can be attributed to the interplay between the inherent capabilities of the base LLMs and the degree of structural guidance provided by our LLM4ATS framework. Powerful proprietary models like GPT-4, with superior instruction-following and reasoning abilities, start from a higher baseline in zero-shot settings and can more effectively utilize the contextual clues and correction feedback from our pipeline, achieving the highest scores. In contrast, open-source models like Qwen2.5-7B show a greater relative performance gain from the framework, as the structured prompts and iterative corrections compensate for their weaker initial instruction comprehension. The convergence of the peak Pass@10 score at approximately 92% across all advanced models, even with our optimized pipeline, indicates a fundamental boundary. This boundary likely stems from a core set of challenging cases where the natural language specifications contain ambiguities or rely on implicit domain knowledge that is not fully codified in our current rule knowledge base, leading to consistent errors that cannot be resolved through syntax or signal validation alone.

Expert assessments further validated the effectiveness of the approach, demonstrating notable improvements in accuracy, readability, and adherence to industry coding standards in the generated scripts. These findings indicate that the pipeline not only automates the script generation process but also produces high-quality, maintainable, and reliable ATSs suitable for real-world applications.

The contributions in this paper pave the way for more efficient and reliable applications of LLMs in the automotive industry, particularly in the domain of automated testing. Future work may explore the integration of more advanced validation mechanisms, expansion to other specialized domains, and further enhancements to the capabilities of LLMs to handle increasingly complex testing scenarios.

7. Discussion

The LLM4ATS framework demonstrates a structured approach to generating domain-specific code through the integration of formal rule knowledge, iterative verification, and retrieval-augmented prompting. While developed and validated in the context of ATS generation, the underlying methodology offers significant implications for applying large language models to other specialized or low-resource programming domains where extensive training data is unavailable but formal specifications exist.

7.1. Generalization to Other Low-Resource or Rule-Based Languages

The core challenge addressed in this paper is the generation of correct code in a domain with strict syntactic and semantic constraints, but without sufficient representative data for conventional large-scale model training. This scenario is common in industrial and embedded systems programming, where many domain-specific languages (DSLs), configuration formats, or legacy scripting languages have precise rules but limited publicly available code corpora.

The workflow proposed in this paper—comprising a rule knowledge nase, RAG (RAG), and a closed-loop verification and correction mechanism—provides a blueprint for such contexts. For a new target language or format, the adaptation process would involve the following:

1.

Constructing a Structured Rule Knowledge Base: Formalizing the language’s syntax, allowed keywords, value type constraints, and structural patterns into a machine-readable format (e.g., schema files and grammar definitions).

2.

Building a Case Knowledge Base: Collecting a limited set of validated example pairs (natural language intent to correct code), which serves as exemplars for retrieval, not for mass training.

3.

Integrating a Domain-Specific Validator: Developing or configuring a lightweight checker (e.g., a parser, a linter, or a simulator) capable of diagnosing rule violations and generating explicit error messages.

The LLMs’s role shifts from memorizing patterns from massive data to reasoning with provided rules and examples. The RAG stage ensures the model is contextualized with relevant rules and similar examples, while the verification stage provides immediate, actionable feedback, enabling the model to correct itself even for unseen or rare combinations of requirements.

7.2. GPT vs. Qwen2.5

The empirical evaluation provides a concrete comparison between proprietary and open-source model families—specifically GPT series (3.5 and 4) and Qwen2.5 series (7B- and 72B-Instruct)—within the proposed workflow. This analysis yields insights for practitioners selecting models for similar industrial applications as shown in

Table 4. Performance and characteristic comparison: GPT vs. Qwen2.5 within the LLM4ATS framework.

Comparison Dimension	GPT Series (GPT-3.5, GPT-4)	Qwen2.5 Series (7B, 72B-Instruct)
Baseline Zero-Shot Performance	Significantly higher (GPT-4: 42%). Demonstrates stronger instruction understanding and generalized reasoning capability.	Relatively lower (Qwen2.5-72B: 28%). Shows weaker comprehension of specialized instructions without guidance.
Final Performance within LLM4ATS	Reaches performance ceiling (GPT-4: 91%). The framework effectively bridges the domain knowledge gap, allowing its full potential to be realized.	Substantial improvement (Qwen2.5-72B: 89%). Proves the framework can significantly unlock the domain-specific potential of open-source models.
Benefit from RAG and Verification	Benefits significantly. Strong reasoning ability enables efficient utilization of retrieved rules and cases, and precise execution of iterative corrections.	Benefit is crucial. It is a necessary precondition for reaching practical-grade performance. RAG provides critical domain context, and the verification loop offers explicit learning signals.
Impact of Fine-tuning (LoRA)	Not applicable (API mode). Relies on prompt engineering and workflow design for domain adaptation.	Provides stable gains. Even with small-scale proprietary data, it enhances baseline accuracy and output stability.
Industrial Deployment Considerations	Strengths: Maintenance-free, stable performance, simple API. Challenges: Data privacy, ongoing API costs, uncontrollable generation logic.	Strengths: Controllable data privacy, deep customization and optimization, no recurring inference costs. Challenges: Requires local GPU infrastructure, technical stack maintenance, certain threshold for model optimization.

.

Furthermore, the framework effectively equalizes the performance gap between model types. The structured knowledge injection and verification mechanism reduces the dependency on the model’s inherent, pre-trained knowledge of niche domains, allowing smaller or less specialized models to perform reliably. This suggests that for rule-dominated industrial tasks, investing in a robust framework can be as impactful as selecting the most powerful base model.

7.3. Limitations and Future Work

The current safety mechanism, which relies on syntax checkers and DBC file validation, ensures basic correctness but has inherent boundaries. Its effectiveness is directly tied to the completeness of the manually curated rule knowledge base and the diagnostic capability of the validator. Furthermore, the initial setup cost for a new domain is non-trivial.

A key limitation is that this approach cannot reliably detect high-level semantic or logical errors. A script may be syntactically perfect and use all signals correctly, yet still contain a command sequence that is logically flawed or potentially hazardous to the vehicle hardware (e.g., activating conflicting systems in an unsafe order). These errors require reasoning about the system’s state and behavior over time, which is beyond the scope of syntax and DBC checks.

To address this challenge and aim for more rigorous safety assurance, future work should explore integrating Formal Verification methods. Inspired by related work like LLM4PLC—which successfully employs model checkers (e.g., NuXmv) to verify temporal logic properties of generated industrial control code—similar techniques could be adapted. This would involve creating a formal model of the vehicle system or test protocol to mathematically verify critical safety and liveness properties of the ATSs before execution.