Next Article in Journal
Factors Influencing Step Ablation in the Expansion Section of a Composite Nozzle in a Solid Rocket Motor
Previous Article in Journal
Reducing Undesirable Vibration of Rotating Payload Connected to Spacecraft Using Magnetic Bearing
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Aerospace
Volume 12
Issue 6
10.3390/aerospace12060498
aerospace-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Using Large Language Models for Aerospace Code Generation: Methods, Benchmarks, and Potential Values

by
Rui He
1,2,*,
Liang Zhang
1,2,
Mengyao Lyu
1,2,
Liangqing Lyu
1,2 and
Changbin Xue
1,2
1
Key Laboratory of Electronics and Information Technology for Space System, National Space Science Center, Chinese Academy of Sciences, Beijing 100190, China
2
University of Chinese Academy of Sciences, Beijing 100049, China
*
Author to whom correspondence should be addressed.
Aerospace 2025, 12(6), 498; https://doi.org/10.3390/aerospace12060498
Submission received: 28 April 2025 / Revised: 23 May 2025 / Accepted: 26 May 2025 / Published: 30 May 2025
Browse Figures
Versions Notes

Abstract

In recent years, Large Language Models (LLMs) have witnessed rapid advancements, revolutionizing various domains. Within the realm of software development, code generation technology powered by LLMs has emerged as a prominent research focus. Despite its potential, the application of this technology in the aerospace sector remains in its nascent, exploratory phase. This paper delves into the intricacies of LLM-based code generation methods and explores their potential applications in aerospace contexts. It introduces RepoSpace, the pioneering warehouse-level benchmark test for code generation of spaceborne equipment. Comprising 825 samples from five actual projects, this benchmark offers a more precise evaluation of LLMs’ capabilities in aerospace scenarios. Through extensive evaluations of seven state-of-the-art LLMs on RepoSpace, the study reveals that domain-specific differences significantly impact the code generation performance of LLMs. Existing LLMs exhibit subpar performance in specialized warehouse-level code generation tasks for aerospace, with their performance markedly lower than that of domain tasks. The research further demonstrates that Retrieval Augmented Generation (RAG) technology can effectively enhance LLMs’ code generation capabilities. Additionally, the use of appropriate prompt templates can guide the models to achieve superior results. Moreover, high-quality documentation strings are found to be crucial in improving LLMs’ performance in warehouse-level code generation tasks. This study provides a vital reference for leveraging LLMs for code generation in the aerospace field, thereby fostering technological innovation and progress in this critical domain.

1. Introduction

The aerospace sector has stringent requirements for safety, reliability, and precision [1,2]. As space missions grow more complex, the software systems for onboard equipment have become more sophisticated and intricate. Serving as the “brain” behind the success of a mission, onboard software not only supports the basic operations of the equipment but also provides precise support in areas such as remote control, data transmission, attitude control, and scientific experiments [3].
Traditional development of aerospace software relies heavily on manual coding, formal verification, and iterative testing. This approach is time-consuming and labor-intensive. It is also prone to human error. Modern aerospace systems are becoming increasingly complex. They incorporate advanced capabilities such as autonomous decision making, real-time data processing, and adaptive control mechanisms. As a result, the need for efficient, fault-tolerant code generation methods has grown urgent [4].
Code generation can be classified into multiple granularity levels: symbol-level, statement-level, function-level, library-level, and software-level [4]. Current research on automatic code generation in aerospace engineering primarily uses traditional formal methods such as syntax trees and symbol tables. These methods effectively address symbol-level generation tasks. However, they show clear limitations when handling advanced tasks that require deep semantic understanding of requirements [5].
In recent years, LLMs have advanced rapidly, enhancing AI productivity across various fields [6,7,8]. In software development, LLM-based code generation has emerged as a key research area in computer science. This is due to its significant application potential and strategic impact on the future of the software industry. State-of-the-art LLMs like Codellama [9], Deepseek-coder [10], and specialized variants have demonstrated exceptional capabilities in automating code generation, debugging, and optimization for general programming tasks. These models leverage massive codebases and natural language datasets to generate context-aware solutions. Against this backdrop, a critical question arises: Can LLMs adapt to the unique challenges of aerospace software development?
Existing studies have explored LLM applications in code generation for web development and data science [11]. In contrast, LLM-based code generation in the aerospace field is still in its infancy, with limited progress to date. Nevertheless, its ability to handle complex requirements and produce optimized code indicates substantial potential value. Customizing these technologies to meet the specific needs of aerospace could revolutionize the development of flight control software, mission planning systems, and scheduling tools. This, in turn, would drive innovation in space technology.
The core research goal of this paper is to investigate the application potential of LLMs in aerospace code generation and address the challenges of automated code generation for complex spaceborne systems. To achieve this, we adopted a multi-stage methodology: first, we systematically summarize the technical framework of LLM-based code generation methods; second, we constructed a RepoSpace, the first repository-level benchmark for on-board device code generation, which contains 825 samples from five real-world projects to evaluate LLM performance in aerospace scenarios; and third, we conducted empirical studies on seven state-of-the-art LLMs using metrics such as CodeBLEU, DepMatch, and complexity, while exploring the impact of RAG technology and prompt engineering.
The remainder of this paper is organized as follows: Section 2 presents a review of the relevant literature; in Section 3, we propose and construct a repository-level benchmark for aerospace code generation; Section 4 describes the experimental setups for evaluating this benchmark; Section 5 presents the experimental results; Section 6 discusses the impact of documentation quality and domain specificity; Section 7 summarizes the main content and contributions of this study and highlights meaningful directions for future research.
Based on this, the main contributions of this paper are as follows:
  • Thoroughly summarized large LLM-based code-generation methods and explored their potential aerospace applications, aiming to offer insights for tackling aerospace challenges.
  • Introduction of the first repository-level benchmark for on-board device code generation, featuring 825 samples from 5 real-world projects to accurately assess LLM capabilities.
  • Conducted extensive evaluations of 10 state-of-the-art LLMs on the benchmark, clearly revealing their performance disparities in on-board device code generation, guiding LLM selection and improvement in aerospace.

2. Background

2.1. Large Language Model for Code Generation

Code generation refers to the automated process of producing source code based on user-defined constraints. Its primary objective is to automatically generate program code that satisfies specific requirements through diverse input modalities.
Early code generation methods relied on formal logical reasoning and rule-based systems, aiming to automatically derive code that complies with specific specifications [5,12,13,14,15]. Current research on automated code generation in aerospace engineering remains predominantly focused on such approaches [16,17]. However, these methods exhibit significant limitations when handling complex programs and dynamic programming tasks.
With the advancement of deep learning, neural network-based approaches have gradually become mainstream in code generation [18,19,20]. Through training on extensive program corpora, these neural models can learn program structures and patterns to a certain extent, thereby enabling more flexible code generation.
In recent years, Transformer-based large language models have significantly advanced program synthesis capabilities by learning from massive code and natural language datasets [8,21,22,23]. Through pre-training and fine-tuning, these models develop a profound understanding of code structure and semantics, demonstrating exceptional performance across various software development tasks.
Code generation technologies based on LLMs can be divided into four categories, as shown in Table 1.

2.2. Evaluation Metrics and Benchmarks for Code Generation

In the field of code generation research, the design of evaluation metrics directly impacts the objective assessment of generated code quality and practicality. Due to the fundamental differences between natural language and programming languages in terms of syntactic structure and semantic logic, constructing an effective and reliable automated evaluation system remains a critical challenge in this domain. This paper adopts an evaluation framework (as shown in Figure 1) from two dimensions: correctness and comprehensive quality, where correctness evaluation is further subdivided into similarity-based measurement and execution-based verification methods.
Similarity-based evaluation methods assess code quality by computing structural similarity between generated code and reference implementations [37]. Traditional natural language processing metrics like BLEU [38] evaluate textual similarity through n-gram overlap, while capable of rapidly determining surface-level code similarity, these metrics struggle to capture programming language-specific syntactic constraints and semantic functional differences. The Exact Match (EM) metric performs formal evaluation through strict matching, but its sensitivity to code formatting and syntactic details may lead to false negatives for valid solutions. To address this, CodeBLEU [39] incorporates abstract syntax tree matching and semantic data flow analysis while retaining the n-gram matching mechanism, establishing a multidimensional evaluation system encompassing syntactic structure, control flow logic, and variable dependencies.
Execution-based verification methods test functional correctness by actually running generated code [40,41,42]. The Pass@k metric employs Monte Carlo sampling to calculate the probability of the top-k candidate solutions passing unit tests, with its variants Pass@1 and Pass@5 focusing on evaluating the model’s first-attempt accuracy and fault tolerance, respectively. Execution accuracy quantitatively assesses test case coverage, directly reflecting code functional completeness. Compared to static similarity measurements, these methods demonstrate higher practical value in real development scenarios, though they are constrained by test case completeness and execution environment configuration costs.
The comprehensive quality indicators focus on assessing the non-functional attributes of code, and some evaluation dimensions still possess a certain degree of subjectivity. Common quantitative quality metrics include code complexity, code size, and code readability. Cyclomatic complexity is calculated by analyzing the syntax tree to determine the number of path nodes contained in the function’s control flow [43]. Code size is measured by analyzing tools that count the number of effective lines of code in a function [44]. For readability, some studies suggest using eye-tracking metrics to quantify the time and frequency with which code snippets are fixated [45].
Ensuring the correctness of the generated code is of paramount importance in the domain of code generation. However, most of these works concentrate on general domains (such as competitive programming, general SE, compiler, and data science), while paying less attention to specific domains (such as on-board equipment of spacecraft). Currently, popular code generation benchmarks can be divided into two categories:
  • Function-level benchmarks. These benchmarks contain human-written problems or requirements and require LLMs to generate independent code snippets.
  • Repository-level benchmarks. These benchmarks require LLMs to generate new programs according to the requirements and context within the current codebase. Compared with code snippet-level benchmarks, codebase-level benchmarks are more challenging and closer to real-world software development scenarios.
Functional baseline datasets are typically based on programming contest questions, algorithmic implementations, or specific programming tasks, and are centered on evaluating the model’s code-generation capabilities for specific functional requirements [46,47,48]. CodeNet, the largest such benchmark, contains over 14 million code samples across 55 languages. ACEOB focuses on Python code efficiency optimization with novel metrics like IOCCB and NPI, while PIE assesses C++ performance optimization through gem5 simulators and adaptive strategies. HumanEval, a widely used Python benchmark with 164 hand-crafted problems, evaluate functional correctness from docstrings and inspired AiXBench for Java evaluation with 336 total problems and new assessment metrics.
Unlike function-level, repository-level benchmarks focus more on the context information of the code [41,49,50]. DS-1000 includes 1000 real-world Python data science problems across seven major libraries, modified to prevent data leakage. Concode collects 100,000+ Java functions with contextual dependencies from GitHub repositories, paired with natural language descriptions, though it relies solely on BLEU scores without functional correctness evaluation.

3. Benchmark-RepoSpace

3.1. Overview

RepoSpace is a benchmark dataset consisting of 825 code samples collected from five real-world spaceborne software projects, designed to evaluate LLMs for repository-level code generation in spaceborne systems. The evaluation framework comprises two main phases: (1) LLM-based code generation and (2) post-generation assessment, as detailed in their respective sections.
Figure 2 shows a sample in RcepoSpce. Each sample consists of five components. During the code generation phase, each evaluated LLM receives three types of inputs: the function signature (1), natural language requirements (2), referred to as “annotations” in later sections, and relevant codebase context (3+4) including external dependencies such as interfaces, functions and variables referenced by but defined outside the target function. Based on these inputs, the LLM generates a candidate function implementation (5) that should satisfy the specified requirements.
The subsequent assessment phase evaluates two key aspects of each generated function: its functional correctness in implementing the requirements and its readiness for integration into new operational contexts. This evaluation considers three critical dimensions: compliance with the required function signature, fulfillment of the specified requirements, and proper handling of codebase dependencies. The systematic evaluation protocol enables comprehensive benchmarking of LLMs for developing mission-critical spaceborne software systems.

3.2. Benchmark Construction

As illustrated in Figure 3, the construction of RepoSpace involves four meticulously designed stages that collectively ensure dataset robustness and diversity, specifically addressing the complexities of code generation for spaceborne equipment. The outcomes of these phases systematically handle critical aspects ranging from project selection to contextual parsing.

3.2.1. Project Selection

To ensure RepoSpace’s practicality and diversity, we adopt established best practices [48,51] for function selection across heterogeneous projects. Our methodology comprises two phases: First, we collaborate with experienced R&D personnel to identify four validated projects from operational repositories. Subsequently, we apply three filtering criteria to real-world functions: (1) we exclude trivial functions with fewer than five lines of code; (2) we eliminate standalone functions; and (3) we remove non-standalone functions lacking essential interface information.

3.2.2. Function Parsing

We extract all functions from selected projects using a customized Solidity-compatible Tree-sitter parser, developed to overcome the limitations of the native implementation. This enhanced parser accurately extracts critical elements including function identifiers, bodies, and requirements. Applying the selection principles from Section 4.1, we obtain 825 representative function samples across diverse projects.

3.2.3. Manual Annotation

Given the critical role of prompts in LLM performance and the significant impact of input requirements on code generation quality, we established a rigorous annotation protocol [52,53]. Three master’s students with aerospace backgrounds and ≥3 years of software development experience participate in the annotation process. To address the labor-intensive nature of manual requirement specification, we employ LLMs for preliminary requirement generation using structured one-shot prompts that enforce specific formatting conventions (functional descriptions with input/output parameters). Final annotations undergo meticulous human verification.
Our dual motivation for incorporating manual annotations includes (1) mitigating LLM memorization effects, as original comments may have been exposed during pretraining and (2) ensuring high-quality documentation for RepoSpace functions.
To validate the quality control of annotated functional descriptions, we conducted a multi-annotator consensus evaluation using Fleiss’ Kappa [54]. Based on systematic annotation guidelines, categorized annotations into four quality tiers: complete, partially complete, ambiguous, and unlabeled entries. Through statistical analysis of observed agreement rates (Po) versus theoretical random agreement rates (Pe) under the assumption of independent classification decisions, Fleiss’ Kappa quantifies inter-annotator consensus on a scale from perfect agreement ( κ = 1 ) to chance-level agreement ( κ  = 0). Empirical results demonstrate a Kappa coefficient ( 0.75 ≤ κ ≤ 1 ) within the excellent agreement range, indicating robust consensus among annotators.

3.2.4. Contextual Parsing

To address prevalent token undefined errors in repository-level code generation and ensure model comprehension verification, we implement efficient contextual dependency analysis [55]. Following [41], we define context dependencies as external code elements (functions, variables, and interfaces) required for function execution. Our two-phase program analysis methodology is as follows:
  • Source file retrieval and parsing to obtain definition lists (types, functions, variables, constants)
  • Static analysis to identify external calls and extract their signatures
The resulting context signatures are stored with corresponding function samples, balancing completeness with computational efficiency [55].

4. Experimental Setup

4.1. Research Questions

RQ-1: What is the impact of domain differences on LLMs code generation?
To investigate the impact of domain differences on the performance of large code models, we selected a general domain repository-level benchmark, ComplexCodeEval, as a control group. We employed two open-source models, DeepSeek-coder and CodeLlama, to evaluate the generation results of models with different parameter sizes in a zero-shot manner. By calculating CodeBLEU, we assessed the performance of LLMs on both ComplexCodeEval and RepoSpace.
RQ-2: How do different LLMs perform in generating non-standalone functions?
To further analyze the repository-level code generation capabilities of LLMs in RepoSpace, we utilized the RAG method to retrieve similar functions and their context, setting the prompt as one-shot. Subsequently, we generated code results using different LLMs and analyzed their performance across four metrics.
RQ-3: How do different prompting strategies affect the effectiveness of these models?
To explore the influence of different prompt templates on model generation effectiveness, we used RAG and lens settings consistent with RQ2 to obtain generation results for the same requirement under different template configurations. We then analyzed and presented the differences across two metrics.

4.2. Metrics

In aerospace software engineering practice, due to strict development process control and security verification requirements, unit test case construction is often separated from core functional implementation and is often tightly coupled to a specific mission. This results in project source code lacking a common domain test case foundation. To address this test resource-constrained scenario, we adopted the following evaluation strategies:
  • For functional correctness verification, we adopted the static evaluation methods Rouge-L and CodeBLEU based on similarity.
  • For code quality assessment, we use the measured complexity.
  • Innovatively, we propose the DepMatch metric to statically analyze the call accuracy of mission-critical libraries in generated code.
In addition to correctness and quality, the ability of the model to correctly invoke dependencies from the project context is equally important in code-generation tasks at the repository level. Therefore, we propose the Depmatch metric to quantify the generated code’s ability to call reference dependencies and thus assess the model’s accuracy in dependency management. Depmatch can be calculated as follows:
D e p M a t c h = 1 N ∑ i = 1 N R i ∩ P i R i
The LLM generates candidate programs for a given set of requirement descriptions N. For each requirement i, we employ static analysis tools to extract both its dependency sets P. These are then compared against the reference dependency set R, which is typically determined through manual annotation or project build files. Where |·| means the number of elements of a set.

4.3. Implementation Details

We selected 7 state-of-the-art LLMs that have been widely adopted in recent code generation research [42,55,56]. Considering the coding requirements of embedded spaceborne software, we focused on models that can simultaneously support code generation in both C and C++. This is because embedded systems typically require efficient and reliable code implementations. C and C++ are the most commonly used programming languages in this domain, providing direct control over hardware and high-performance execution efficiency. Additionally, we paid special attention to the latest models released since 2022. Due to functional limitations, our selection excludes smaller models with fewer than 1 billion parameters. Table 2 presents the seven cutting-edge LLMs included in our experiments, along with their respective sizes and architectures.
In selecting these models, we considered multiple factors to ensure that they effectively support our research objectives. First, the chosen models have demonstrated excellent performance across various benchmark tests and have a solid performance record, as thoroughly validated in the relevant literature. Furthermore, our selection encompasses a diverse range of models, characterized by several key features: (1) we prioritized open-source models due to concerns about safety and reproducibility; (2) the selected models range in the parameter size from 6.7 billion to 671 billion, covered different scales to analyze the potential impact of the model’s size on code generation quality; (3) we included models suitable for both general tasks and those specifically designed for code generation and understanding, ensuring that our experiments can effectively evaluate the performance of different types of models in specific tasks.
During the inference phase, we maintained consistent experimental Settings in all models: the maximum context window was 1024 tokens, and the temperature was set to 0. A greedy sampling strategy was used to generate a single output for each requirement.

5. Result

5.1. RQ1—What Is the Impact of Domain Differences on LLMs Code Generation?

Figure 4 shows the CodeBLEU at the repository level of LLMS studied using the greedy sampling method on RepoSpace and ComplexCodeEval. We used the two series, DeepSeek-Coder and CodeLlama, as evaluation models and directly adopted the CodeBLEU results [57] on ComplexCodeEval in the previous work. Based on Figure 4, we obtained the following observations.
Compared with the existing stored-level benchmark ComplexCodeEval, the correctness of the studied model on our repository benchmark RepoSpace has decreased significantly. In particular, the best-performing model CodeLlama-34B has a CodeBLEU score of 34.08% in ComplexCodeEval, but its CodeBLEU score on the library-level code generation task of RepoSpace is only 18.03%. Similar trends can also be observed in other models. This decline may be attributed to the complexity of generating code that depends on other contexts, which is more challenging than generating independent code. This discovery is consistent with the recent work of [41]. In conclusion, our results indicate that existing LLMS still have limited performance in addressing complex coding tasks, such as stored-level code generation.
Furthermore, we observe that the performance of the model in the general domain code generation task does not necessarily reflect its domain-specific stored-level code generation capability. For example, on ComplexCodeEval, the CodeBLEU metric of CodeLlama-34B (34.08%) performs similarly to that of DeepSeek-Coder-33B (33.96%) and is slightly higher than the latter. However, there is a certain gap in the performance of the stored-level code generation task of RepoSpace. This further confirms the necessity of constructing a class-level code generation benchmark.
To sum up, compared with the general domain, the performance of existing LLMS in the aerospace-specific domain is much lower. Furthermore, the stored-level coding capability in the general domain cannot be equivalently represented by the stored-level coding capability in large language models in the aerospace field. These findings strongly confirm the motivation and necessity of building a benchmark for warehouse library-level code generation in the aerospace field.

5.2. RQ2—How Do Different LLMs Perform in Generating Non-Standalone Functions?

Table 3 shows the overall performance of the state-of-the-art large language model on RepoSpace.
The performance of these LLMs on RepoSpace dropped dramatically compared to previous benchmarks. For example, the highest CodeBLEU score on the domain-specific codebase level benchmark [58] is 60.39%. In comparison, CodeLlama’s CodeBLEU on RcepoSpace only reached 16.73%. These drops indicate that RepoSpace is more challenging, and that there may have been a data breach in the previous benchmark. Specifically, DeepSeek-Coder implemented the highest Rouge-L and CodeBLEU in the 6.7B to 7B models, surpassing the others. Notably, DeepSeek-R1-Distill-Qwen-7B claims performance comparable to ChatPGT-O1-mini in benchmarks such as LiveCodeBench and CodeForces. However, its performance is not as good as the CodeLlama-7B. This discrepancy may be due to DeepSeek-R1-Distill’s lack of knowledge about on-board data processing tasks, which highlights the importance of professional benchmarks like RepoSpace. In models 32B through 34B, Qwen2.5-Coder outperforms the others in both Rouge-L and CodeBLEU. Overall, DeepSeek-V3 performed best, with CodceBLEU reaching 33.48%. Notably, DeepSeek-R1-Distill-Qwen-32B is significantly better than its 7B counterpart.
In the RAG system, code search is a core component, and there are currently two main approaches: information retrieval (IR) based methods and deep learning (DL) based methods. Typically, IR-based code search engines first index specific fields of code snippets within the codebase, and then retrieve similar code that matches the query based on their scoring algorithms. Deep learning-based code search techniques utilize deep neural networks to convert queries and code snippets from the codebase into feature vectors. After obtaining the vector representations, the common practice is to calculate the cosine similarity between the query and the code snippets.
Retrieval-Augmented Generation enhances the generation model through the retrieved information and achieves gratifying results in code generation [34,59]. Also at the repository level of code generation, LLMs benefit from more code context. We treated the current code repository as a retrieval corpus and have constructed an RAG system based on deep learning techniques. Then, we selected CodeBERT as the semantic retriever. This model takes a requirement as input and retrieves the most similar code snippets from the code repository. The retrieved code snippets, along with their corresponding dependencies, are filled into a template to provide richer context for the generation model, thereby enhancing the accuracy of the inference. Then, we used tree-sitter to parse and obtain the list of called functions from the generated code, and compare it with the list of called functions in the original requirement implementation code. The results are calculated based on the DepMatch metric formula described in Section 4.2. In repository-level code generation tasks, in addition to correctness and quality, the model’s ability to correctly call dependencies from the project context is equally important. Discussing the actual values of DepMatch can better reflect this cross-file calling capability.
Without the example and context as a hint, the DepMatch of LLMs is all less than 1, making it difficult to call the repository by requirement. The pre-training data of LLM is usually from public data, which is difficult to cover the sensitive areas, such as spaceborne data processing missions in the space field. At the same time, due to the coding styles and domain names of different developers, it is difficult for LLMs to directly infer the correct function name from the requirement description.
Figure 5 presents a unique success case. In the absence of context, the generative model fabricated a non-existent function call, resulting in incorrect code. In reality, there was a function for writing to a memory address provided in the codebase. After introducing the relevant code, the generative model successfully called the appropriate function and generated the correct code.
With the introduction of similar functions and contexts, CodeBLEU and DepMatch of LLMs have significantly improved. For example, DeepSeek-Coder’s DepMatch improved by 30.32% after introducing examples and context. This similar phenomenon appeared in previous studies [58]. We attribute these improvements to the related algorithms and dependencies in similar functions and the domain knowledge contained in the context. This inspired the researchers to explore more advanced RAG techniques to improve warehouse-level code generation.

5.3. RQ3—How Do Different Prompting Templates Affect the Effectiveness of These Models?

Considering the excellent performance of LLMs in small sample learning and prompt, it is important to examine the impact of different prompt templates and prompt word quality. Therefore, in this RQ, we first analyze the impact of different prompt templates, and then examine the performance of LLMs when code generation is performed in a manually annotated manner.
We used the best-performing model in RQ2 (i.e., DeepSeek), and to investigate the impact of prompt templates, we examined three different features that distinguish different prompt templates, as shown in Figure 6.
  • “Detailed/Brief” refers to whether a detailed description of the task is provided. For example, the first pink instruction in Figure 6 is “detailed” because it provides specific information about the input format. In contrast, the “brief” description of the second pink directive provides only the basic requirements.
  • Single step/Step by step refers to whether additional instructions are added before the prediction. For example, the first blue instruction prompt (i.e., “Step by Step”) further specifies the model-generating code based on the above example, while the second blue instruction prompt (i.e., “Single Step”) does not contain this information.
DeepSeek-Coder shows robust performance on retrieval-enhanced code-generation tasks with multiple types of prompt templates. We experimented with all the prompt template feature combinations, and the results are shown in Table 4. We found that DeepSeek-Coder achieves improvements in code generation through prompt templates with various characteristics.
In the template mix, the “detailed, Single Step” template performed poorly, while the “Detailed, Step by Step” prompt template ranked highest in both metrics. Other prompt templates behave similarly. These results suggest that DeepSeek-Coder is generally robust when generating code using our retrieval enhancement framework and is likely to be less affected by changes in prompt templates. This phenomenon may be due to DeepSeek-Coder’s strong contextual understanding ability, enabling it to effectively extract key information across different prompt templates. At the same time, the model’s training process includes a diverse range of tasks and prompt formats, allowing it to flexibly utilize various input formats for reasoning, thereby enhancing its robustness.

6. Discussion

Here, we mainly discuss the performance differences in different document strings. As described in Section 3.2.3, we used an LLM (DeepSeek-Coder-671B in this article) to automatically generate the documentation string of the function and manually verify the quality of the comments. In this section, we evaluate the LLM using the original document string and the manually labeled document string. Figure 7 shows the performance of different LLMS when using the original document string and the manually labeled document string.
As shown in Figure 7, compared with using the original document string, when using the manually labeled document string, both the CodeBLEU and Rouge-L scores of LLM have been significantly improved. However, at the same time, it will also lead to an increase in the overall complexity of the code.
This phenomenon can mainly be attributed to two reasons. Firstly, automatically generated document strings are usually more detailed and accurate than the original document strings, providing LLMS with comprehensive information about functions and helping the model better understand and generate code. Secondly, the quality of the original document strings varies greatly. Many document strings are relatively short and unclear, which limits the performance of the model in the code generation task. On the contrary, the automatically generated documentation strings that have been carefully designed and iteratively optimized can describe the objective function more accurately, thereby significantly improving the performance of the model.
As shown in Figure 7d, the DepMatch of LLMs significantly increases after introducing context. For instance, the DepMatch of DeepSeek-V3 improved by 151% under both settings. A similar phenomenon was also observed in previous studies [50]. From the results, responses generated using the original document strings almost never match the correct dependency names, while responses generated using manually annotated document strings surprisingly call a small number of actual dependencies correctly. This may be related to the use of aerospace domain-specific terminology in the latter to describe functional features. We attribute the improvement to the domain knowledge included in the context.
Overall, these results indicate that the use of high-quality document strings is crucial for improving the performance of LLMS in stored-level code generation tasks. Future research can further explore how to optimize the process of generating document strings to further improve the performance of LLMS in code-related tasks.

7. Conclusions

In this paper, we initiate our exploration into the potential benefits of utilizing LLMs for code generation within the aerospace domain. We then introduce RepoSpace, an innovative benchmark specifically designed to evaluate the performance of LLMs in complex aerospace development scenarios. Unlike existing general benchmarks, all code data in RepoSpace is sourced from verified space missions, ensuring its authenticity and relevance. To enhance the comprehensiveness and professionalism of our dataset, we have implemented rigorous automated data processing procedures alongside manual information annotation. Finally, we conducted a thorough investigation into the performance of several state-of-the-art LLMs in generating repository-level code pertinent to the aerospace sector.
Our findings revealed that domain-specific variations significantly influence the code generation capabilities of LLMs. Current LLMs face considerable challenges when tasked with aerospace-specific repository-level code generation, demonstrating markedly lower performance compared to their effectiveness; in general, domain tasks. Notably, while zero-shot learning approaches exhibit limitations, RAG proves to be an effective strategy for improving the code generation performance of LLMs by leveraging relevant contextual information. Furthermore, although the enhancement in performance is modest, employing an appropriate prompt template can facilitate improved outcomes from LLMs.
This research contributes to the field by establishing RepoSpace as a specialized benchmark for assessing LLM performance in the aerospace domain and elucidating the impact of domain-specific tasks on model capabilities. Through a systematic analysis of LLM performance in aerospace code generation, we provide valuable insights for future research, underscoring the importance of customizing models and datasets for specialized domain applications. At the same time, our findings also highlight certain limitations, particularly concerning the availability of testing resources. The current evaluation predominantly focuses on static code similarity and the accuracy of dependent calls, which constrains the ability to perform dynamic verification of the generated code within real-world aerospace environments. Addressing this challenge will be a pivotal direction for our future research endeavors.

Author Contributions

Conceptualization, R.H. and L.L.; methodology, R.H.; software, R.H. and L.Z.; formal analysis, R.H.; resources, C.X.; data curation, R.H., L.Z. and M.L.; writing—original draft preparation, R.H.; writing—review and editing, R.H. and L.L.; visualization, R.H. and L.Z.; resources, C.X. All authors have read and agreed to the published version of the manuscript.

Funding

This work was supported by Strategic Priority Research Program of the Chinese Academy of Sciences, cultivated by the Space Science Laboratory of Beijing Huai rou Comprehensive National Science Center under grant XDA15000000.

Data Availability Statement

The data presented in this study are available on request from the corresponding author.

Conflicts of Interest

The authors declare no conflicts of interest.

References

  1. Yang, M.; Gu, B.; Guo, X.; Dong, X.; Wang, Z.; Chen, R. Aerospace embedded software dependability guarantee technology and application. Sci. Sin. (Technol.) 2015, 45, 198–203. [Google Scholar]
  2. Yang, M.; Gu, B.; Duan, Z.; Jin, Z.; Zhan, N.; Dong, Y.; Tian, C.; Li, G.; Dong, X.; Li, X. Intelligent program synthesis framework and key scientific problems for embedded software. Chin. Space Sci. Technol. 2022, 42, 1–7. [Google Scholar] [CrossRef]
  3. Yung, K.L.; Tang, Y.M.; Ip, W.H.; Kuo, W.T. A systematic review of product design for space instrument innovation, reliability, and manufacturing. Machines 2021, 9, 244. [Google Scholar] [CrossRef]
  4. Chen, X.; Gao, F.; Han, X.; Weihua, M. Large model code generation technology and its potential applications to aerospace. Aerosp. Control 2025, 43, 8–16. [Google Scholar]
  5. Alnusair, A.; Zhao, T.; Yan, G. Rule-based detection of design patterns in program code. Int. J. Softw. Tools Technol. Transf. 2014, 16, 315–334. [Google Scholar] [CrossRef]
  6. Shen, S.; Zhu, X.; Dong, Y.; Guo, Q.; Zhen, Y.; Li, G. Incorporating domain knowledge through task augmentation for front-end JavaScript code generation. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Singapore, 14–18 November 2022; pp. 1533–1543. [Google Scholar]
  7. Fried, D.; Aghajanyan, A.; Lin, J.; Wang, S.; Wallace, E.; Shi, F.; Zhong, R.; Yih, W.T.; Zettlemoyer, L.; Lewis, M. Incoder: A generative model for code infilling and synthesis. arXiv 2022, arXiv:2204.05999. [Google Scholar]
  8. Nijkamp, E.; Pang, B.; Hayashi, H.; Tu, L.; Wang, H.; Zhou, Y.; Savarese, S.; Xiong, C. Codegen: An open large language model for code with multi-turn program synthesis. arXiv 2022, arXiv:2203.13474. [Google Scholar]
  9. Roziere, B.; Gehring, J.; Gloeckle, F.; Sootla, S.; Gat, I.; Tan, X.E.; Adi, Y.; Liu, J.; Sauvestre, R.; Remez, T.; et al. Code llama: Open foundation models for code. arXiv 2023, arXiv:2308.12950. [Google Scholar]
  10. Zhu, Q.; Guo, D.; Shao, Z.; Yang, D.; Wang, P.; Xu, R.; Wu, Y.; Li, Y.; Gao, H.; Ma, S.; et al. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence. arXiv 2024, arXiv:2406.11931. [Google Scholar]
  11. Zhang, Z.; Wang, Y.; Wang, C.; Chen, J.; Zheng, Z. Llm hallucinations in practical code generation: Phenomena, mechanism, and mitigation. arXiv 2024, arXiv:2409.20550. [Google Scholar]
  12. Park, D.H.; Kim, S.D. XML rule based source code generator for UML CASE tool. In Proceedings of the Eighth Asia-Pacific Software Engineering Conference, Macao, China, 4–7 December 2001; pp. 53–60. [Google Scholar]
  13. Koziolek, H.; Burger, A.; Platenius-Mohr, M.; Rückert, J.; Abukwaik, H.; Jetley, R.; P, A.P. Rule-based code generation in industrial automation: Four large-scale case studies applying the cayenne method. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Software Engineering in Practice, Seoul, Republic of Korea, 5–11 October 2020; pp. 152–161. [Google Scholar]
  14. Jugel, U.; Preußner, A. A case study on API generation. In Proceedings of the System Analysis and Modeling: About Models: 6th International Workshop, SAM 2010, Oslo, Norway, 4–5 October 2010; Revised Selected Papers 6; pp. 156–172. [Google Scholar]
  15. Kolovos, D.S.; García-Domínguez, A.; Rose, L.M.; Paige, R.F. Eugenia: Towards disciplined and automated development of GMF-based graphical model editors. Softw. Syst. Model. 2017, 16, 229–255. [Google Scholar] [CrossRef]
  16. Märtin, L.; Schatalov, M.; Hagner, M.; Goltz, U.; Maibaum, O. A methodology for model-based development and automated verification of software for aerospace systems. In Proceedings of the 2013 IEEE Aerospace Conference, Big Sky, MT, USA, 2–9 March 2013; pp. 1–19. [Google Scholar]
  17. Deshmukh, M.; Weps, B.; Isidro, P.; Gerndt, A. Model driven language framework to automate command and data handling code generation. In Proceedings of the 2015 IEEE Aerospace Conference, Big Sky, MT, USA, 7–14 March 2015; pp. 1–9. [Google Scholar]
  18. Sutskever, I. Sequence to Sequence Learning with Neural Networks. arXiv 2014, arXiv:1409.3215. [Google Scholar]
  19. Sun, Z.; Zhu, Q.; Mou, L.; Xiong, Y.; Li, G.; Zhang, L. A grammar-based structural cnn decoder for code generation. In Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA, 27 January–1 February 2019; Volume 33, pp. 7055–7062. [Google Scholar]
  20. Svyatkovskiy, A.; Deng, S.K.; Fu, S.; Sundaresan, N. Intellicode compose: Code generation using transformer. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual, 8–13 November 2020; pp. 1433–1443. [Google Scholar]
  21. Feng, Z.; Guo, D.; Tang, D.; Duan, N.; Feng, X.; Gong, M.; Shou, L.; Qin, B.; Liu, T.; Jiang, D.; et al. Codebert: A pre-trained model for programming and natural languages. arXiv 2020, arXiv:2002.08155. [Google Scholar]
  22. Ni, C.; Yin, X.; Yang, K.; Zhao, D.; Xing, Z.; Xia, X. Distinguishing look-alike innocent and vulnerable code by subtle semantic representation learning and explanation. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, San Francisco, CA, USA, 3–9 December 2023; pp. 1611–1622. [Google Scholar]
  23. Yin, X.; Ni, C.; Wang, S. Multitask-based evaluation of open-source llm on software vulnerability. IEEE Trans. Softw. Eng. 2024, 50, 3071–3087. [Google Scholar] [CrossRef]
  24. Ahmad, W.U.; Chakraborty, S.; Ray, B.; Chang, K.W. Unified pre-training for program understanding and generation. arXiv 2021, arXiv:2103.06333. [Google Scholar]
  25. Wang, Y.; Le, H.; Gotmare, A.D.; Bui, N.D.; Li, J.; Hoi, S.C. Codet5+: Open code large language models for code understanding and generation. arXiv 2023, arXiv:2305.07922. [Google Scholar]
  26. Luo, Z.; Xu, C.; Zhao, P.; Sun, Q.; Geng, X.; Hu, W.; Tao, C.; Ma, J.; Lin, Q.; Jiang, D. Wizardcoder: Empowering code large language models with evol-instruct. arXiv 2023, arXiv:2306.08568. [Google Scholar]
  27. Houlsby, N.; Giurgiu, A.; Jastrzebski, S.; Morrone, B.; De Laroussilhe, Q.; Gesmundo, A.; Attariyan, M.; Gelly, S. Parameter-efficient transfer learning for NLP. In Proceedings of the International Conference on Machine Learning, Long Beach, CA, USA, 9–15 June 2019; pp. 2790–2799. [Google Scholar]
  28. Li, X.L.; Liang, P. Prefix-tuning: Optimizing continuous prompts for generation. arXiv 2021, arXiv:2101.00190. [Google Scholar]
  29. Liu, H.; Tam, D.; Muqeeth, M.; Mohta, J.; Huang, T.; Bansal, M.; Raffel, C.A. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Adv. Neural Inf. Process. Syst. 2022, 35, 1950–1965. [Google Scholar]
  30. Dettmers, T.; Pagnoni, A.; Holtzman, A.; Zettlemoyer, L. Qlora: Efficient finetuning of quantized llms. Adv. Neural Inf. Process. Syst. 2023, 36, 10088–10115. [Google Scholar]
  31. Li, J.; Li, G.; Li, Y.; Jin, Z. Structured chain-of-thought prompting for code generation. ACM Trans. Softw. Eng. Methodol. 2025, 34, 1–23. [Google Scholar] [CrossRef]
  32. Schäfer, M.; Nadi, S.; Eghbali, A.; Tip, F. An empirical evaluation of using large language models for automated unit test generation. IEEE Trans. Softw. Eng. 2023, 50, 85–105. [Google Scholar] [CrossRef]
  33. Jiang, X.; Dong, Y.; Wang, L.; Fang, Z.; Shang, Q.; Li, G.; Jin, Z.; Jiao, W. Self-planning code generation with large language models. ACM Trans. Softw. Eng. Methodol. 2024, 33, 1–30. [Google Scholar] [CrossRef]
  34. Li, J.; Li, Y.; Li, G.; Jin, Z.; Hao, Y.; Hu, X. Skcoder: A sketch-based approach for automatic code generation. In Proceedings of the 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), Melbourne, Australia, 14–20 May 2023; pp. 2124–2135. [Google Scholar]
  35. Zhang, X.; Zhou, Y.; Yang, G.; Chen, T. Syntax-aware retrieval augmented code generation. In Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, 6–10 December 2023; pp. 1291–1302. [Google Scholar]
  36. Zhang, K.; Zhang, H.; Li, G.; Li, J.; Li, Z.; Jin, Z. Toolcoder: Teach code generation models to use api search tools. arXiv 2023, arXiv:2305.04032. [Google Scholar]
  37. Lin, C.Y. Rouge: A package for automatic evaluation of summaries. In Proceedings of the Text Summarization Branches out, Barcelona, Spain, 25–26 July 2004; pp. 74–81. [Google Scholar]
  38. Papineni, K.; Roukos, S.; Ward, T.; Zhu, W.J. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, Philadelphia, PA, USA, 6–12 July 2002; pp. 311–318. [Google Scholar]
  39. Ren, S.; Guo, D.; Lu, S.; Zhou, L.; Liu, S.; Tang, D.; Sundaresan, N.; Zhou, M.; Blanco, A.; Ma, S. Codebleu: A method for automatic evaluation of code synthesis. arXiv 2020, arXiv:2009.10297. [Google Scholar]
  40. Zeng, Z.; Wang, Y.; Xie, R.; Ye, W.; Zhang, S. Coderujb: An executable and unified java benchmark for practical programming scenarios. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, Vienna, Austria, 16–20 September 2024; pp. 124–136. [Google Scholar]
  41. Yu, H.; Shen, B.; Ran, D.; Zhang, J.; Zhang, Q.; Ma, Y.; Liang, G.; Li, Y.; Wang, Q.; Xie, T. Codereval: A benchmark of pragmatic code generation with generative pre-trained models. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, Lisbon, Portugal, 14–20 April 2024; pp. 1–12. [Google Scholar]
  42. Li, Y.; Choi, D.; Chung, J.; Kushman, N.; Schrittwieser, J.; Leblond, R.; Eccles, T.; Keeling, J.; Gimeno, F.; Dal Lago, A.; et al. Competition-level code generation with alphacode. Science 2022, 378, 1092–1097. [Google Scholar] [CrossRef] [PubMed]
  43. Choi, J.; An, G.; Yoo, S. Iterative refactoring of real-world open-source programs with large language models. In Proceedings of the International Symposium on Search Based Software Engineering, Porto de Galinhas, Brazil, 15 July 2024; pp. 49–55. [Google Scholar]
  44. Cummins, C.; Seeker, V.; Grubisic, D.; Elhoushi, M.; Liang, Y.; Roziere, B.; Gehring, J.; Gloeckle, F.; Hazelwood, K.; Synnaeve, G.; et al. Large language models for compiler optimization. arXiv 2023, arXiv:2309.07062. [Google Scholar]
  45. Li, Z.; He, Y.; He, L.; Wang, J.; Shi, T.; Lei, B.; Li, Y.; Chen, Q. FALCON: Feedback-driven Adaptive Long/short-term memory reinforced Coding Optimization system. arXiv 2024, arXiv:2410.21349. [Google Scholar]
  46. Puri, R.; Kung, D.S.; Janssen, G.; Zhang, W.; Domeniconi, G.; Zolotov, V.; Dolby, J.; Chen, J.; Choudhury, M.; Decker, L.; et al. Codenet: A large-scale ai for code dataset for learning a diversity of coding tasks. arXiv 2021, arXiv:2105.12655. [Google Scholar]
  47. Pan, Y.; Shao, X.; Lyu, C. Measuring code efficiency optimization capabilities with ACEOB. J. Syst. Softw. 2025, 219, 112250. [Google Scholar] [CrossRef]
  48. Chen, M.; Tworek, J.; Jun, H.; Yuan, Q.; Pinto, H.P.D.O.; Kaplan, J.; Edwards, H.; Burda, Y.; Joseph, N.; Brockman, G.; et al. Evaluating large language models trained on code. arXiv 2021, arXiv:2107.03374. [Google Scholar]
  49. Lai, Y.; Li, C.; Wang, Y.; Zhang, T.; Zhong, R.; Zettlemoyer, L.; Yih, W.T.; Fried, D.; Wang, S.; Yu, T. DS-1000: A natural and reliable benchmark for data science code generation. In Proceedings of the International Conference on Machine Learning, Honolulu, HI, USA, 23–29 July 2023; pp. 18319–18345. [Google Scholar]
  50. Li, J.; Li, G.; Zhao, Y.; Li, Y.; Liu, H.; Zhu, H.; Wang, L.; Liu, K.; Fang, Z.; Wang, L.; et al. Deveval: A manually-annotated code generation benchmark aligned with real-world code repositories. arXiv 2024, arXiv:2405.19856. [Google Scholar]
  51. Liu, J.; Xia, C.S.; Wang, Y.; Zhang, L. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Adv. Neural Inf. Process. Syst. 2023, 36, 21558–21572. [Google Scholar]
  52. Jiang, J.; Li, Z.; Qin, H.; Jiang, M.; Luo, X.; Wu, X.; Wang, H.; Tang, Y.; Qian, C.; Chen, T. Unearthing Gas-Wasting Code Smells in Smart Contracts with Large Language Models. IEEE Trans. Softw. Eng. 2024, 51, 879–903. [Google Scholar] [CrossRef]
  53. Zhou, Y.; Muresanu, A.I.; Han, Z.; Paster, K.; Pitis, S.; Chan, H.; Ba, J. Large language models are human-level prompt engineers. In Proceedings of the Eleventh International Conference on Learning Representations, Virtual Event, 25–29 April 2022. [Google Scholar]
  54. Fleiss, J.L. Measuring nominal scale agreement among many raters. Psychol. Bull. 1971, 76, 378. [Google Scholar] [CrossRef]
  55. Liao, D.; Pan, S.; Sun, X.; Ren, X.; Huang, Q.; Xing, Z.; Jin, H.; Li, Q. A 3-CodGen: A Repository-Level Code Generation Framework for Code Reuse with Local-Aware, Global-Aware, and Third-Party-Library-Aware. IEEE Trans. Softw. Eng. 2024, 50, 3369–3384. [Google Scholar] [CrossRef]
  56. Khan, M.A.M.; Bari, M.S.; Do, X.L.; Wang, W.; Parvez, M.R.; Joty, S. xcodeeval: A large scale multilingual multitask benchmark for code understanding, generation, translation and retrieval. arXiv 2023, arXiv:2303.03004. [Google Scholar]
  57. Feng, J.; Liu, J.; Gao, C.; Chong, C.Y.; Wang, C.; Gao, S.; Xia, X. Complexcodeeval: A benchmark for evaluating large code models on more complex code. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, Sacramento, CA, USA, 27 October–1 November 2024; pp. 1895–1906. [Google Scholar]
  58. Gu, X.; Chen, M.; Lin, Y.; Hu, Y.; Zhang, H.; Wan, C.; Wei, Z.; Xu, Y.; Wang, J. On the effectiveness of large language models in domain-specific code generation. ACM Trans. Softw. Eng. Methodol. 2025, 34, 1–22. [Google Scholar] [CrossRef]
  59. Li, J.; Zhao, Y.; Li, Y.; Li, G.; Jin, Z. Acecoder: An effective prompting technique specialized in code generation. ACM Trans. Softw. Eng. Methodol. 2024, 33, 1–26. [Google Scholar] [CrossRef]
Aerospace 12 00498 g001
Figure 1. Code generation effect evaluation classification.
Figure 1. Code generation effect evaluation classification.
Aerospace 12 00498 g001
Aerospace 12 00498 g002
Figure 2. An overview of RepoSpace. Each sample consists of five components.
Figure 2. An overview of RepoSpace. Each sample consists of five components.
Aerospace 12 00498 g002
Aerospace 12 00498 g003
Figure 3. The process of constructing RepoSpace.
Figure 3. The process of constructing RepoSpace.
Aerospace 12 00498 g003
Aerospace 12 00498 g004
Figure 4. CodeBLEU scores comparison of different models on RepoSpace and ComplexCodeEval Benchmarks.
Figure 4. CodeBLEU scores comparison of different models on RepoSpace and ComplexCodeEval Benchmarks.
Aerospace 12 00498 g004
Aerospace 12 00498 g005
Figure 5. A uniquely successful case in the Local File (repo-level generation) setting.
Figure 5. A uniquely successful case in the Local File (repo-level generation) setting.
Aerospace 12 00498 g005
Aerospace 12 00498 g006
Figure 6. Prompt template structures for method body generation in code: task implementation descriptions.
Figure 6. Prompt template structures for method body generation in code: task implementation descriptions.
Aerospace 12 00498 g006
Aerospace 12 00498 g007
Figure 7. Impact of documentation quality on LLM performance in aerospace code generation: comparison of (a) Rouge-L, (b) CodeBLEU, (c) Complexity, and (d) DepMatch (Original Annotations vs. Human Annotations).
Figure 7. Impact of documentation quality on LLM performance in aerospace code generation: comparison of (a) Rouge-L, (b) CodeBLEU, (c) Complexity, and (d) DepMatch (Original Annotations vs. Human Annotations).
Aerospace 12 00498 g007
Table 1. Classification of Code Generation Technologies Based on Large Models and Their Potential Applications in Aerospace.
Table 1. Classification of Code Generation Technologies Based on Large Models and Their Potential Applications in Aerospace.
CategoryDescriptionPotential ApplicationsReferences
Pre-trainTrains the model through a large corpus of unlabeled code to enable understanding of the structure and semantics of a programming language.Suitable for generating basic spacecraft embedded software modules (such as data acquisition and communication protocol analysis), providing initial implementations of algorithms for orbit optimization, mission scheduling, and attitude control, as well as generating general analytical tool code for telemetry data, image processing, and scientific calculations.[24,25]
Fine-tuningOptimizes the predictive training model through a supervised learning framework to enable it to execute specific task instructions.Provides strong support for scenarios with high complexity and high precision requirements, such as generating control algorithm codes for specific spacecraft types. The model is effectively enhanced for space-specific missions and can more accurately respond to complex requirements in the field.[26,27,28,29,30]
Prompt engineeringGuides the model to generate code by optimizing prompt words without additional training.Can be used for prototype code development, quickly generating task scripts or algorithm prototypes, and validating new task concepts or techniques. Additionally, in real-time mission generation, it can dynamically create flight control logic or data processing programs to meet current needs, making it especially suitable for aerospace scenarios with changing requirements.[31,32,33]
Retrieval augmentedProvides powerful technical support for space missions by combining external retrieval capabilities with precise knowledge enhancements.Can generate code templates similar to previous tasks by retrieving historical task documents or technical databases, and also generate code compatible with existing systems or tools, thereby improving development efficiency and consistency.[34,35,36]
Table 2. Overview of the studied LLMs.
Table 2. Overview of the studied LLMs.
TypeNameSize
Code LLMQwen2.5-Coder7 B/32 B
DeepSeek-Coder6.7 B/33 B
Magicoder-S-DS6.7 B
OpenCodeInterpreter-DS6.7 B
CodeLlama7 B/34 B
General LLMDeepSeek-R1-Distill-Qwen7 B/32 B
DeepSeek-V3671 B (API)
Table 3. Performance of LLMs on RepoSpace, evaluated using Rouge-L, CodeBLEU, Complexity, DepMatch. The table presents results under the one-shot setting with RAG and Context.
Table 3. Performance of LLMs on RepoSpace, evaluated using Rouge-L, CodeBLEU, Complexity, DepMatch. The table presents results under the one-shot setting with RAG and Context.
LLMsSizeRouge-LCodeBLEUComplexityDepMatch
6.7B-7B
Qwen2.5-Coder7B29.8630.465.8129.15
DeepSeek-Coder6.7B26.6332.185.2130.32
Magicoder-S-DS6.7B24.5531.505.5227.76
OpenCodeInterpreter-DS6.7B29.5530.165.4128.09
CodeLlama7B27.2531.765.5827.72
DeepSeek-R1-Distill-Qwen7B23.1228.156.1829.79
32 B-671B
CodeLlama34B32.6133.255.7927.65
Qwen2.5-Coder32B32.6333.485.5929.31
DeepSeek-Coder33B33.6833.945.8331.19
DeepSeek-R1-Distill-Qwen32B35.7132.186.0229.51
DeepSeek-V3671B (API)36.7534.584.5132.34
Table 4. Results of different prompt templates for DeepSeek-Coder-6.7B in RQ2.
Table 4. Results of different prompt templates for DeepSeek-Coder-6.7B in RQ2.
Det.Ts.Rouge-LCodeBLEU
25.6331.38
✓25.1231.76
✓ 23.4229.91
✓✓26.6332.18
Det.: using detailed (with ✓) or brief instruction (without ✓). Ts.: using two-step (with ✓) or one-step instruction (without ✓).
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

He, R.; Zhang, L.; Lyu, M.; Lyu, L.; Xue, C. Using Large Language Models for Aerospace Code Generation: Methods, Benchmarks, and Potential Values. Aerospace 2025, 12, 498. https://doi.org/10.3390/aerospace12060498

AMA Style

He R, Zhang L, Lyu M, Lyu L, Xue C. Using Large Language Models for Aerospace Code Generation: Methods, Benchmarks, and Potential Values. Aerospace. 2025; 12(6):498. https://doi.org/10.3390/aerospace12060498

Chicago/Turabian Style

He, Rui, Liang Zhang, Mengyao Lyu, Liangqing Lyu, and Changbin Xue. 2025. "Using Large Language Models for Aerospace Code Generation: Methods, Benchmarks, and Potential Values" Aerospace 12, no. 6: 498. https://doi.org/10.3390/aerospace12060498

APA Style

He, R., Zhang, L., Lyu, M., Lyu, L., & Xue, C. (2025). Using Large Language Models for Aerospace Code Generation: Methods, Benchmarks, and Potential Values. Aerospace, 12(6), 498. https://doi.org/10.3390/aerospace12060498

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop