Neural Methods for Programming: A Comprehensive Survey and Future Directions

Abstract

The advancement of neural-based models has driven significant progress in modern code intelligence, accelerating the development of intelligent programming tools such as code assistants and automated software engineering systems. This study presents a comprehensive and systematic survey of neural methods for programming tasks within the broader context of software development. Guided by six research questions, this study synthesizes insights from more than 250 scientific papers, the majority of which were published between 2015 and 2025, with earlier foundational works (dating back to the late 1990s) included for historical context. The analysis spans 18 major programming tasks, including code generation, code translation, code clone detection, code classification, and vulnerability detection. The survey methodologically examines the development and evolution of neural approaches, the datasets employed, and the performance evaluation metrics adopted in this field. It traces the progress in neural techniques from early code modeling approaches to advanced Code-specific Large Language Models (Code LLMs), emphasizing their advantages over traditional rule-based and statistical methods. A taxonomy of evaluation metrics and a categorized summary of datasets and benchmarks reveal both progress and persistent limitations in data coverage and evaluation practices. The review further distinguishes neural models designed for natural language processing and programming languages, highlighting the structural and functional characteristics that influence model performance. Finally, the study discusses emerging trends, unresolved challenges, and potential research directions, underscoring the transformative role of neural-based architectures, particularly Code LLMs, in enhancing programming and software design activities and shaping the future of AI-driven software development.

1. Introduction

Artificial intelligence (AI) and its subfields, such as machine learning (ML), deep learning (DL), and generative AI, have become integral to modern activities by improving efficiency and quality in daily tasks. DL-based natural language processing (NLP) enables automation beyond human capabilities [1], including language-related tasks such as autocompletion, long-form text predictions [2], and multilingual translation. The fundamental building blocks of such advancements in the field are neural networks. Hence, they can be considered key drivers of AI-based automation worldwide, enhancing various aspects of daily life. Applications of neural network-based language models, such as ChatGPT [3], showcase improvements in NLP. However, programming, which involves structured instructions to solve computational problems, remains a challenging domain due to its strict syntax, logical complexity, and complexity of semantic analysis. Tasks like writing new code, translating or refactoring existing code, and debugging have traditionally been performed manually or with rigid tools for decades, often struggling to keep up with dynamic and evolving project requirements. This complexity has driven interest in neural approaches for programming tasks, as data-driven models can learn intricate patterns and generalize more effectively to novel problems. Recently, state-of-the-art (SOTA) NLP models—such as OpenAI’s Generative Pre-trained Transformer (GPT) models [3]—have been adapted for programming language (PL) tasks [4], and Code-specific Large Language Models (Code LLMs) have been tailored for code-related tasks [5], thereby reducing traditionally time-consuming manual processes such as code editing, comment writing, summarization, and debugging while enabling automation in a range of programming tasks, including source code translation [6,7], code generation [5], summarization [8], automatic code editing [9], decompilation [10], and code similarity detection [11], among others.

Despite the active utilization of neural-based NLP techniques in code-related tasks, there remains a noticeable gap in comprehensive analyses examining the rapid and evolving integration of neural methods into programming-focused applications. Although previous surveys have examined scientific research on neural methods for programming tasks, they often suffer from a limited scope—either focusing on specific tasks [12,13], covering only a narrow subset of neural network-based techniques used in these tasks [14], emphasizing general AI/ML methods with minimal attention to neural techniques [15], or concentrating on broader software engineering (SE) applications with insufficient focus on the programming-centric tasks. Some reviews [16,17,18] classify the applications of neural models on code into three categories—code generation models, representational models, and pattern mining (code understanding) models—with comparisons of their usage in NLP and programming tasks, whereas others analyze the role of neural models in code understanding, by distinguishing between sequence-based and graph-based approaches. However, many [19,20,21] omit discussions on the role of neural methods specifically tailored to programming tasks, overlooking the challenges these methods face in capturing the complex syntactic and semantic structures of code and the logical idiom constraints unique to each programming language.

Given the limitations of previous surveys [22,23,24,25,26]—such as their task-specific focus (e.g., code summarization and program repair) and the omission of critical discussions [27,28]—researchers may struggle to identify gaps for further exploration in research themes targeting neural networks applied to programming tasks. To address this, our survey consolidates previous surveys and provides a comprehensive review of existing studies, bridging research gaps and guiding future investigations. In summary, this survey outlines the following concepts and contributions:

• Review existing survey papers on neural methods for programming tasks.
• Compare neural methods applied to NLs and PLs.
• Compare rule-based, statistical, and neural methods for code-related tasks.
• Review 18 programming tasks and outline future research directions for each.
• Examine datasets and benchmarks used in these tasks.
• Analyze performance evaluation metrics for neural models in the reviewed papers.
• Summarize key findings from the reviewed papers on neural models applied to source code.
• Discuss major challenges in applying neural-based models for programming tasks.
• Explore the role of LLMs in programming.
• Identify research gaps and propose future research directions.

This survey serves as a roadmap for researchers interested in the applications of neural networks to programming languages and programming tasks, highlighting the evolution of neural networks from early single-layer models [29] to modern large-scale architectures [30] as depicted in Figure 1. By providing this historical perspective, we contextualize recent achievements and outline directions for future research. To the best of our knowledge, no existing survey is solely dedicated to neural methods with a comprehensive review tracing their application to programming tasks, from early developments to the latest advancements in Code LLMs trained exclusively on code. To maintain consistency and enhance clarity throughout our analysis, we use the term programming tasks to refer to the various code-related activities discussed in this study. While different literature employs a range of terms, such as SE tasks [14], coding tasks [31], code-related tasks, and programming language tasks [32], to describe similar or overlapping concepts, we adopt a unified terminology to reduce ambiguity for readers. However, given the breadth and interdisciplinary nature of these tasks, often involving source code, bug reports, and natural language artifacts, we occasionally use the term SE tasks when appropriate.

Our research scope concentrates on neural methods applied specifically to programming-related SE, as many other SE activities can often be addressed through general-purpose NLP and deep learning surveys. Consequently, this work represents the first comprehensive and systematic review dedicated exclusively to examining how neural approaches have been applied within programming-centric SE domains. To guide this study, we propose the following research questions:

• RQ1: How do neural approaches compare to rule-based and statistical methods in the context of programming tasks?
• RQ2: What is the current landscape of datasets and benchmarks for neural methods in programming tasks, and what are the critical gaps?
• RQ3: Which evaluation metrics best capture model performance on code, both syntactically and functionally, and where do standard NLP metrics fall short?
• RQ4: What roles do LLMs (e.g., GPT-4, LLaMA, Claude) play in programming tasks?
• RQ5: What are the main bottlenecks in scaling and deploying neural-based programming solutions to real-world codebases, and how can they be addressed?
• RQ6: How have neural methods for programming evolved, and which model- and system-level advances have driven this progression?

Finally, we emphasize that our survey follows established systematic review practices—we defined a set of search queries, inclusion/exclusion criteria, and we applied consistent paper collection procedures as noted in Section 3. Our methodology is based on standard guidelines for software engineering systematic literature reviews (e.g., Brereton et al. [33] and Kitchenham and Charters [34]). We extend these practices by including quantitative and qualitative trend analysis to enhance comprehensiveness and rigor, ensuring that our review is reproducible.

To facilitate the discussion, the rest of the paper is structured as follows. Section 2 reviews existing surveys. Section 3 outlines the paper selection process. Section 4 compares the neural models used for NLs and PLs. Section 5 presents neural methods for 18 programming tasks. Section 6 reviews the datasets and evaluation metrics. Section 7 compares neural methods with traditional methods such as rule-based and statistical methods. Section 8 presents answers to the research questions and outlines future research directions, while Section 9 concludes.

2. Related Works

This section reviews prior surveys examining AI approaches in programming and SE, particularly the role of neural methods in programming-related tasks. While some works broadly address AI in SE, they often overlook the huge gains and critical technical challenges specific to neural methods on code-focused applications. Al-Hossami and Shaikh [27] present a taxonomy of AI methods applied to source code and conversational SE systems, spanning traditional neural models such as Multilayer Perceptron, Recurrent Neural Networks (RNNs), and Long Short-Term Memory (LSTM) to Transformers. They categorize tasks into open-domain and task-oriented dialogue systems, referencing datasets like CoNaLa, CodeSearchNet, and other GitHub-derived corpora. Similarly, Watson et al. [35] review DL utilization across SE tasks including defect prediction, program comprehension, and bug fixing. They identify key neural architectures such as autoencoders, Siamese Networks, encoder–decoder models, and CNNs, and common metrics like accuracy, ${\mathrm{F}}_1$, Mean Reciprocal Rank (MRR), Recall@k, and Bilingual Evaluation Understudy (BLEU), while noting limitations include weak reproducibility, preprocessing inconsistencies, and overfitting risks across reviewed works.

Cao et al. [36] survey explainable AI in SE, classifying methods by portability (model-agnostic/specific), timing (ante hoc/post hoc), and scope (local/global). Their dataset coverage includes Devign (Deep Vulnerability Identification via Graph Neural Networks), National Vulnerability Database (NVD), StackOverflow, and CodeSearchNet. They highlight explainable AI’s stronger presence in security and defect prediction, its underutilization in early SE phases, and concerns over inconsistent evaluations and lack of standard baselines. The review in [37] emphasizes encoder–decoder architectures in code modeling and generation, exploring a wide array of code representations. Similarly, Samoaa et al. [17] systematically map how DL techniques model code using token, tree, graph, and hybrid formats across classification, clone detection, and summarization tasks. They advocate for standardized benchmarks with industry-level evaluation. Hou et al. [38] discuss the dominance of decoder-only and encoder–decoder architectures (e.g., GPT-3/4, Codex, and CodeT5) with a review on publicly available datasets like HumanEval and Mostly Basic Programming Problems (MBPP). They emphasize the role of prompt engineering and parameter-efficient tuning techniques like Low-Rank Adaptation, prefix-tuning, and adapters, though most models are validated only on academic datasets. Similarly emphasizing LLMs, Fan et al. [39] raise concerns about hallucinations, weak evaluation frameworks, and a lack of rigorous metrics for assessing generated code and other artifacts.

More narrowly scoped reviews target specific programming tasks. Akimova et al. [24] review studies on software defect prediction that utilize RNNs, LSTMs, Tree-LSTMs, GNNs, and Transformer-based models. While neural approaches outperform classical methods, challenges remain in benchmark standardization, class imbalance, and metric selection. Wu et al. [40] analyze vulnerability detection methods, comparing RNN-based and GNN-based models. They stress the need for better code embeddings, representation standards, and interpretability. Xie et al. [25] classify the research works in code search into three taxonomies such as semantic modeling of query, code semantics modeling, and semantic matching phases. Their findings show that pre-trained Transformers like CodeBERT and multi-view inputs significantly improve retrieval, while noting the high training cost and blurred task boundaries with clone detection. Dou et al. [41] empirically review LLMs for code clone detection, comparing open-source (e.g., LLaMA, Vicuna, and Falcon) and proprietary models (e.g., GPT-3.5 and GPT-4) against traditional tools like SourcererCC and NiCad. Using benchmarks such as BigCloneBench and CodeNet (Java, C++, and Python), they examine clone types (1–4) through zero-shot prompts, chain-of-thought prompting, and embedding-based methods (e.g., text-embedding-ada-002 and CodeBERT). LLMs notably outperform classical tools for Type-3/4 clones, especially for Python. GPT-4 demonstrates robustness under multi-step chain-of-thought prompting. Yet, limitations include prompt design constraints, lack of few-shot settings, potential training/evaluation overlap, and computational resource restrictions.

Zhong et al. [42] offer a taxonomy of neural program repair, emphasizing the review of automatic program repair (APR) techniques that leverage various data representation approaches, such as abstract syntax tree (AST) and control flow graph (CFG), to extract context, and employ different neural-based methods, such as encoder–decoder, tree-based, and graph-based architectures, to generate correct or plausible bug-fixing patches. Similarly, Huang et al. [43] trace the APR evolution from search-based to constraint-based, template-based, and learning-based techniques, highlighting models from RNNs and Transformers to static-analysis-integrated encodings, tested on various benchmarks like Defects4J and QuixBugs. Zhang et al. [22] also review learning-based APR tools that frame bug fixing as neural translation, identifying gaps such as limited multi-hunk fix support and inadequate benchmark standardization. Wang et al. [44] provide a broad survey of LLMs in software testing and repair, noting that decoder-only and encoder–decoder models such as GPT-3, Codex, Text-to-Text Transfer Transformer (T5), CodeT5, BART, and LLaMA often outperform traditional techniques. However, they raise caution about benchmark data leakage, privacy risks, and the computational cost of real-world deployment.

Le et al. [28] classify data-driven vulnerability assessment into five themes and subthemes, examining various AI techniques: traditional ML like Support Vector Machine (SVM) and random forest, deep learning like Siamese Neural Networks and GNNs, and knowledge graphs. They also explore datasets like ExploitDB and the Common Vulnerability Scoring Systems, recommending robust validation strategies and evaluation metrics suited for imbalanced data such as Matthews Correlation Coefficient, Mean Absolute Error, and Root Mean Square Error. With a similar research focus, Xiaomeng et al. [23] review ML-based static code analysis for vulnerability detection. Their work spans traditional models like SVMs to deep learning (CNNs, RNNs, and LSTMs), emphasizing reduced manual effort but also highlighting issues like class imbalance and poor generalizability across projects and languages.

Zakeri-Nasrabadi et al. [26] provide a taxonomy of code similarity techniques, covering token-based, tree-based, graph-based, metric-based, and learning-based approaches. They assess benchmarks such as BigCloneBench and metrics like precision, recall, and F1, noting the dominance of languages like Java and C++ and the reproducibility limitations due to some restricted tools for open access. Katsogiannis et al. [45] survey neural approaches for Text-to-SQL translation, reviewing architectures such as seq2seq models, grammar-guided decoders, and GNNs for schema linking. While pre-training-enhanced language models show strong accuracy on benchmarks like WikiSQL and Spider, challenges remain in scalability and generalization to new domains. Grazia and Pradel [46] present a comprehensive review of code search over researches spanned across three decades. They categorize query mechanisms—ranging from NL and code snippets to formal patterns and input/output examples—and discuss indexing strategies, retrieval methods, and ranking techniques. Their survey addresses real-world search behaviors and emphasizes open challenges, such as supporting version history, cross-language search, and more robust evaluation frameworks.

Zheng et al. [47] review 149 studies, comparing those LLMs trained/fine-tuned on code like Codex, CodeLLaMA, and AlphaCode against general LLMs using HumanEval and Automated Programming Progress Standard (APPS) benchmarks. Their findings show that code-specific models typically outperform general models using metrics like Pass@k and BLEU. Zan et al. [48] evaluate 27 LLMs for NL-to-code tasks, identifying three key performance drivers: model size, access to premium datasets, and expert fine-tuning. However, they note an overemphasis on short Python snippets, limiting generalization. Wong et al. [49] survey the impact of LLMs trained on “Big Code” in integrated development environment (IDE), examining their ability to support tasks like code autocompletion and semantic comprehension. They also caution about drawbacks such as model bias, security concerns, and latency that may hinder real-time applicability. Similarly, Zheng et al. [50] focus on Transformer-based pre-trained language models and LLMs with parameter size greater than or equal to 0.8B, across seven core tasks such as test generation, defect detection, and code repair. These models perform well in syntax-aware tasks but struggle with semantic understanding. Their review also underscores fragmented benchmarks, limited error analysis, and insufficient attention to ethical and cost-related concerns. Xu and Zhu [51] offer a deep dive into pre-trained langauge models like CodeBERT, GraphCodeBERT, and CodeT5, detailing their distinct learning objectives and use of input structures such as AST and data-flow graph (DFG). However, they note issues including scarce multilingual datasets and underexplored graph-based techniques.

Some surveys have also explored application of AI and its variant for SE life-cycle. Crawford et al. [21] review the integration of AI into software project management, tracing its progression from early expert systems and rule-based planners to contemporary neural network applications. Their findings highlight AI’s value in improving estimation accuracy, risk management, and analytics for agile workflow. However, persistent challenges remain, including data privacy concerns and the lack of interpretability in AI models, both of which hinder stakeholder trust and limit industry adoption. Batarseh et al. [52] map AI methods to five SE phases, finding active prototyping in testing and requirements engineering, yet limited industrial validation and evaluation standardization. Durrani et al. [19] take a survey on broad perspective across domains of SE phases. However, their treatment of SE tasks and overreliance on quantitative summaries mined from AI tools (e.g., dimensions.ai) detracts from the unique advances in neural methods specific to programming. Their focus on older models like random forests overlooks the transformative capacity of modern neural models that now even support complex program generation and logical reasoning.

In contrast, our survey concentrates on programming-related SE tasks. While some documentation or design tasks may benefit from lightly fine-tuned general NLP models, we focus on how neural-based architectures are designed to understand, generate, and interact with source code. The limitations of prior surveys—such as narrow benchmarks, limited real-world validation, and fragmented methodologies—underscore the need for a unified, code-centric view of modern neural-based applications in SE.

Table 1. Summary of previous survey works’ methodologies, strengths, weaknesses, and robustness.

Survey Paper	Methodology	Advantages	Disadvantages	Robustness
Allamanis et al. (2018) [16]	Narrative survey classifying ML models for ‘ Big Code’ and software naturalness using linguistic analogies.	Offers a clear taxonomy of ML models; shows foundational framing about conventional and idiomatic naturalness of code.	Lacks standardized benchmarks; some models (e.g., Transformers) were emerging and not fully covered.	Foundational and widely cited but predates large-scale pre-trained models.
Xiaomeng et al. (2018) [23]	Narrative review on ML in secure code review and vulnerability detection.	Good historical context; contrasts DL with earlier feature-based methods.	Outdated; lacks attention to recent graph neural networks (GNNs)/Transformer models.	Limited coverage and moderately outdated, but contextually informative.
Akimova et al. (2021) [24]	Review on DL-based defect prediction papers, categorized models, datasets, metrics, and investigated trends.	Comprehensive coverage of DL techniques, including data-labeling and generalization issues.	Narrow focus on certain PLs, limited Transformer models coverage.	Moderately useful for defect prediction, limited for closely related SE tasks.
Yang et al. (2022) [14]	Systematic literature review on DL-in-SE papers; organized by DL model architectures, tasks, datasets, and metrics.	Broad task coverage; excellent model categorization; metrics and dataset taxonomy included.	Reproducibility issues; underexplored cross-project transfer learning; lacks deep industrial insights, no coverage of LLMs.	Comprehensive and structured, though no coverage of the emergence of neural-based LLMs.
Zhang et al. (2022) [12]	Structured survey of automatic source code summarization, classifies the task into modeling, generation, and evaluation phases.	Balanced treatment of classic information retrieval and DL methods, covers datasets, metrics, and emerging models.	Limited to code-comment pairs discussion, no critical analysis on the Recall-Oriented Understudy for Gisting Evaluation (ROUGE), Bilingual Evaluation Understudy (BLEU), and other weak metrics issues.	Good for code summarization, well-scoped and methodical, though limited cross-language coverage.
Samoaa et al. (2022) [17]	Systematic mapping study on code representations across SE tasks.	Maps token, tree, graph modalities; clear representation analysis.	Lacks model-specific analysis; omits modern pre-trained models.	Moderately strong on representation types, less on effectiveness.
Xu and Zhu (2022) [51]	Review of pretrained language models in code intelligence.	Covers pre-training and fine-tuning objectives, pre-trained language models’ architectures, and preprocessing, and code structure extraction pipelines in pre-trained language models.	Minimal coverage of non-Transformer and structure-based code intelligence models.	Strong overview of pre-trained language models and Transformer-based models, weaker on fine-grained discussion on programming tasks.
Le et al. (2022) [28]	Investigated data-driven vulnerability assessment by examining ML, DL and NLP techniques.	Practical view on the Common Vulnerability Scoring Systems’ alignment with reviewed themes and real-world focus.	Lacks depth insight on vulnerability-mitigation related challenges, e.g., patch modeling.	Comprehensive taxonomy of data-driven vulnerability in five themes.
Al-Hossami and Shaikh (2022) [27]	Broad review of AI in source code and conversational systems.	Unique angle on dialogue-based AI; educational application emphasis.	Limited depth on models and benchmarks; no technical coverage related to AI for programming.	Moderately exploratory and broad coverage of AI applications in education, but less technical with respect to PLs.
Amalfitano et al. (2023) [18]	Tertiary study synthesizing secondary surveys on AI for software testing.	Provides a holistic, high-level view of AI contributions to software testing; maps AI subdomains to testing tasks.	Lacks depth on datasets, models; omits emerging techniques and several aspects are not yet widely reviewed.	Averagely comprehensive in coverage, but breadth over depth reduces specificity.
Fontes and Gay (2023) [15]	Systematic mapping of studies applying ML to automated test generation; categorized by ML type, task, and metrics.	Captures detailed discussion on test and oracle generation; explores diverse ML methods.	Limited discussion on evaluation quality; limited coverage on neural networks, and no-idea about LLMs; replication gaps.	Strong in breadth but limited by inconsistent evaluation standards.
Xie et al. (2023) [25]	Review of DL-based code search systems with focus on encoding query and source code, and measuring their embedding similarity.	Highlights the concepts of deep code search, benchmarking, model limitations, and practical deployment issues.	Retrieval-centric; lacks broader coverage on SE use cases that align with programmer’s workflow focus.	Robust in evaluating code search methods with coverage of popular datasets and benchmarks. But no clue about LLMs.
Zhang et al. (2023) [22]	Review of automatic program repair (APR) structured by pipeline, dataset, and evaluation.	Clear taxonomy of APR approaches, focus on real bug benchmarks.	The review is Java centric, and the evaluation heavily depends on test-suite plausibility.	Highly detailed and accurate for APR pipelines.
Zakeri-Nasrabadi et al. (2023) [26]	Systematic literature review on code clone studies with method and dataset taxonomy.	Broad code similarity and clone detection taxonomy; tools/data gaps clearly identified.	Limited coverage of pre-trained language models and no clue about LLMs.	Excellent for classical clone detection methods.
Uddin et al. (2025) [13]	Analysis of DL for vulnerability detection; introduces a life-cycle from data construction to model deployment.	Life-cycle modeling gives structured perspective; highlights real-world gaps and practical considerations.	Narrower scope, and the analysis is biased toward C/C++ datasets.	Robust within the domain tailored for vulnerability detection, but it is less generalizable.

compares existing survey and review articles. It outlines each prior work’s main focus, key strengths, identified limitations, and basis of validation, allowing readers to quickly compare how existing approaches relate to ours. Based on our review of previous survey studies, we observe that prior works on the application of neural methods and general AI/ML techniques for software engineering and programming tasks can be broadly categorized into four groups as summarized in

Table 2. High-level taxonomy of previous survey papers in the field.

Area of Focus	Examples
Task-specific surveys	Code Summarization [12], Comment Generation [53], Code Generation [54], Code Translation [55], Code Clone Detection [56], Code Vulnerability Detection [13], Program Repair [22], etc.
Surveys focused on subsets of neural methods	Review solely on either DL [14,57], LLMs [58], or Code LLMs [47]
Surveys focused on the phases of the Software Development Life Cycle, with limited emphasis on the coding phase.	Software Development Life Cycle phases [19,20] such as Requirement Engineering [52], System Design, and Project Management.
Surveys focused on traditional AI/ML methods with minimal focus on neural methods	Surveys with vast focus on traditional AI/ML methods [18,23] such as Random Forest, Decision Trees, XGBoost, Naive Bayes, Logistic Regression, and SVM.

.

Our survey distinguishes itself from prior reviews in several key ways. Unlike earlier works that narrowly focus on specific deep learning techniques or isolated SE subdomains, our survey systematically covers almost all types of neural methods across a broad range of programming-centric SE tasks. Additionally, we compare the application of neural methods in both PL and NL, highlighting representative models used in each context. Furthermore, our work stands out by explicitly comparing rule-based, statistical, and neural approaches, offering a holistic understanding of how neural models outperform earlier paradigms as detailed in Section 7. Overall, our survey adopts a broader perspective by integrating both qualitative synthesis and quantitative analysis, examining over 250 papers across 18 distinct programming task domains. We also provide trend-based insights on publication growth, dataset and benchmark usage, and model performance metrics as described in Section 6.

3. Scientific Paper Selection Process

Our paper selection process is visually summarized in Figure 2. In this study, we have collected and analyzed more than 250 research papers, exploring their primary objectives, contributions, and future research directions. Most of these works were published between 2015 and 2025, covering a decade of intensive research on neural methods for programming tasks. A few earlier foundational studies, dating back to the late 1990s, were also included to provide historical continuity and contextualize recent advances as depicted in Figure 3.

Our investigation delves into various applications of neural methods in programming tasks, including advanced code representation techniques that enhance automation in software development [59].

Table 3. Distribution of collected papers by source.

S.No	Publication Source	Number of Publications
1	ACL, NeurIPS, ICLR	62
2	ACM	53
3	IEEE	49
4	IEEE/ACM (Jointly)	35
5	ArXiV	32
6	Elsevier, MDPI, Springer	29
7	Others	15
	Total	275

Note. IEEE/ACM (Jointly) in the above table indicates those papers published in the conferences or journals that are co-sponsored or jointly organized by both IEEE and ACM.

provides an overview of the digital sources in which these papers are published, while Figure 3 highlights that recent years have witnessed a significant increase in publications within this domain.

The methodology for selecting research papers in this survey aligns with established approaches, such as those employed in [60]. Specifically, we manually defined key phrases and search terms related to various programming tasks to retrieve relevant articles from online digital libraries. Our approach to identifying relevant literature was methodical and targeted. We focused on search phrases involving neural approaches within the context of programming and programming-centric SE. Specifically, we frequently used keyword combinations that included terms like “neural methods”, “neural models”, and “neural networks”, paired with common prepositions followed by the phrases “programming tasks”, “programming”, and “software engineering”, as well as specific code-related task names. The primary search patterns are shown below:

• (“neural methods” | “neural models” | “neural networks”) (“in” | “for”) (“programming tasks” | “programming” | “software engineering”)
• (“neural methods” | “neural models” | “neural networks”) (“in” | “for” | “on” | “and” ) [specific task]

where “|” stands for “or”, and [specific task] is a placeholder for areas such as code translation, code generation, code clone detection, code summarization, program synthesis, and other programming-related tasks.

We also applied inclusion and exclusion criteria to refine our selection. Only studies written in English were considered. Furthermore, we specifically included studies that explore the application of neural methods within programming-centric software engineering tasks. Conversely, studies focusing on the reverse, such as applying software engineering techniques to improve AI systems or those discussing neural methods without a clear application to software or programming tasks, were excluded from our review. After the initial collection, we refined the selection by filtering articles based on specific attributes to ensure alignment with our research objectives and focus. The paper selection criteria predominantly include the following aspects:

• Breakthrough research and SOTA approaches: Most selected articles focus on SOTA neural network architectures, including Transformer-based neural networks, encoder–decoder models, and other deep learning frameworks [61]. These studies provide cutting-edge advancements in neural-based code processing.
• Recent publications and citation-based filtering: A significant portion of the papers have been published recently. To ensure that we include impactful research articles, we adopt a citation score-based filtering approach as in [60]. However, for studies published between 2022 and 2025, we prioritize the relevance of the topic without referring to citation counts.

To ensure the inclusion of impactful but recently published studies, we intentionally excluded citation counts as a criterion when selecting papers from 2022 to 2025. Many papers from this period were initially released as preprints on platforms like arXiv and had not accumulated substantial citations, even though they were later accepted by peer-reviewed venues in 2024 and 2025. Relying on citation metrics could have led to the under-representation of valuable studies that were still in the process of gaining visibility. Notably, key contributions—such as OpenAI’s GPT-4 and various iterations of Meta’s LLaMA models—were disseminated via arXiv rather than formal publication channels [62,63,64]. This reflects a broader trend toward early sharing in the AI community. To fairly represent the most relevant and innovative research from this evolving landscape, we prioritized the relevance and impact of contributions over citation numbers. During the filtering process, we first reviewed each paper’s abstract for relevance. If clarity was insufficient, we further examined the introduction, model architecture and research approach, experimental setup, and discussion sections.

In further illustrating the thematic relationships within the reviewed literature, Figure 4 presents a keyword co-occurrence network generated from the bibliometric analysis of the collected papers. The network highlights how keywords such as code, source code, learning, neural, and language models form the core of the research landscape, demonstrating strong associations with programming tasks including code translation, code generation, code summarization, and other programming tasks. These interlinked clusters reveal the growing convergence between neural methods, neural networks, code understanding and generation, and software engineering automation.

4. Neural Methods for Natural Languages vs. Programming Languages

Recent developments in large-scale NLP models have revolutionized human language processing [1,65]. Given the structural and lexical similarities between programming and natural languages, researchers have adapted neural-based NLP techniques to PLs [66]. PLs are essential tools for operating computing devices, enabling application and system development for programmers. Unlike NLs, which evolve through speech and cultural influences, PLs are designed by programmers to serve as a medium of communication between humans and machines. While NLs are analyzed through the norms and grammars of their speakers, PLs are based solely on the syntax, semantics, and grammar rules defined by their designers, making them a specialized subset of NLP. As such, the application of neural-based methods to PLs has benefited from advances in NLP-driven text processing. However, neural approaches tailored to PLs must account for significant differences from those used for NLs.

In terms of using neural methods for PLs, the following are remarkable opportunities we can exploit:

• Simplified Linguistic Rules: PLs avoid challenges like pronunciation, accents (e.g., UK vs. US English), writing styles (e.g., formal vs. informal) [67], and text writing direction (e.g., Arabic vs. English), making automation easier than in NLs.
• Abundant Open-Source Code Corpora: Repositories like GitHub and coding competitions offer large collections of code written in individual PLs, providing abundant monolingual data that enhance neural model performance on language-specific programming tasks.

On the other hand, there are also some pitfalls in using neural methods for PLs as shown below:

• Limited Learning Modes: Unlike NLs, PLs are learned solely through reading and writing, preventing neural-based automation from leveraging voice or speech processing [68].
• Grammar Sensitivity: PLs are highly syntax dependent, where minor errors, like missing whitespace in Python or misplaced semicolons in compiled languages, can cause the entire program to fail.

Therefore, the notable distinction between PLs and NLs lies in syntax and ambiguity: PLs are inherently unambiguous and structured to be deterministically parsed, whereas NLs often exhibit ambiguity due to diverse dialects and informal usage. This syntactic rigidity implies that code tokens follow strict formal rules, such as matching braces and consistent indentation that are not typical considerations in NLP tasks. As a result, code tends to be more predictable than human language, given that developers frequently reuse established idioms and avoid non-standard constructs. Consequently, neural models for code often integrate structural representations, such as ASTs or program graphs, rather than processing flat token sequences alone. For instance, GraphCodeBERT [69] enhances a Transformer-based architecture by incorporating DFGs alongside the sequence of tokens to capture variable dependencies and value propagation within the code.

While general-purpose neural language models like BERT [70] have been fine-tuned to apply for both code and NL domains, models trained explicitly for code typically include additional architectural components. These may involve embedding ASTs or data-flow structures using neural network with code-comprehending traits, or using specialized attention masks. Some models also augment input representations with tokens specific to programming constructs, such as language keywords or data types. By contrast, neural models in NLP often emphasize shallow parsing and focus on linguistic challenges such as co-reference resolution and semantic interpretation.

Hence, in this context, we review some prominent NLP models and compare their equivalent models trained for PLs. For example, the Salesforce’s CodeT5 model [71] was developed from Google’s T5 model. Both employ Byte Pair Encoding tokenization and follow an encoder–decoder architecture. However, while T5 was trained on the C4 corpus, CodeT5 was pre-trained on CodeSearchNet and fine-tuned using CodeXGLUE, enabling it to process both NL and source code. T5 has since evolved into t5x and seqio models. Similarly, CodeT5 has been extended into CodeT5+, incorporating objectives such as instruction tuning, text-code matching, span denoising, causal language model pre-training, and contrastive learning. Inspired by such development, large multilingual corpora combining natural and programming language tokens are being introduced in the field [72].

Additionally, models derived from BERT have demonstrated strong performance in both NLP and programming tasks. For instance, cuBERT [73] utilizes standard Python tokenization with Word2Vec-based fine-tuning for code classification. CoCLUBERT [74] enhances code clustering through objectives like Deep Robust Clustering, CoCLUBERT-Triplet, and CoCLUBERT-Unsupervised, which outperforms its cuBERT baseline model. Microsoft’s CodeBERT [75] employs bimodal Masked Language Modeling with NL and code, along with unimodal (replaced token detection) objectives. Several CodeBERT variant models like GraphCodeBERT [69], UniXcoder [76], LongCoder [77], and others address some programming challenges.

Cross-lingual pre-training and fine-tuning techniques [78] have further enhanced PL models adaptability. A key example is the unsupervised neural machine translation (NMT) approach introduced in [65], which employs Cross-lingual Masked language pre-training, Denoising Autoencoding (DAE), and Back-Translation techniques. This approach forms the backbone for Facebook’s (currently known as Meta) TransCoder models, which were designed for code translation [79].

Furthermore, the code summarization research in [8] has introduced mechanisms such as the “copy attention mechanism”, improving the quality of code summarization. Unlike its equivalent NLP-based text summarization model presented in [80], which relies on vanilla Transformers and Recall-Oriented Understudy for Gisting Evaluation (ROUGE) scores, the code summarization model integrates multiple performance metrics, including BLEU, Metric for Evaluation of Translation with Explicit ORdering (METEOR), and ROUGE.

A key distinction between neural models for NLs and PLs lies in their training data. Unlike NL corpora, source code cannot be treated as plain text. Therefore, adapting an NLP method for PLs requires specialized preprocessing, tokenization, and model architecture. These differences can help to distinguish why directly transferring NLP models to programming tasks frequently leads to suboptimal results. Neural architectures tailored for programming tasks, through the incorporation of structural features such as ASTs, control and data flow analysis, and type information, tend to outperform their general-purpose counterparts on tasks like code completion, summarization, and bug detection.

To complement the conceptual discussions above,

Table 4. Comparative perspective on neural models for natural and programming languages.

Aspect	NL Models (e.g., BERT, T5)	PL Models (e.g., CodeBERT and CodeT5)
Representative tasks	Question answering, sentiment analysis, abstractive summarization, cross-lingual transfer [65,70,80].	Code search, code summarization, defect prediction, code translation, code completion [8,69,71,75].
Training data and structural bias	Large-scale corpora such as C4 and multilingual web crawls; sequence-focused tokenization with limited structural priors [70,72,80].	Curated repositories (CodeSearchNet, CodeXGLUE) with AST/DFG-aware tokenization and structure-enhanced encoders [69,71,75].
Common metrics	Accuracy, F1, BLEU, ROUGE, cross-lingual transfer scores [65,70,80].	CodeBLEU, BLEU, Mean Reciprocal Rank (MRR), Precision@k, and exact match for code translation [8,71,75].
Empirical comparisons	Fine-tuned BERT variants serve as baselines for syntax-agnostic NLP tasks; performance drops observed on code understanding benchmarks.	CodeBERT improves over BERT on code search; CodeT5 outperforms T5 on summarization and generation tasks tailored to code [71,75].
Observed limitations	Difficulty modeling rigid syntax and long-range dependencies in source code; limited grounding in program semantics.	Sensitive to language-specific idioms and repository noise; struggles with cross-language generalization and semantic correctness without execution feedback [69,78].

provides a comparative summary of neural models designed for natural and programming languages. It distinguishes their representative tasks, data sources, and evaluation metrics. The comparison highlights how PL models (e.g., CodeBERT and CodeT5) extend NL counterparts (e.g., BERT and T5) by incorporating structural information such as ASTs and Data Flow Graphs (DFGs), which enable improved performance in code-related tasks while introducing new challenges in cross-language generalization and semantic correctness.

5. Neural Methods for Programming and Code-Centric SE Tasks

In this section, we explore neural methods for programming-centric SE tasks in fine-grained detail, covering the research focus, outcomes, and future directions for each task. Table 5 shows an overall summary of the 18 programming tasks, including their main objectives related to utilizing neural methods, along with their open research issues. In selecting these tasks, we prioritized those most frequently addressed in the large volume of literature, capturing widely studied areas with consistent terminology. Tasks mentioned under variant names (e.g., code recommendation vs. code completion; vulnerability prediction vs. vulnerability detection; and static/dynamic analysis vs. code analysis) were mapped to our taxonomy. Unlike some of the prior survey papers, for instance, the review by Durrani et al. [19] where the analysis relies on automated indexing tools such as Dimensions.ai, our review uses a rigorous manual literature collection and researcher-driven analysis. Our approach, which encompasses 18 distinct programming tasks, highlights the multidimensional contributions of neural methods to the field of programming. We emphasize the evolving capabilities of neural models that are the backbone of code assistant tools such as GitHub Copilot, which have not yet been sufficiently acknowledged by recent survey work such as in [19].

5.1. Source Code Translation

Source code translation can be viewed in two ways. First, it refers to the migration of outdated versions of source code to a newer version within the same PL [81]. Second, it involves translating source code from one PL to another [7,79]. In a broader sense, code translation can encompass shifts between different programming paradigms (e.g., declarative to imperative or procedural to object-oriented). It may also involve porting Application Programming Interfaces (APIs) from one language to another [82]. Although early traditional tools for code translation have some limitations, they are cost effective compared to manual translation and provide benefits to businesses across various domains. Source code translation has applications across multiple fields, including high-tech software production, cyber-security, and more. The code translation utilization areas can be seen as follows (but not limited to):

• Platform- or application-specific optimizations for high-performance computing applications [83], especially transitioning software from outdated to modern languages.
• Upgrading software to secure, up-to-date, and well-documented PLs to enhance long-term maintainability and compliance.
• Facilitating the innovation of new PLs by simplifying legacy code migration.
• Advanced program analysis and verification [84].

Existing source code translation methods can be broadly classified into three categories: rule-based, statistical, and neural translation approaches. Additionally, for cases involving a small number of lines of code, manual translation can be a viable option. Similar categorical approaches are also found in other programming tasks, such as code generation, code search, code summarization, etc.

Table 5. An overall summary of programming tasks using neural models, their objectives, and corresponding open research issues.

Task	Objective	Open Research Issues
Automatic Code Edit	Auto-modification of code.	Stable benchmark datasets for evaluation.
Code Analysis	Evaluates structural and runtime behavior of source code.	Need for multiple representations for efficiency.
Code Authorship and Identification	Utilizes neural models to attribute code to developers.	Challenges in coding style variability, multi-language authorship, and AI-generated code identification.
Code Change Detection	Tracks commit updates in large-scale software development.	Potential for IDE integration with push notifications.
Code Classification	Categorizes code based on syntax, semantics, and algorithms.	Demands of expanding classification criteria.
Code Clone Detection	Identifies duplicate or near-duplicate code snippets to maintain software quality.	Demands of distinguishing code clone detection from code similarity detection.
Code Completion	Enhances productivity by auto-completing blocks of codes.	Privacy and security concerns.
Code Generation	Automates code creation using code-assistants.	Ensuring correctness and execution reliability.
Code Modeling and Representation	Uses various representations for better source code understanding.	Designing optimal representation strategies remains an open issue.
Code Search and Retrieval	Enables retrieval of relevant code snippets.	Safety and security of AI-searched code remain a concern.
Code Similarity Detection	Identifies duplicate code using neural networks.	Challenges in dataset reliability.
Code Summarization	Generates human-readable descriptions for code.	Evaluation is highly dependent on NLP metrics.
Code Vulnerability Detection	Uses various neural methods to detect security flaws.	False positives persist and need to integrate neural methods with static/dynamic analysis.
Comment Generation	Generate descriptive comments for code.	Need for multilingual support.
Decompilation	Converts machine-level code into high-level source code.	Need for improving generalization across languages.
Program Repair and Bug Fix	Improve program debugging and efficiency.	Integrating neural methods with program verification systems for runtime bugs.
Program Synthesis	Converts NL instructions into code using neural models.	Evaluation benchmarks remain an issue.
Source Code Translation	Migrates code across different versions of the same language or translates it between different languages [85].	Difficulties in ensuring semantic and functional equivalence between the source and translated programs.

Table 5. An overall summary of programming tasks using neural models, their objectives, and corresponding open research issues.

Task	Objective	Open Research Issues
Automatic Code Edit	Auto-modification of code.	Stable benchmark datasets for evaluation.
Code Analysis	Evaluates structural and runtime behavior of source code.	Need for multiple representations for efficiency.
Code Authorship and Identification	Utilizes neural models to attribute code to developers.	Challenges in coding style variability, multi-language authorship, and AI-generated code identification.
Code Change Detection	Tracks commit updates in large-scale software development.	Potential for IDE integration with push notifications.
Code Classification	Categorizes code based on syntax, semantics, and algorithms.	Demands of expanding classification criteria.
Code Clone Detection	Identifies duplicate or near-duplicate code snippets to maintain software quality.	Demands of distinguishing code clone detection from code similarity detection.
Code Completion	Enhances productivity by auto-completing blocks of codes.	Privacy and security concerns.
Code Generation	Automates code creation using code-assistants.	Ensuring correctness and execution reliability.
Code Modeling and Representation	Uses various representations for better source code understanding.	Designing optimal representation strategies remains an open issue.
Code Search and Retrieval	Enables retrieval of relevant code snippets.	Safety and security of AI-searched code remain a concern.
Code Similarity Detection	Identifies duplicate code using neural networks.	Challenges in dataset reliability.
Code Summarization	Generates human-readable descriptions for code.	Evaluation is highly dependent on NLP metrics.
Code Vulnerability Detection	Uses various neural methods to detect security flaws.	False positives persist and need to integrate neural methods with static/dynamic analysis.
Comment Generation	Generate descriptive comments for code.	Need for multilingual support.
Decompilation	Converts machine-level code into high-level source code.	Need for improving generalization across languages.
Program Repair and Bug Fix	Improve program debugging and efficiency.	Integrating neural methods with program verification systems for runtime bugs.
Program Synthesis	Converts NL instructions into code using neural models.	Evaluation benchmarks remain an issue.
Source Code Translation	Migrates code across different versions of the same language or translates it between different languages [85].	Difficulties in ensuring semantic and functional equivalence between the source and translated programs.

Manual code translation involves an expert translating code between languages, ensuring both source and target languages meet functional requirements. While effective for small functions or short code snippets, this approach is impractical for larger codebases, as it is time-consuming, error-prone, and inefficient.

The rule-based source code translations [86,87], also known as conventional methods, are mainly dependent on the AST of the program’s source code [7]. However, there are also variations in the implementation of these code translation approaches.

In [83], the source code’s AST is converted to an eXtensible Markup Language (XML), with user-defined rules applied to optimize code translations based on platform-specific features. This research developed the Xevolver tool, built on top of the ROSE compiler infrastructure, which uses XML ASTs to enable code modifications between the source and target languages. Another example is the OP2-Clang tool [88], which utilizes the Clang/LLVM AST matcher to optimize parallel code generation. In [89], a rule-based technique using ASTs is also applied in security and code optimization for Java programs, resulting in an Eclipse plug-in tool.

Statistical methods in source code translation draw from techniques used in statistical machine translation (SMT) for NLP [90]. These approaches comprehend the source code as lexical token sequences, applying statistical language models to map these tokens between source and target languages. For instance, in [90], the source code is modeled using SMT, achieving high BLEU scores when translating Java to C#. Similarly, in [91], a phrase-based language model is applied to translate C# to Java, using a parallel corpus of 20,499 method pairs. In [92], the Semantic LAnguage Model for Source Code (SLAMC) incorporates semantic information, such as token roles, data types, scopes, and dependencies, to improve SMT-based code translation quality.

Neural networks, particularly neural language models, have greatly advanced source code translation. Numerous neural language models leverage fine-tuning techniques to excel in code-related tasks. Models like CodeBERT and CodeT5 fine-tuned for such tasks, have achieved SOTA performance [71,75].

Facebook’s TransCoder [79] employs unsupervised learning to translate between Python, Java, and C++, using monolingual word embeddings, DAE, and BT. Its self-trained extension, TransCoder-ST [93], incorporates automated unit test generation to preserve semantics during code translation. Unit tests play a crucial role not only in code translation but also in a wide range of programming tasks. As such, they are essential for self-supervised neural models to validate the functional correctness of their predictions [94]. PLBART [95] is another versatile model excelling in multiple code-related tasks, including code translation. It outperforms models like RoBERTa and CodeBERT in Java-to-C# translation based on evaluation metrics of BLEU and CodeBLEU [96].

Despite these commendable progress, challenges remain. Traditional NLP metrics like BLEU fail to capture semantic correctness in PL codes, as variations in coding style can yield low scores despite identical functionality. Effective evaluation must consider both syntax and semantics of PL code. Additionally, code translation models are often treated as a supplement to other tasks. A dedicated data pipeline and specialized neural models are needed to refine the defects that arise during code translation and to address the unique complexities of source code translation.

5.2. Code Generation

Automated code generation is crucial for various programming tasks, with diverse demands leveraging generated code’s resources [97]. Some methods use code documentation text for code generation, such as doc-strings of Python functions [98], while others integrate back-translation heuristics.

CodeT5 [71], a code model built on top of Google’s T5 model [99], incorporates NL and PL tokens, enabling bimodal (NL-PL) and PL-only training for task-specific code generation and understanding.

ASTs play a key role in structured code generation. Abstract Syntax Description Language-based semantic parsing [100], Neural Attribute Machines [101], and other research works have leveraged the AST-based structural representations of a program for code generation. Another approach stated in [102] utilizes an API-driven Java program generation, which leverages combinatorial techniques.

CoDET, short for CODE generation with generated Tests [103], evaluates correctness via test-case based code coverage, although high coverage alone does not guarantee correctness. Execution-based testing such as mutation score evaluation [104] and others offer a more reliable alternative to ensure the correctness of the model-generated codes.

Although improvements have been made over time, generated code often requires manual modifications, and ensuring error-free execution remains an ongoing challenge [105]. Code LLMs enhance efficiently the automation of code generation [106,107]; however, executability issues persist.

5.3. Comment Generation

Code comments in general-purpose programming languages provide a high-level understanding of source code written by developers. Neural models can generate descriptive comment texts for a given program [108].

An empirical study in [109] evaluates T5 and n-gram models in code comment completion tasks. The experimental demonstrations in [110] train a model that learns source code along with its comments written in Japanese NL texts, in which it distinguishes between the code and comments using procedure learning. In this research, LSTM is used for comment generation, leveraging problem statements written in the NL texts to improve program understanding. While the research work in [111] attempts automatic comment generation using NMT models, the performance is suboptimal.

In [108], the Deep Code Comment Understanding and Assessment (DComment) model is proposed to understand and classify generated comments based on their quality. Some studies explore code-to-comment translation [112] as an alternative, while evaluating the generated comments’ quality using NLP metrics. However, performance evaluation issues are still concerning as highlighted in Section 6.2.

Most comment generation models produce English-based comments, which can be a barrier for non-English-speaking programmers. Developing models that can generate comments in a multilingual fashion could enhance collaboration in software development.

5.4. Decompilation

Decompilation reverses compilation, converting machine-level code (binary or assembly code) into high-level source code. Traditional rule-based methods are costly with limited performance, prompting exploration of neural methods as alternatives. The research in [10] trains RNNs on machine code compiled from C, which is later extended into a two-phase approach [113]: generating code snippet templates and then populating them with appropriate identifier/variable values.

Neutron [114] applies attention-based NMT techniques for decompilation, while BTC (Beyond the C) [115] offers a platform-independent neural decompiler supporting multiple languages, including Go, Fortran, OCaml, and C. As reported in [116], the researchers have explored the intermediate representations (IR) of program codes with seq2seq Transformers for decompilation.

Most existing tools remain rule-based and limited to specific languages, restricting their effectiveness. While a one-fit-for-all decompiler may be infeasible, future research should focus on leveraging efficient neural models to enhance the generalization of this task across diverse languages.

5.5. Code Search and Retrieval

Developing software from scratch is time-consuming, which requires software engineers to integrate open-source code. However, effective code retrieval requires robust search tools. While traditional search engines excel at NL queries, they struggle with accurate code snippet search. Neural methods offer a promising alternative.

Code-Description Embedding Neural Network (CODEnn) [117] is a deep learning-based code search model that leverages RNN-based sequence embeddings for joint source code and description representations. CARLCS-CNN, short for Co-Attentive Representation Learning Code Search-CNN [118], enhances code search with co-attention mechanism and CNNs to improve search accuracy. The Code-to-Code Search Across Languages (COSAL) model proposed in [119] enables cross-lingual code search using static and dynamic analysis, while Deep Graph Matching and Searching (DGMS) [120] employs Relational Graph Convolutional Networks (RGCNs) and attention mechanisms for unified text-code graph matching.

Traditional information retrieval techniques have also been enhanced with neural approaches. For instance, the study in [121] integrates Word2Vec into retrieval tasks. The Multi-Modal Attention Network (MMAN) in [122] utilizes LSTM, Tree-LSTM, and Gated Graph Neural Networks (GGNNs) to capture syntactic and semantic information. In [123], the authors combine CNNs and joint embeddings for Stack Overflow queries. The Multi-Programming Language Code Search (MPLCS) model proposed in [124] extends retrieval across multiple PLs, benefiting low-resource PLs.

AI-driven tools like Microsoft Copilot and ChatGPT [3] have revolutionized code search, retrieving relevant snippets via NL prompts. Unlike traditional engines, ChatGPT bypasses irrelevant search results but still may return non-executable code. Future improvements could integrate SE tools with neural models to search a code which is ready-made for execution with no errors, enhancing usability and reliability in software development.

5.6. Code Completion

IDEs with auto-completion features significantly enhance software development productivity [125]. Various neural methods have been explored for code completion and related tasks. In [126], LSTMs with attention mechanisms were used for code completion and other group of coding tasks. The study in [66] proposes a pre-trained Transformer-based model with multitask learning to predict code tokens and their types, which is finally fine-tuned for efficient code completion.

Microsoft’s IntelliCode Compose [127] is a cloud-based tool that suggests entire lines of code with correct syntax. This model provides monolingual embeddings while leveraging multilingual knowledge, benefiting low-resource programming languages. Additionally, it enhances privacy by preventing exposure of sensitive information, addressing a key limitation in prior auto-completion tools.

Meta’s research work in [128] demonstrates the effectiveness of transfer learning for code auto-completion. Their approach is built on top of the GPT-2 and BART Transformer-based models, and applies auto-regressive and DAE objectives to improve predictions.

A critical concern in code completion is privacy and security as noted in [127]. Furthermore, since prompts used for code completion tasks often have similar structures, there is a risk that generated programs will share similar designs, potentially introducing common vulnerabilities. Future research should address these concerns by enhancing model diversity and privacy safeguards in automated code completion. Additionally, most current code auto-completion assistants struggle to retain the long-context of earlier parts of the code written by the programmer. As a result, they often suggest generic and repetitive line completions. To enable more intelligent and context-aware assistance, it is essential to develop code completion systems capable of capturing long-range dependencies, potentially across an entire file or even multiple files within a project. Such capabilities are critical to providing seamless and relevant code suggestions.

5.7. Automatic Code Edit

Editing and fixing awkward parts of code structures are routine yet time-consuming tasks for programmers. These repetitive activities consume significant effort, but recent neural models aim to automate them [129].

CodeEditor [130] is a pre-trained code editing model whose results demonstrate improved performance and generalization capabilities in code-editing tasks. It consists of three stages: collecting code snippets from repositories like GitHub, generating inferior versions (versions somewhat different from the ground truth) of the code for pre-training, and evaluating the pre-trained model under three settings, namely fine-tuning, few-shot, and zero-shot.

The multi-modal NMT-based code editing engine referred as MODIT in [9] processes code in three steps: preprocessing, token representation via an encoder–decoder attention mechanism, and output generation. The preprocessing phase integrates three input modalities: code location, contextual information, and the commit messages, which guide the editing process.

A major challenge in this field is the lack of stable benchmark datasets for source code editing. The exponential growth of demands on software development further compounds this issue, as modern applications span huge numbers of lines of code, making manual code editing tiresome. For automatic code-editing models to be effective and scalable in complex coding environments, they should be trained on diverse and inclusive source code corpora. Addressing this gap is crucial for advancing automated code editing techniques.

5.8. Code Summarization

Programmers frequently read source code written by others, requiring the detailed intuition of a particular program. Automating this process through code summarization, that is, generating human-readable descriptions for code snippets, can significantly enhance efficiency. Neural methods have shown great success in this area.

M2TS, short for Multi-Scale Multi-Modal Approach Based on Transformer for Source Code Summarization [131], utilizes AST structures and token semantics using a cross-modality fusion approach for improved code summarization. Another model in [132] applies reinforcement learning with triplet code representations such as CFG, AST, and plain texts to enhance code summary generations.

In [133], a hybrid model combines token sequences (encoded using c-encoder) with semantic graphs (encoded using g-encoder) for richer context. Structural information from code snippets has inspired research such as AST-Trans [134], which transforms AST graphs into sequences using traversal and linearization techniques, thereby reducing computational costs. In [135], the model predicts action words from code blocks, identifying their intent and job class to enhance summarization.

Although significant improvements have been achieved in the research area of code summarization, evaluating the performance of code summarization models remains challenging. NLP-based metrics fail to capture source code semantics, often leading to misleading results. Research into specialized evaluation metrics tailored for source code is essential to ensure accurate and meaningful assessments of code summarization models.

5.9. Code Change Detection

Tracking updates in large-scale software projects is challenging, especially for a software development distributed across multiple teams, which may require frequent commits and updates from each team. Neural methods have been introduced to assist programmers in efficiently managing code changes.

CORE, short for COde REview engine [136], employs LSTM models with multi-embedding and multi-attention mechanisms to review code changes, effectively capturing semantic modifications. Another model, CC2Vec [137], represents code changes by analyzing modifications across updated source code files. It preprocesses change patches by tokenizing modified code blocks and constructing a code vocabulary. Structural information from added and removed code is then processed via a Hierarchical Attention Network for better representation.

IABLSTM (short for Impact Analysis with Attention Bidirectional Long Short-Term Memory) [138] utilizes a Bidirectional Long Short-Term Memory (Bi-LSTM) with an attention mechanism to detect source code changes. It identifies differences between the original and modified code using cosine similarity, leveraging AST paths and vectorized representations. Another approach in [139] applies NMT techniques to track meaningful code modifications in pull requests, aiding collaborative development.

Along with the previous efforts, code change detection remains an active research area. A promising direction is integrating these models with push notification systems within IDEs, providing real-time alerts about modifications. This would enhance efficiency in team-based software development, ensuring developers stay informed seamlessly.

5.10. Code Similarity Detection

The rising demand for software applications has led to increased source code duplication, where the same code is reused across projects. This issue undermines creativity, infringes upon intellectual property rights, and raises privacy concerns. Neural methods have emerged as promising solutions for detecting code similarity.

The model in [11] applies NLP techniques for code similarity detection. It preprocesses source code through stemming, segmentation, embedding, and feature extraction, generating vectorized representations of code pairs. Cosine distance is then used to compute similarity scores. Another approach in [140] focusing on Scratch, a visual PL, utilizes Siamese Bi-LSTM model to capture syntactic and semantic code similarities. Manhattan distance metric is utilized to evaluate similarity between the Scratch files.

Cross-language code similarity detection is another active research area. The approach in [141] transforms source code of multiple PLs into control flow charts, applying graph similarity detection techniques to compare them.

A major challenge in code similarity detection research is the reliability of datasets. Open-source repositories may contain default IDE-generated templates, leading to unintentional similarities. Future research should focus on differentiating manually written and machine-generated code to enhance code similarity detection. Addressing these challenges is crucial for maintaining ethical and professional standards in software development.

5.11. Program Synthesis

Ideally, computer-literate end users should be able to interact with machines using clear NL instructions. However, for this to be possible, intelligent systems are needed to convert NL instructions into executable code. Program synthesis [142,143] addresses this challenge by developing models that automatically generate programs based on user-defined instructions.

PATOIS [144] is a program synthesis model which incorporates code idioms via structural components called probabilistic Tree Substitution Grammars (pTSG). Its encoder embeds NL specifications, while its decoder generates token representations of ASTs. LaSynth [145] is an input–output-based program synthesis model, focused on compiled languages like C, by integrating latent execution representations. During training, its loss function combines latent executor loss and token prediction loss. The model also generates input–output pairs to improve supervised program synthesis approaches.

CodeGen [146] is a Salesforce’s Code LLM, which employs an auto-regressive method similar to GPT models, predicting code tokens based on prior prompts. Its multi-turn programming benchmark is designed to scale problem sets according to the size of the model and dataset.

However, program synthesis model evaluation benchmarks remain an open challenge. Some studies propose synthetic datasets [147], while others generate domain-specific input–output examples [148]. Further research is needed to establish standardized and versatile evaluation benchmarks, ensure the executability of source code generated by program synthesis, and develop optimal performance evaluation metrics.

5.12. Code Modeling and Representation

This part of our survey presents existing approaches to code representation and modeling in neural-based programming tasks [149]. Effective programming tasks require robust code representation and modeling techniques [150], as these representations directly influence how neural models interpret and process source code.

Several studies have explored different code representation techniques. In [151], source codes are utilized as pair-wise AST node paths, offering a generalizable approach. Another work [152] employs IR and contextual flow graphs (XFG) to enhance semantic code representation. Graph-based methods have also gained attention in the code representation research. Message passing and grammar-based methods in [153] represent the semantics of code structurally. GraphCodeBERT [69] leverages data flow structures to model relationships between variables and computational flow. The Open-vocabulary neural language model (Open Vocab NLM) in [154] integrates Gated Recurrent Units (GRUs) and sub-word tokenization, enabling it to process billions of tokens dynamically. AST-based approaches are evolved in [155], where ST-trees mitigates long-range dependency issues using bidirectional GRUs. Flow2Vec [156] converts code to low-dimensional vector representations using high-order proximity embeddings. CodeDisen [157] leverages a variational autoencoder to disentangle syntax and semantics across PLs. SPT-code [158] incorporates code sequences, ASTs, and NL descriptions for tasks like summarization, completion, bug-fixing, translation, and code search. Another model in [116] aligns embeddings of different PLs augmented along with their IR representations and actual tokens of the source code for the code translation task.

This survey highlights a wide range of representations, from raw source code tokens, ASTs, and CFGs to IR and DFG. While combining various code representations often improves comprehension, systematic research is needed to determine the optimal combination of these possible code representation options. Enhancing the neural-based code-modeling relies on robust, task-specific representation strategies. Hence, exploring the effective augmentation of various code representation alternatives could significantly broaden the research in this field.

5.13. Code Classification

Understanding and identifying source code across multiple PLs requires intuitively grasping the objectives and distinguishing features of programs. Traditional methods, such as manual cross-checking or using predefined rules, are effective but impractical for large-scale projects, especially for developers unfamiliar with multiple PLs. Code classification offers an efficient solution to these challenges by categorizing source code based on syntax, semantics, or other defined characteristics.

In [159], Multinomial Naive Bayes (MNB) is used as the classifier, and the Term Frequency-Inverse Document Frequency (TF-IDF) is employed for feature extraction. This method classifies code snippets from 21 PLs using the Stack Overflow corpus and identifies specific language versions. Deep learning has also further improved classification accuracy. For example, Reyes et al. [160] employs an LSTM with word embeddings and dropout regularization, outperforming traditional classifiers. The research in [161] uses CNNs to classify source code across 60 PLs, achieving high F1 scores. A CNN-based approach in [162] predicts source code categories based on algorithmic structures rather than keywords, using an online judge system as its corpus. In [163], the authors leverage topic modeling and JavaParser preprocessing to identify code block functionalities for Java code classification. Beyond standard code classification tasks, Barr et al. [164] combine deep learning with combinatorial graph analysis to detect and classify code vulnerabilities, using code2vec and LSTM embeddings. The research work in [165] classifies semantic units of machine learning code using LSTM-GRU with attention mechanisms, applying MixUp augmentation to overcome data scarcity.

Code classification methods in the papers reviewed so far can fall into four broad categories such as algorithm or problem domain based, language based, code functionality based, and vulnerability based as shown in Figure 5. However, additional criteria such as coding paradigms (imperative, declarative, functional, and object-oriented) can be a further supplement to code classification models. Future research should focus on expanding classification taxonomies, ensuring a comprehensive categorization basis for an all-inclusive code classification task.

5.14. Code Vulnerability Detection

Source code vulnerabilities arise when errors and security flaws remain unpatched, making automated detection essential for mitigating risks. In [166], CNN and RNN deep learning models are utilized to extract feature representations and detect vulnerabilities in C and C++ code.

The Automated Vulnerability Detection framework based on Hierarchical Representation and Attention Mechanism (AVDHRAM) proposed in [167] improves vulnerability detection by structuring source code into five hierarchical levels: program, function, slice, statement, and token. However, the primary focus of this model is statement-level analysis enhanced by attention mechanisms. Another approach by Mao et al. [168] introduces a Bi-LSTM model that serializes ASTs while incorporating attention mechanisms to improve accuracy in classifying vulnerable functions. Transformer-based vulnerability detection presented in [169] uses fine-grained code slices for analyzing function calls, pointers, and expressions. In [170], three deep learning models combined with multiple code representations improve vulnerability detection.

Graph-based models like the one in [171] employ GGNN and inductive learning, using Common Vulnerability and Exposure (CVE) functions to create code slice graphs. In this work, the attention layer effectively identifies relationships among graph nodes, enhancing vulnerability detection. Wang et al. [172] combine gated graph networks and control flow analysis to distinguish benign from buggy code, addressing data scarcity by leveraging training data from open-source repositories.

Although some remarkable findings are undeniable in this area, vulnerability detection models still face challenges such as false positives and limited generalization to unseen flaws. Identifying vulnerabilities alone is insufficient; thus, classification according to severity, type, and potential impacts is crucial for comprehensive vulnerability management. Integrating neural models with static and dynamic program analysis or formal verification [173] could improve robustness. Future research should explore these integrations experimentally, advancing comprehensive vulnerability detection methodologies.

5.15. Code Analysis

Source code analysis involves multiple stages, serving various purposes and applications [174]. Prior research has explored different attributes for code quality assessment. In [175], static and dynamic properties of CFGs are utilized for code analysis. Ramadan et al. [176] perform code analysis by evaluating the execution speed of a particular program based on AST structures, generating code pairs to identify the snippet with faster execution speed.

A dynamic program analysis algorithm proposed in [177] leverages AST and CFG representations, storing contextual information in Translation Units. In [178], a hybrid analysis tool combines historical and structural schemes, featuring a Java-based parser, a Python-based Code History Miner, and an interactive interface. GENESISP, introduced in [179], integrates source and binary code data for Debian-based environments, employing 13 analysis tools. Though its neural basis is unclear, its user interface facilitates database collection and analysis of open-source software.

Although the current research in source code analysis is commendable, most neural models rely heavily on AST representations. While ASTs capture structural aspects, they fail to convey comprehensive functional and semantic information about the code. Therefore, integrating multiple representations alongside AST information can enhance the robustness of source code analysis.

5.16. Code Authorship Identification

With growing concerns over intellectual property in software development, accurate code authorship identification is essential for innovation and guaranteeing ownership rights [180]. Several studies have explored neural-based approaches to tackle this problem.

Abuhamad et al. [181] introduce a CNN-based method utilizing word embeddings and TF-IDF representations, demonstrating promising authorship attribution results on the Google Code Jam (GCJ) dataset. Kurtukova et al. [182] propose a hybrid neural network combining CNN-Bidirectional GRU (C-BiGRU), LSTM, Bi-LSTM, and other models, training on vectorized datasets before applying them to anonymized source code for authorship identification. Expanding on this, in [180], they tackle more complex real-world scenarios such as multi-language authorship attribution, coding style variations, and AI-generated code identification. Omi et al. [183] introduce a model capable of identifying multiple contributors to a single codebase by converting code snippets into AST paths and using ensemble classifiers. A CNN-based model, along with a KNN classifier, is presented in [184], emphasizing explainability in authorship attribution.

A key challenge in authorship attribution is the variability in coding styles across different PLs. Programmers often write in multiple styles, complicating the identification task. Moreover, IDEs generate code templates, such as ASP.NET MVC scaffolding in Microsoft Visual Studio, and AI-assistants can also produce large amounts of code, making it difficult to distinguish programmer-written and machine-generated code. Although progress has been made, future work should address the evolving programming practices, mixed data sources, and auto-generated code for robust authorship attribution and identification.

5.17. Program Repair and Bug Fix

Troubleshooting errors and fixing bugs in source code are daily challenges for every programmer. Manually handling these tasks for large codebases is cumbersome, necessitating automated tools. Ensemble Learning using Convolution Neural Machine Translation for Automatic Program Repair (ENCORE) is proposed in [185], leveraging a seq2seq encoder–decoder architecture with an attention mechanism and ensemble learning. The model follows three stages: input representation (tokenization), training ensemble models, and validating patches. ENCORE successfully repairs code errors in four PLs and shows potential adaptability for more PLs. Liu et al. [186] address lexical and semantic gaps between bugs and their corresponding fixes by employing lexical and semantic correlation matching, combined with focal loss, to tackle data imbalance in buggy and non-buggy classifications.

While most SOTA approaches fine-tune the pre-trained NLP models for programming tasks, Jiang et al. [187] pre-train a pure PL model on large codebases, then fine-tune it for APR. Their approach introduces a code-aware beam search strategy to manage syntax constraints and sub-word tokenization to mitigate out-of-vocabulary (OOV) issues. CIRCLE (Continual Repair aCross Programming LanguagEs) [188] is a cross-language APR model, which extends T5-based pre-trained NLP models with continual learning to repair code across multiple PLs. This approach enhances adaptability and generalization.

In real-world software development, bugs are often fixed following the post-failure measures. However, integrating self-healing mechanisms [189] with bug detection and fault localization [190] could revolutionize APR by preventing data loss or system crashes. Despite their promise, LLMs and Code LLMs struggle with runtime bugs unseen during their training stage. Addressing this requires integrating LLMs with external validation systems that provide automatic feedback for invalid patches. Ongoing research explores patch validation and refinement [189], paving the way for end-to-end APR frameworks that enhance the entire software development life cycle.

5.18. Code Clone Detection

Most modern software applications rely heavily on code reuse across their development stage. While this practice accelerates software development and productivity, it can lead to issues such as code bloating and software quality degradation, collectively referred to as code clone problems. Code clone detection models address these challenges. For instance, the research work in [191] proposes a CNN-based model with two convolutional and pooling layers, leveraging the BigCloneBench dataset for evaluation of the model on code clone detection.

Zhang et al. [192] design a clone detection method that aligns similar functionalities between source code snippets, even when their structure is different. This model uses sparse reconstruction and attention-based alignment techniques, relying on similarity scores to detect clones. Meanwhile, Zeng et al. [193] focus on computational efficiency, significantly reducing runtime. They use a weighted Recursive Autoencoder (RAE) and process source code in two phases: feature extraction to generate program vectors and clone detection via Approximate Nearest Neighbor Search (ANNS). They use Euclidean distance as a similarity metric. Functional clone detection, which identifies functionally equivalent but different code implementations, is explored in [194]. This hybrid model combines sequential tokens, syntactical ASTs, and structural CFGs to represent the source code and is evaluated on Java-based clone corpora. Recent research has expanded to cross-language clone detection as demonstrated in [195], which identifies clones across multiple PLs.

While sometimes the term code clone detection overlaps with code similarity detection, they serve distinct purposes. Although code clone detection can be considered a subset of code similarity detection, many studies use these terms interchangeably which may encounter confusion for readers. Future research should clearly distinguish these two tasks while developing an effective clone detection strategy for large-scale projects. Finally, leveraging LLMs or Code LLMs in code clone detection tasks holds promise for enhancing accuracy and scalability. By adopting recently up-to-date methodologies and expanding cross-language code clone detection capabilities, future research can offer robust solutions and clarify the distinction between code clone detection and code similarity detection, thereby improving software quality and maintainability.

To offer a deeper analytical view, we further grouped neural methods across the programming tasks based on their architectural families and learning paradigms. Broadly, these include sequence-based models (e.g., RNNs, LSTMs, and GRUs), Transformer-based large language models (e.g., CodeBERT, CodeT5, and PLBART), and graph-based neural networks (e.g., GGNN and RGCN). Each group exhibits distinct strengths and limitations. Sequence-based models excel at capturing short-range dependencies but struggle with long-contextual understanding; Transformer-based architectures handle long-range dependencies effectively, but they demand extensive computational resources and training data; graph-based approaches model code structure more explicitly but often sacrifice generalization due to task-specific designs.

Table 6. Comparison of major neural method families applied to programming and SE tasks.

Group Name	Advantages	Disadvantages	Common Application Areas
Sequence-based (RNN, LSTM, GRU)	Captures token-level dependencies; effective for small datasets; easy to fine-tune.	Limited context retention; sequential processing slows inference.	Comment generation, code completion, bug-fixing.
Transformer-based (CodeBERT, CodeT5, CodeT5+, PLBART, GPT-family)	Captures long-range context; supports parallel training; high generalization across PLs.	High computational cost; prone to hallucination; sensitive to noisy data.	Code translation, generation, summarization, synthesis.
Graph-based (GGNN, RGCN, AST-GNN)	Encodes structural and semantic information explicitly; interpretable representations.	Task-specific tuning required; limited scalability on large codebases.	Code analysis, clone detection, code search and retrieval.

summarizes the overall comparison of these neural methods’ categories in the context of programming-centric SE tasks.

6. Data, Benchmarks, and Evaluation Metrics

6.1. Datasets and Benchmarks

The scientific literature on the applications of neural networks in programming and SE primarily focuses on model designs and architectural advances. Neural networks form the foundation of deep learning and large parameter size models. The emergence of LLMs and Code LLMs, with a large number of parameters, marks a significant milestone in the field. However, a review of publication trends reveals that datasets and benchmarks often receive less emphasis and are treated as supplementary rather than core components of the research. This neglects the crucial role of data in the advancement of deep learning and AI, particularly in programming tasks, where high-quality training data is essential for model effectiveness.

In this context, we take a chronological deep dive into the major datasets and benchmarks introduced for programming tasks. One such example is the Django dataset introduced in [196], which is a parallel corpus of Python code and corresponding NL text descriptions. Salesforce’s encoder–decoder architecture-based CodeT5 model and a retrieval-augmented model kNN-TRANX were among the models evaluated on this dataset [197]. However, this dataset is small in size (18,805 pairs) and is narrowly scoped which ultimately support only the Python-related code generation tasks. Two years later (in 2017), a data corpus consisting of 108,726 Python Code and Doc-string pairs (PCSD) was released, where it incorporates class declarations, class methods, module doc-strings, and commit SHAs in its latest version [98,198]. The M2TS model for code summarization introduced in Section 5.8 is among the models evaluated on this dataset [131]. However, referring to the literature that explores the dataset, we observe that the PCSD data corpus has some noise, and the comments (doc-strings) miss some details.

The CoNaLa dataset is a benchmark comprising approximately 2.4 K curated pairs of English-Python data, extracted from Stack Overflow’s programming-related question–answer discussions and documentation [198]. Several neural models were evaluated on this dataset, such as CodeLLaMA, CodeT5+, and CodeGen among others [199].

Another notable dataset is Devign, presented at NeurIPS 2019 and consisting of 27,652 vulnerable and 31,313 non-vulnerable C language functions [200]. As the name suggests, this dataset is dedicated to identifying vulnerability patterns in C programs. It was manually curated from four popular open source C-language projects. The dataset is well-regarded for its quality in vulnerability identification tasks, as data collection and labeling were performed by a team of security experts, requiring approximately 600 man-hours.

Given that neural methods for programming tasks are still evolving, data processing methodologies are largely derived from NLP. Consequently, most of the code corpora utilized in scientific research articles focused on the application of neural methods in SE are mainly dominated by code generation using NL texts [201]. Notable examples include datasets mapping NL to SQL code [202]. In recent related studies, other open-sourced datasets have been designed and released for both NL-to-code and code-to-code generation tasks [203,204,205]. Semantic parsing is integral to the processing of the code corpus, while static parsing methodologies [206] aid in understanding structural code information.

Recently, numerous large-scale code-specific datasets have been released to boost the neural methods involved in the programming tasks [207,208]. For instance, in [209], a Python class-level code generation dataset containing 842K class skeletons (methods and attributes with doc-strings) has been released. This dataset was extracted from 13,174 real-world GitHub projects.

Online coding competition platforms [210], repositories like GitHub, and technical Q&A forums such as Stack Overflow [198] are key sources of code corpora. However, to date, many of the code corpora released by researchers and practitioners lack standardized dataset pipelines and detailed documentation. In response, commendable efforts have been made to provide well-designed datasets through the HuggingFace Datasets Hub [211].

However, the development of standardized dataset pipelines for programming tasks is still in its early stages. Although many processed code-related datasets are available via the HuggingFace Datasets API, several challenges arise when exploring datasets for the same task. For instance, datasets for tasks like vulnerability detection, code translation, or code generation may come from different contributors. These datasets often suffer from inconsistent data formats, varying feature sets, and undocumented preprocessing steps, making reproducibility difficult for future researchers.

To summarize,

Table 7. An overview of datasets curated for programming and software engineering tasks, utilized by neural models in code intelligence research.

Dataset (Year)	Task(s)	Trained/Evaluated Models	Metrics
Defects4J (2014)	Bug detection and repair (Java)	GenProg, Nopol, jKali, ASTOR, RepairAgent [212], ThinkRepair [213]	Repair success rate, Pass@k
BigCloneBench (2014)	Code clone detection	GNN, Code2Vec, CodeBERT, GraphCodeBERT	Precision, Recall, F1
Django (2015)	NL-to-code generation, pseudo-code generation	BART, CodeT5, kNN-TRANX	BLEU, CodeBLEU, EM
QuixBugs (2017)	Program repair (buggy algorithms)	GenProg, PolyCoder, Codex	Ranking test-passing patches, BLEU, Pass@k
PCSD (2017)	Code summarization, Comment Generation	Seq2Seq, CodeBERT, DECOM	BLEU, ROUGE, METEOR
CoNaLa (2018)	Code generation	CodeLLaMA, CodeGen, CodeGen2, CodeT5	BLEU, CodeBELEU, EM
Concode (2018)	Code generation	Encoder-decoder, Bi-LSTM,	BLEU, EM
CodeSearchNet (2019)	Code search and retrieval	Bag-of-words, CNN, RNN, CodeBERT, GraphCodeBERT	MRR, BLEU, ROUGE, Normalized Discounted Cumulative Gain at rank k (NDCG@k)
Devign (2019)	Vulnerability detection (C/C++)	GGNN, CNN, LSTM	Accuracy, F1
JuICe (2019)	Contextual code generation, API call	Transformer, LSTM	EM, BLEU, Accuracy, Precision, Recall
Bears (2019)	APR and bug fixing	MUFIN, etc.	Test suite correctness
TransCoder (2020/2022/2023)	Code translation (Java, C++, Python, Go, Rust)	TransCoder, TransCoder-ST, TransCoder-IR, rule-based baselines, GPT-3.5, LLaMA, CodeGen	BLEU, EM, Computational Accuracy (CA), Pass Rate
CodeXGLUE (2021)	Defect detection, code translation, clone detection, and more.	CodeBERT, GraphCodeBERT, CodeGPT, Encoder–Decoder	BLEU, ROUGE, Accuracy, F1, MRR
MBPP (2021)	Python code generation	GPT-3, WizardCoder, etc.	Pass@k, BLEU
HumanEval (2021)	Python program synthesis	Codex, GPT-3, GPT-Neo, etc.	Pass@k
APPS (2021)	Code generation	GPT-3, GPT-Neo, GPT-J, Codex	Pass@k
CodeNet (2021)	Code classification, code similarity, code translation	Multilayer Perceptron, Graph Convolutional Networks (GCNs), GNN, PLBART	Accuracy, BLEU, Runtime metrics
XLCoST (2022)	Code translation, code summarization, code search, and more.	CodeBERT, PLBART, CodeT5, RoBERTa, and more.	BLEU, CodeBLEU, MRR
CodeContests (2022)	Code generation	AlphaCode, GPT-4o mini, Qwen2.5-Coder-7B-Instruct, and more.	Test-case success rate, Pass@k
SecurityEval (2022)	Vulnerability detection (Python)	GPT-4o, InCoder, GitHub Copilot, AutoSafeCoder	Vulnerability occurrence rate, Pass@k
DiverseVul (2023)	Vulnerability detection (C/C++)	CodeBERT, CodeGPT, PolyCoder, CodeT5, NatGen, and more.	Accuracy, Precision, Recall, F1, False Positive Rate
CodeTransOcean (2023)	Multilingual code translation	GPT-4, CodeT5+, Claude-3.5	EM, BLEU, CodeBLEU, Debugging Success Rate (DSR@k), Pass@k
DS-1000 (2023)	Data science code generation	CodeGemma, DeepSeek-Coder, CodeLLaMA, Incoder, WizardCoder, and more.	Pass@k, Test Cases + Surface-Form Constraints
ClassEval (2023)	Class-level code generation	GPT-4, GPT-3.5, SantaCoder, WizardCoder, Intstruct-CodeGen, CodeGeeX, and more.	Pass@k
MCoNaLa (2023)	Multi-NL code generation and summarization on Python	gpt-3.5-turbo, mT5, CodeT5, mBART, PLBART	EM, BLEU, CodeBLEU
PrimeVul (2024)	Vulnerability detection (C/C++)	GPT-3.5, GPT-4, CodeT5, UniXCoder, StarCoder2, and more.	F1, Precision, Recall
EvoCodeBench (2024)	Code generation (Python)	GPT-4, GPT-3.5, StarCoder2, CodeLLaMA-7B, DeepSeekCoder-7B, StarCoder2-7B, and more.	Pass@k, Recall@k, Debugging Success Improvement (DSI)
Statement-level Code Summ. (2024)	Statement-level summarization	GPT-4, GPT-3.5, CodeLLaMA, StarChat	BLEU, SBCS (SentenceBERT + Cosine Similarity), SBED (SentenceBERT + Euclidean Distance), Human Eval
Python ClassGen (2025)	Class-level Python code generation	GPT-4	ROUGE-L, BLEU, Tree Similarity of Edit Distance (TSED)
DEFECTS4J-TRANS [Transformed Defects4J] (2025) [214]	APR	Magicoder-S-DS, WaveCoder-Ultra, CodeQwen1.5, DeepSeek-Coder-Instruct, and more.	plausibility and correctness of patches (plausible patch, correct patch)

provides a summary on a significant number of datasets used in programming and SE tasks, arranged in chronological order of their release. It provides a concise summary of each dataset, such as models evaluated on these datasets, targeted tasks, and evaluation metrics used to measure model performance. The critical analyses of various evaluation metrics along with their mathematical formulation are discussed in Section 6.2 below.

6.2. Evaluation Metrics

The progressive development of neural models for programming tasks, particularly those trained on code, is heavily influenced by NLP techniques. Even with the emergence of LLMs and Code LLMs, capable of interpreting code across various PLs, NLP-based evaluation methods remain widely adopted [49]. While not ideally suited for programming tasks, these metrics serve as a foundational baseline in the absence of standardized, code-specific evaluation frameworks. However, recently emerging code-specific evaluation metrics seem to be promising for measuring the rapidly growing range of programming tasks. Based on the evaluation strategies utilized in the major scientific studies reviewed in this survey, the metrics commonly employed to assess model performance in programming tasks can be categorized into five major groups as detailed below.

6.2.1. Text-Match and N-Gram Overlap-Based Metrics

Match-based or n-gram overlapping metrics assess model performance by quantifying the surface-level word matching (n-gram overlap) between model-generated outputs and ground-truth reference outputs. One of these metrics is the Exact Match (EM), which allows no deviation between the generated and reference code. Due to its rigidity, EM is generally not recommended for code generation tasks [215].

A more widely adopted metric is BLEU, originally developed for machine translation [216]. BLEU has been applied to programming tasks such as code translation and summarization. However, it overlooks key syntactic and semantic characteristics inherent in PLs.

ROUGE is another metric in this group, primarily used for evaluating code summarization and comment generation. It relies on matching the longest common subsequence between the generated and reference outputs [217]. However, like BLEU, it fails to account for semantically equivalent yet syntactically different code snippets.

To address such limitations, CodeBLEU incorporates both syntactic and semantic information through AST matching and data-flow matching [96]. Another alternative evaluation metric is CrystalBLEU, which improves upon BLEU and CodeBLEU by ignoring trivial n-gram overlaps that may result in a misleading match between code snippets with different logical purposes. CrystalBLEU has shown efficiency improvements ranging from 1.9 to 4.5 times over BLEU and CodeBLEU and also outperforms other BLEU variants such as Ruby [218,219]. METEOR, another widely used metric in this category, incorporates synonymy and stemming and has been adapted to certain code evaluation tasks [220].

Some of the core formulations for these metrics are provided below.

Assume that N denotes the maximum n-gram length (typically 4); $w_n$, which is commonly computed as $1/N$, represents the weight for each n-gram precision $p_n$; $p_n$ is the modified n-gram precision; and $\mathrm{BP}$ is the brevity penalty. The BLEU score is computed as follows:

$$\mathrm{BLEU}=\mathrm{BP}·exp\left(\sum _{n=1}^N{w}_{n}log{p}_n\right)$$

$$\mathrm{BP}=\left\{\begin{array}{cc}1\hfill & \mathrm{if}c>r\hfill \\ exp\left(1-{\displaystyle \frac{r}{c}}\right)\hfill & \mathrm{if}c\le r\hfill \end{array}\right.$$

where c and r denote the lengths of the candidate and reference outputs, respectively.

The CodeBLEU score integrates semantic and structural signals. Assuming ${\mathrm{BLEU}}_{\mathrm{weight}}$ denotes the token-wise weighted n-gram precision, ${\mathrm{Match}}_{\mathrm{ast}}$ and ${\mathrm{Match}}_{\mathrm{df}}$ compute the syntactic and semantic matches, respectively, and $\alpha ,\beta ,\gamma ,\delta$ are tuning hyperparameters, the CodeBLEU score is computed as follows:

$$\begin{array}{cc}\hfill \mathrm{CodeBLEU}=& \alpha ·\mathrm{BLEU}+\beta ·{\mathrm{BLEU}}_{\mathrm{weight}}\hfill \\ \hfill & +\gamma ·{\mathrm{Match}}_{\mathrm{ast}}+\delta ·{\mathrm{Match}}_{\mathrm{df}}\hfill \end{array}$$

Despite improvements, all these metrics primarily evaluate surface-level match and lack insight into the actual execution or correctness of the model-generated code.

6.2.2. Embedding-Based Similarity and Learned Metrics

Embedding-based and learned metrics aim to capture semantic similarity between generated and reference code by comparing their vector representations or contextual embeddings. These metrics are well-suited for tasks such as code summarization [221], clone detection, code generation, and code search.

A foundational metric in this category is Cosine Similarity (Embedding CosSim), which ranges from $-1$ to $+1$ and measures the angular similarity between two vectors (e.g., P and Q):

$$cos(\mathrm{P},\mathrm{Q})={\displaystyle \frac{\mathrm{P}·\mathrm{Q}}{\parallel \mathrm{P}\parallel \parallel \mathrm{Q}\parallel }}$$

where $·$ denotes the dot product and $\parallel ·\parallel$ is the vector norm. Cosine similarity has been widely used in assessing various models like CodeT5+ and GPT-series models. Recent evaluations of these models involve Verilog code generation and comprehension tasks among the notable examples [222].

CodeBERTScore, built upon cosine similarity, utilizes pre-trained CodeBERT embeddings to compare code/comment pairs and has demonstrated high alignment with human judgment [223]. However, its applicability is restricted to a few supported PLs.

Euclidean distance is another metric, defined for two vectors P and Q as shown below:

$$d(\mathrm{P},\mathrm{Q})=\sqrt{\sum _i{(P_i-Q_i)}^2}$$

Manhattan distance, also known as the L1 distance, is defined as

$$d(\mathbf{P},\mathbf{Q})=\sum _i|P_i-Q_i|$$

These two distance metrics are often employed in code similarity and clone detection tasks where structural information is less critical.

Another important metric is MRR commonly used in code search and retrieval-augmented generation applications. MRR computes the average of the inverse ranks of the first correct result and has been widely adopted in evaluations on CodeSearchNet [224].

BLEURT, a learned regression-based metric initially introduced for NLP evaluation, has recently been adapted for code summarization tasks [225]. Other metrics in this category include BARTScore, GPTScore, and NDCG, each tailored for specific code-related tasks.

Despite their advantages, these metrics face the following limitations:

• They are unsupervised and depend heavily on the quality of the embeddings.
• They often treat code and comments as plain text, disregarding structural and semantic intricacies such as AST nodes and API hierarchies.

Hence, while embedding-based and learned metrics enhance evaluation by moving beyond surface-level token matching, the growing complexity and diversity of programming tasks suggest that structure-aware, execution-based metrics may provide more accurate and holistic assessments.

6.2.3. Classification Metrics

Classification metrics broadly span from binary classification to multi-class and sequence-level accuracy. They are used to evaluate the correctness of classifiers on specific code-related tasks within the context of this survey. The values of classification evaluation metrics typically range from 0 to 1, or 0 to 100%. These evaluation metrics such as accuracy, precision, recall, F1, and AUC (Area Under the ROC Curve) are widely recognized as standard and reliable classification metrics not only in programming and code-related tasks but also across various other research domains involving classification activities. Several tasks such as code clone detection, code smell detection, source-code classification, and code vulnerability detection depend on such evaluation metrics [166,200,226,227,228,229,230].

Most classification metrics are derived from the following four fundamental attributes of a classification model’s inference results:

• True Positive (TP): represents the correct prediction of the positive cases.
• True Negative (TN): represents the correct prediction of the negative cases.
• False Positive (FP): represents the negative cases that are predicted incorrectly as positive ones.
• False Negative (FN): represents the positive cases that are predicted incorrectly as negative ones.

The accuracy, recall, precision and F1 score evaluation metrics are computed as follows:

$$\mathrm{Accuracy}={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}$$

$$\mathrm{Recall}={\displaystyle \frac{TP}{TP+FN}}$$

$$\mathrm{Precision}={\displaystyle \frac{TP}{TP+FP}}$$

$$\mathrm{F}1\mathrm{score}=2\times {\displaystyle \frac{Precision\times Recall}{Precision+Recall}}$$

Different code-related tasks utilize evaluation metrics based on various scenarios. The accuracy metric is generally recommended when the data classification labels are balanced across classes. The precision metric is best suited for evaluating tasks where minimizing false positives is critical, such as in bug detection since frequently flagging bugs can be costly in such cases. In contrast, the recall metric is essential in tasks where minimizing false negatives is important, such as in assessing overall security coverage, where it is crucial to ensure that all security vulnerabilities are detected. In general, the F1 score and AUC metrics are considered effective in situations where data imbalance is a concern [231].

6.2.4. Security-Specific and Threat-Coverage Metrics

The evaluation metrics in this category are commonly used to assess neural models involved in program security analysis and threat prevention tasks. While a variety of metrics can be applied to such tasks, these particular metrics are distinguished by their specific areas of focus, which are often evident from their name and their computation terminologies. Threat Coverage (also known as vulnerability class coverage) serves as a primary example of this group of metrics. It measures the number of known categories of threats or vulnerabilities that a neural model can address or exploit during bug-hunting tasks. The core idea behind this evaluation metric is to quantify the diversity of threats a particular neural model can handle. As recently noted by Google DeepMind, analyzing security threat chains should be a fundamental building block of cyber-security systems, particularly in response to attacks initiated by AI-based incidents [232].

The Vulnerability Detection Score is a pairwise evaluation metric in this category [233]. It assesses a model’s ability to distinguish between vulnerable and non-vulnerable code, with a focus on minimizing the FN rate while keeping the FP rate below a particular fixed threshold. The Vulnerability Occurrence Rate is another example of a security analysis metric. It represents the fraction of code snippet samples in a dataset that contain at least one vulnerability. The study in [234] also explores how other metrics correlate with the vulnerability occurrence rate. Another evaluation metric in this category is the Attack/Exploit Success Rate, defined as the number of successful attacks divided by the total number of attack attempts [235]. Additionally, there are other criteria for evaluating the performance of neural models, such as plausibility, fixed bugs count, and correctness of fixed bugs. Most of these require manual intervention to make a final judgment.

6.2.5. Execution-Based and Functional Correctness Metrics

Execution-based evaluation metrics are among the most commendable and ideal metrics for programming-centric tasks involved in the generation of code snippets as model outputs. The top-k metrics, such as Pass@k, which are popular program-execution-based evaluation methods, assess a model’s performance by determining whether any of the k generated programs pass the given tests. In essence, Pass@k checks whether at least one of the top k model-generated programs passes all test cases, considering the problem to be solved if any of the k model-generated candidate code snippets is correct.

Similarly, the research work in [236] introduces three metrics—Success@k, Build@k, and Average Pass Rate—for evaluating code translation performance across entire repositories. Given M repositories in repository-level code translation benchmarks, we discuss these three metrics below along with their mathematical formulations.

Success@k (Test Pass Rate across Multiple Trials) measures the proportion of repositories that successfully pass all test cases in at least one of the selected k experimental rounds. Given a set $T_j$ of repositories that pass all test cases in the j-th round, the cardinality $|T_j|$ indicates the number of repositories that succeeded in that round. The Success@k metric is computed as shown below, where $\left({\displaystyle \genfrac{}{}{0pt}{}{m}{k}}\right)$ denotes the binomial coefficient, representing the number of ways to choose k rounds out of m total experimental rounds, and the union captures all repositories that pass in at least one of the selected trials:

$$\mathrm{Success}@k={\displaystyle \frac{1}{\left(\genfrac{}{}{0pt}{}{m}{k}\right)\times M}}\sum _{1\le j_1<j_2<\dots <j_k\le m}\left|T_{j_1}\cup T_{j_2}\cup \dots \cup T_{j_k}\right|$$

Build@k (Build Success Rate across Multiple Trials), similar to Success@k, indicates the proportion of repositories that successfully compile in at least one of the k selected rounds. Let $B_j$ denote the set of repositories that were successfully built in the j-th trial, then

$$\mathrm{Build}@k={\displaystyle \frac{1}{\left(\genfrac{}{}{0pt}{}{m}{k}\right)\times M}}\sum _{1\le j_1<j_2<\dots <j_k\le m}\left|B_{j_1}\cup B_{j_2}\cup \dots \cup B_{j_k}\right|$$

Average Pass Rate, also known as the average test case pass rate, measures the mean proportion of test cases passed across all repositories and all experimental rounds. Assuming that $M_{pt}^{(r,s)}$ denotes the number of passed test cases for the r-th repository in the s-th round, and $M_{at}^r$ denotes the total number of test cases for the r-th repository, the Average Pass Rate is computed as follows:

$$\mathrm{Average}\mathrm{Pass}\mathrm{Rate}={\displaystyle \frac{1}{m}}\sum _{s=1}^m\left({\displaystyle \frac{1}{M}}\sum _{r=1}^M{\displaystyle \frac{M_{pt}^{(r,s)}}{M_{at}^r}}\right)$$

Another notable example is the Computational Accuracy (CA) metric, introduced in 2020 by Meta AI and adapted by other researchers [79,237], which is defined as follows:

$$\mathrm{CA}={\displaystyle \frac{1}{N}}\sum _{i=1}^N\mathrm{CA}(y_i,{\widehat{y}}_i)$$

$$\mathrm{CA}(y_i,{\widehat{y}}_i)=\left\{\begin{array}{cc}1\hfill & \mathrm{if}{\mathrm{Exec}}_i\left(y_i\right)={\mathrm{Exec}}_i\left({\widehat{y}}_i\right)\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

where N denotes the total number of samples, while ${\mathrm{Exec}}_i\left(y_i\right)$ and ${\mathrm{Exec}}_i\left({\widehat{y}}_i\right)$ represent the execution results of the ground truth sample $y_i$ and the model-predicted sample ${\widehat{y}}_i$, respectively. Given k model-generated candidates, CA@k then evaluates if at least one of them produces the correct output for all test inputs.

In addition, the Test Case Average (TCA@k) [238] quantifies the average number of test cases passed by k generated code fixes per problem. DSR@k proposed in [239] and adapted in [240] defines the proportion of samples whose output is correct within k debugging steps. Repair@k [241] measures the percentage of tasks successfully fixed after k rounds of feedback-driven code repair. Related metrics like DSI and Compilation@k or Repairable Rate (RR) [240] also contribute to the evaluation of a model’s iterative fix and recovery abilities. Another related metric, though more common in retrieval-based tasks, is Recall@k [242] which measures the fraction of relevant or ground-truth-equivalent code snippets retrieved within the top-k candidates of the returned results.

The release of HumanEval, a benchmark dataset from the experimental research in [243], marks a shift toward execution-based evaluation. This approach assesses generated code by testing its functionality, addressing the limitations of match based, embedding based, and other metrics that are involved in evaluating model-generated code. Ongoing research continues to refine execution-based metrics, aiming for more reliable evaluation of AI-generated code [238].

While execution-based metrics provide a fine-grained assessment of code correctness, their practical implementation poses significant challenges. These metrics rely heavily on high-quality test cases that are compatible with each generated code sample. For example, the CA metric, originally introduced for evaluating code translation, may encounter limitations in more complex scenarios. In multilingual settings like the research work explored in [239], which involves translation between diverse PLs, applying CA becomes less feasible. The core issue lies in the difficulty of establishing a unified execution framework capable of supporting a broad range of languages and their associated libraries, runtime engines, and dependencies.

Therefore, the adoption and effectiveness of execution-based evaluation could be significantly improved if future research develops standardized and accessible unified frameworks. Such frameworks should provide consistent runtime environments across multiple PLs to facilitate reproducibility and enable more rigorous evaluation in programming tasks.

7. Neural Methods vs. Traditional Approaches in Programming Tasks

This section explores the key advantages of neural networks for programming tasks and examines the challenges of neural-based approaches. It also provides a comparative analysis of the neural methods and other existing methods.

7.1. Rule-Based Approach

In general, rule-based methods rely primarily on predefined “if–then” rules and follow established procedures to guide the knowledge-discovery process to accomplish certain tasks [244,245,246]. They are powerful for defining structured rules, making them useful as supplementary components in statistical and neural models. However, these approaches suffer from maintainability issues. The programming ecosystem frequently evolves due to business strategy changes and user feedback, requiring continuous updates. Rule-based methods struggle to adapt to these evolving requirements, making them inflexible for long-term scalability.

7.2. Statistical Approach

Several SMT algorithms can serve as benchmarks for adapting NLP-based techniques to programming tasks. SMT approaches basically outperform rule-based methods by incorporating syntactic and semantic representations. For instance, Nguyen et al. [215] utilize phrase-based SMT for program translation in three phases: (1) treating source code as syntactic sequences between source and target languages, (2) integrating semantic information for API migrations, and (3) mapping sememes (semantic units) between PLs. Similarly, other studies have explored the use of SMT methods in generating pseudo-code using the source code of certain languages [53,196]. Such enhancements make SMT approaches superior to rule-based methods, which lack flexibility.

Despite their advantages, statistical models also face challenges. For example, the SLAMC model in [92] does not account for inheritance in object-oriented languages, leading to OOV issues. The mppSMT model in [215], which adopts a divide-and-conquer strategy for code migration, also struggles with dependency learning, API misalignment, and OOV problems. Furthermore, its applicability is primarily limited to Java and C#.

7.3. The Novelty of Neural Methods

Neural networks, inspired by the human brain, offer a novel and promising approach to programming tasks. These models excel in understanding the logic and structure of source code, which itself reflects human cognitive-based solutions. This synergy of analyzing human-created code with human-brain inspired neural net algorithms creates an exciting research intersection [247].

Neural methods leverage vast numbers of neural networks, especially deep neural networks (DNNs) and LLMs, allowing models with billions of parameters to enhance code understanding and generation tasks. While optimal solutions remain an ongoing research challenge, the use of large pre-trained models trained on both NL and PL datasets has significantly improved performance. However, despite their advantages, neural models face several challenges, particularly in the availability of well-structured parallel benchmark datasets.

High-quality training data is essential for the effectiveness of any neural network, but constructing such datasets is resource intensive. Hence, we can say that neural-based approaches provide significant improvements over rule-based and statistical methods, yet overcoming high-quality parallel data limitations and other challenges remains a key area for future research.

Table 8. Neural methods vs. their counterparts (rule-based and statistical methods).

Method	Strengths	Limitations
Rule-based Methods	Effective for creating precise rules for corpora with complex code patterns.	Hard to maintain and scale. Requires extensive manual effort.
Statistical Methods	Offer more diverse algorithmic implementations than rule-based methods.	Human intervention is needed for better accuracy. SMT methods do not learn from previous experiences.
Neural Methods	Learn patterns from past data, delivering fast and accurate results.	Performance drops with limited training data and requires significant computational resources.

Note: Eighteen studies were examined for this comparison.

summarizes the pros and cons of these three approaches.

8. Discussion and Future Work

In this section, we revisit the research questions that guided our study and highlight potential directions for future work.

8.1. Answers to the Research Questions

8.1.1. RQ1: How Do Neural Approaches Compare to Rule-Based and Statistical Methods in the Context of Programming Tasks?

Neural methods represent the most recent paradigm in applying machine learning to programming tasks, such as source code translation, by leveraging DNNs and large-scale pre-training. In contrast, rule-based approaches rely on manually crafted transformation rules (often operating over ASTs), and statistical methods adapt classical SMT techniques to code (treating programs as literal text sequences).

Neural methods mark a significant leap forward compared to rule-based and statistical approaches for programming tasks, by providing greater adaptability, improved semantic accuracy, and scalable performance on large datasets. However, they demand extensive computational resources, robust parallel or monolingual corpora, and face ongoing challenges in evaluation and interpretability. Rule-based and statistical techniques are still relevant in resource-constrained or safety-critical settings, but for most modern software engineering needs, where languages, libraries, and idiomatic styles rapidly evolve, neural approaches have emerged as the state of the art. As the field progresses, hybrid solutions that integrate symbolic constraints into neural pipelines, alongside richer benchmarks and evaluation frameworks, will likely alleviate the remaining gaps between quality of automated programming tasks and human expert performance.

8.1.2. RQ2: What Is the Current Landscape of Datasets and Benchmarks for Neural Methods in Programming Tasks, and What Are the Critical Gaps?

The landscape of datasets and benchmarks for the application of neural method to programming tasks has expanded dramatically over the past decade, evolving from small, narrowly scoped corpora to large-scale, multi-task corpora. Early efforts (around 2015–2017) focused primarily on mapping NL to code in a single PL, most notably Python, while more recent initiatives (2020–2025) have sought to cover a broader array of tasks (e.g., vulnerability detection, code translation, and code summarization) and multiple PLs (including Java, C/C++, Go, Rust, and other multilingual settings). Despite this progress, several critical gaps remain: most datasets still suffer from limited scope (both in terms of size and language diversity), inconsistent preprocessing and documentation, noisy annotations (particularly in NL annotations or particular task’s labels), and a lack of standardized evaluation pipelines.

The field of programming has benefited from a proliferation of datasets and benchmarks spanning several software engineering and programming tasks. Early released code corpora (e.g., Django, PCSD, and CoNaLa) laid the groundwork but often suffered from limited scale and scope. Subsequent benchmarks (e.g., TransCoder, CodeXGLUE, CodeNet, and XLCoST) addressed multi-language and multi-task needs but sometimes at the cost of noisy annotations or inconsistent data format across similar tasks. Most recently, specialized datasets such as [209,242,248,249] have pushed toward class-level generation and real-world repository-level code intelligence tasks along with execution-based evaluation practice. However, critical gaps, such as the lack of standardized preprocessing, detailed documentation, under-representation of many PLs and domains, insufficient benchmarks for multi-file settings, interactivity, and explainability in program repair, as well as real-world performance, still persist. Addressing these gaps will require community-wide efforts to create unified dataset schemas, enrich annotation quality, broaden coverage, and develop richer evaluation methodologies, ultimately enabling more robust, reproducible, and impactful research in the application of neural methods for programming tasks.

8.1.3. RQ3: Which Evaluation Metrics Best Capture Model Performance on Code, Both Syntactically and Functionally, and Where Do Standard NLP Metrics Fall Short?

Evaluating code generation and code understanding models demands metrics that go beyond surface-level text similarity. While traditional NLP metrics (e.g., BLEU, ROUGE, and METEOR) provide a convenient baseline, especially during early stages of model development, their reliance on n-gram overlap prevents them from fully capturing either the syntactic correctness or the functional (execution) behavior of generated code. Even enhanced variants like CodeBLEU and CrystalBLEU, which incorporate AST and data-flow matching, cannot guarantee the generated snippets’ compilation or runtime tests. As a result, they serve as convenient proxies during early model inference trials but offer only limited insight into true code correctness.

The most reliable way to assess code model performance combines syntax checks (e.g., Compilation@k and Build@k) with execution-based metrics such as Pass@k and CA. Compilation@k ensures that generated programs meet language-grammar requirements, while Pass@k and CA verify that at least one of the top-k candidates successfully compiles and produces correct outputs against a comprehensive test suite. Metrics like Success@k, Average Pass Rate, and TCA@k further enrich this evaluation by measuring test pass rates over multiple trials or averaging partial successes, offering a fine-grained view of both build success and functional correctness of output results.

Standard NLP metrics fall short because they neither enforce compilability nor check whether code handles edge cases or real inputs correctly. In other words, they treat code as “flat text”, ignoring compilation requirements and runtime behavior. Embedding-based scores (e.g., CodeBERTScore and Cosine Similarity) improve semantic alignment but still treat code as text and cannot substitute for actual execution checks. Consequently, a robust evaluation framework for code requires prioritizing execution-based measures, backed by reliable test harnesses and multi-language runtime environments, while using text-based and embedding-based metrics only as supplementary signals during early development.

8.1.4. RQ4: What Roles Do LLMs (e.g., GPT-4, LLaMA, and Claude) Play in Programming Tasks?

LLMs originally developed for NLP have been rapidly repurposed to address a wide spectrum of programming tasks, fundamentally altering how code is implemented, reviewed, and maintained. By leveraging the vast corpora of open-source code and documentation, models such as GPT-4, LLaMA, and Claude have demonstrated an ability to generate syntactically correct and often semantically meaningful code snippets from high-level specifications. These models are now routinely employed in interactive, multi-turn dialogues where developers describe functionality or constraints in natural language, and the LLM iteratively refines code to meet functional objective of particular developer defined problem. LLMs generate test cases, and documentations for an entire program. In this capacity, LLMs serve not only as autocompletion engines but also as dynamic programming partners that adapt to evolving requirements and project contexts.

Beyond standalone code generation, LLMs have been integrated into specialized workflows that mirror traditional software-development life-cycle phases [250]. For instance, self-collaborative frameworks distribute subtasks among multiple LLM agents, each responsible for activities such as requirements analysis, architectural design, unit-test creation, and bug-fix proposals; intermediate artifacts generated by one agent become prompts for the next, producing an end-to-end automated pipeline. In parallel, security-oriented studies reveal that LLM outputs can introduce or exacerbate vulnerabilities, prompting the development of hybrid pipelines that combine LLM-driven patch generation with automated test harnesses or static analyzers to validate correctness and security [251]. Furthermore, code-specific LLMs trained and/or fine-tuned on large PL datasets underlie tools such as GitHub Copilot [252], Cursor, Windsurf, Replit and other programming-specialized AI-chatbots, and utilize LLM-powered AI agents enabling language-agnostic autocompletion, cross-language translation, and even real-time code-review suggestions within integrated development environments.

While LLM-powered development workflows significantly reduce manual implementations, automate boilerplate coding, accelerate test case synthesis, and facilitate rapid prototyping, they also introduce new challenges related to model hallucinations, and hence prompt a need for rigorous validation. Hallucinated library calls, misaligned data-flow assumptions, or omitted edge-case handling can produce code that compiles but behaves incorrectly or insecurely. The computational footprint of large models further constrains on-premises adoption. Despite these limitations, the integration of LLMs into code generation, completion, self-collaborative workflows, and security validation represents a transformative shift in programming, enabling developers to redirect their expertise toward higher-order design and architectural challenges while entrusting routine, syntax-oriented tasks to sophisticated generative models.

8.1.5. RQ5: What Are the Main Bottlenecks in Scaling and Deploying Neural-Based Programming Solutions to Real-World Codebases, and How Can They Be Addressed?

Neural-based programming tasks face several interrelated bottlenecks that hinder their scalability and deployment in real-world codebases. First, the scarcity of high-quality parallel corpora [253] across diverse PLs remains a critical obstacle. The majority of existing datasets are often narrowly focused, covering only a handful of modern PLs along with limited patterns such as NL query and code [254,255], or code–comment pairs, while legacy languages (e.g., COBOL and FORTRAN) and domain-specific languages (e.g., hardware description languages and smart contracts [256]) remain under-represented. Moreover, individual dataset pipelines lack standardization: tokenization strategies, comment handling heuristics, and metadata schemas are often undocumented, hindering reproducibility and fair model comparison. Even when aligned corpora exist (e.g., function-level translations between Java and C#), most corpora are limited to a widely studied couple of language pairs, leaving many language combinations with insufficient supervised data. Additionally, noisy or incomplete annotations, such as shallow doc-strings or heuristically derived vulnerability labels, further degrade model performance, as downstream fine-tuning depends on precise semantic alignments.

Second, the computational and evaluation challenges of neural approaches pose significant barriers to real-world adoption. Transformer-based models trained on massive code corpora require extensive GPU/TPU resources over prolonged periods, imposing both financial and environmental costs that are prohibitive for many academic and industrial settings. Even model inference phase may require optimized set of infrastructure to deliver low-latency, high-throughput code generation, or program understanding services. From an evaluation standpoint, conventional NLP metrics (e.g., BLEU, METEOR, and ROUGE) fail to capture code-specific syntactic and semantic correctness: two semantically equivalent implementations may receive drastically different n-gram scores, while superficially similar, functionally incorrect code can be rewarded [219]. Although specialized metrics such as CodeBLEU [96] have integrated AST-based comparisons, they do not reflect the runtime behaviors of evaluated program. Dynamic evaluation measures such as Pass@k, DSK@k, and CA incorporate test-based validation [79,257]. However, the availability of comprehensive and end-to-end evaluation pipelines remain scarce—particularly for multi-file projects, which demand integration tests, style readability, and each programming language’s coding idiom assessments. Additionally, the black-box nature of neural models presents significant challenges of interpretability and debugging practices. Understanding why a model omits a critical null check or mismatches API calls is far more complex than tracing crashes in traditional programs. Therefore, greater emphasis must be placed on human-in-the-loop evaluation strategies [258,259].

To overcome these bottlenecks, a multifaceted strategy is needed. First, primary efforts should focus on building standardized, end-to-end dataset pipelines that span both modern and legacy PLs, provide rich metadata (e.g., ASTs, DFGs, doc-strings, test cases, and error logs), and automate reproducible preprocessing steps (e.g., tokenization, de-duplication, and splitting). Semi-supervised and synthetic data generation techniques, leveraging unsupervised back-translation, template-based augmentation, and cross-lingual transfer, can help mitigate parallel data scarcity. Second, efficiency-focused model architectures (e.g., distillation, quantization, and efficient-tuning) combined with mixed-precision training can reduce the compute footprint, thereby democratizing access to large-parameter models for resource-constrained researchers. Third, robust evaluation frameworks should integrate both static (e.g., Compilation@k) and dynamic (e.g., Pass@k and CA) metrics, alongside novel benchmarks for multi-file, long-form code generation, idiomatic constraint, and security vulnerability severity [234]. Finally, enhancing model interpretability through attention visualization, rationale generation, and the integration of symbolically grounded validators or test-based debugging loops will foster greater trust and facilitate the deployment of neural-based solutions in production codebases.

8.1.6. RQ6: How Have Neural Methods for Programming Evolved, and Which Model- and System-Level Advances Have Driven This Progression?

The utilization of neural methods in programming tasks has been progressing rapidly. Examining the trends of utilizing automatic tools in programming and software development methodologies, we observe a transition from rule-based to statistical methods, and ultimately to neural-based approaches. This shift has significantly improved quality, performance, adaptability, and novelty. Neural-based methods, in particular, have experienced rapid growth, with many models achieving SOTA performance. As shown in Figure 6, in the earliest stage (1943–late 1980s), neural methods in programming were largely theoretical: the perceptron, basic neuron models, and backpropagation laid the mathematical foundation. The enhancement of that period was purely conceptual, advancing the understanding of learning algorithms rather than delivering tangible gains in code analysis or generation.

During the shallow neural network era (1990s–mid 2000), multilayer perceptrons and early convolutional neural networks began to show incremental enhancements over statistical baselines in tasks like code classification, effort estimation, and defect prediction. From mid 2000 until around 2016, deep learning models, such as Word2Vec, RNNs, LSTM, GRU, and other Seq2Seq models augmented with attention, marked a better improvement for the advancement of neural networks. This gave rise to experimental tools for comment generation, code completion suggestion, code summarization, and automated patch recommendation. Although these systems captured syntactic patterns, they lacked semantic awareness: variable scopes, control flows, and data dependencies across larger contexts remained difficult to represent, limiting their reliability for complex codebases.

Between 2017 and 2021, two parallel streams of innovation converged: graph-based models that embedded ASTs and CFGs into graph neural networks (e.g., GGNN and RGCN), and the Transformer architecture which revolutionized the sequence modeling, enabling parallel processing of long sequences with long-range dependencies, hence marking the significant shift to pre-training various neural models on massive code corpora (e.g., CodeBERT and CodeT5). The former produced richer, semantically grounded embeddings for tasks such as bug localization and code summarization; the latter endowed foundation models with deep contextual understanding and transfer learning capabilities. Systems integrating these models began to automate several tasks such as code search, code translation, and program repair with far greater precision and generality than previous systems.

Since around 2022, large-scale LLMs and Code LLMs (e.g., Codex, CodeGen, CodeLLaMA, and CodeT5+) have advanced the use of neural methods into interactive AI-pair programming and AI-powered software engineering workflows. Embedded in the IDE code-assistants, such as GitHub Copilot, Visual Studio IntelliCode, and CodeWhisperer, these models provide real-time, context-aware code completions, automated reviews, and patch suggestions. Crucially, modern neural-based systems often validate model outputs by running tests or static analyzers before presenting results, ensuring higher correctness. Emerging multi-agent systems further point toward the autonomous orchestration of programming tasks—from requirement parsing to development of functionally correct program codes—transforming neural models from simple code assistants into intelligent co-developers. Hence, the advancement of neural methods in programming and SE represents a rapidly evolving revolution, emerging in recent years and continuing to reshape efficiency and effectiveness across a wide range of tasks, models, and the entire programming ecosystem.

8.2. Future Work and Open Issues

Although neural methods, particularly recent variants such as LLMs, have significantly advanced day-to-day programming tasks, several research challenges remain. We categorize these challenges into five thematic areas, providing concise discussions while preserving the full scope of open research issues investigated during our review.

8.2.1. Evaluation Metric Issues

Code Semantics and Runtime-Behavior-Aware Metrics

Existing NLP metrics (e.g., BLEU and ROUGE) often fail to capture functional equivalence. Syntactic measures such as CodeBLEU, embedding-based metrics, and AST-based distances serve as preliminary filters but require supplementation with dynamic, execution-driven evaluation. Future work should implement end-to-end pipelines that first ensure Compilation@k and then validate runtime behavior via extensive test suites with Pass@k, CA, or DSR@k.

Multi-Objective Scoring

Code quality extends beyond correctness to other factors such as readability, maintainability, robustness, and performance. However, no standard metrics exist for code style, maintainability, or security severity. Benchmarks should integrate the following:

• Readability/Lint Scores: Combine linters (e.g., PEP8 and Checkstyle) and complexity analyzers into aggregate scores.
• Security-severity Evaluations: Quantify vulnerability severity (e.g., exploitability and levels of severity) rather than binary detection.
• Functional Performance Tests: Include runtime and memory benchmarks, especially for large or multi-file projects, to measure efficiency trade-offs.

Interactive and Explainable Evaluation

As interactive code synthesis grows, benchmarks should simulate iterative feedback loops by tracking metrics such as the number of iterations to achieve 100% test pass or the time to convergence. Additionally, when models produce natural-language explanations, new evaluation solutions are required to score logical consistency alongside functional correctness, potentially incorporating human-in-the-loop judgments. Moreover, the rise of LLM-powered AI agents underscores the urgent need for standardized, interactive, real-time evaluation frameworks, an area that remains largely unexplored in the AI research community.

8.2.2. Data Pipelines and Code Representation

Unified Dataset Schema and Code Representation

There is no consensus on a unified representation for code samples (e.g., raw text, raw code, AST, CFG, and IR) and their associated metadata (e.g., NL descriptions, test suites, and error logs). Future datasets should adopt a fine-grained schema for source code, specifying the following:

• Tokenization conventions (e.g., tokenization and comment stripping).
• Possible combinations of code representation formats (e.g., ASTs, CFGs, and DFGs).
• Normalization of NL texts aligned with code (e.g., wording, structure, and phrasing).
• Test-suite inclusion [260], covering unit tests and integration tests.

Reproducible, End-to-End Pipelines

Many published corpora omit detailed preprocessing steps, hindering reproducibility. Future efforts should document all transformations (e.g., code normalization, dataset splits, data-format, filtering and utilized tools) to allow an exact replication of baselines.

Expanding Language and Domain Coverage

Most corpora focus on Python, Java, JavaScript, C/C++, C#, and Go. Under-represented languages (e.g., Rust, Kotlin, Swift, and CUDA) lack large, high-quality parallel datasets. Future research should perform the following:

• Mine and Curate Cross-Lingual/Domain Corpora: Use automatic alignment (e.g., class-level and project-level) to build parallel datasets for less-common pairs (e.g., Solidity−Java).
• Incorporate Non-English NL Data: Collect doc-strings and comments in Arabic, Japanese, Korean, etc., to enable truly multilingual AI-powered programming agents (e.g., Korean−Java).
• Gather Domain-Specific Corpora: Develop benchmarks for Infrastructure-as-Code (e.g., Terraform), hardware design (e.g., Verilog), embedded systems, and financial smart contracts (e.g., Solidity), reflecting their unique syntax and semantics.

8.2.3. Hybrid Neural Methods with Symbolic Architectures

Neuro-Symbolic Integration for Semantic Guarantees

Purely neural method generators risk hallucinations, such as improper handling of errors during the reasoning process. Hybrid pipelines can interleave:

• Neuro-symbolic Validators: Post-generation static analysis (e.g., linters, model checkers, formal program verifiers) to flag or correct errors.
• Runtime Test Feedback: Compile and run generated code immediately. On failure, invoke a repair module or re-prompt the model.

Symbolic Augmentation for Certain Tasks

For vulnerability detection, integrating taint analysis, symbolic execution, or fuzzing can lower false positives and improve generalization. Program repair systems benefit from neural-based patch generation combined with symbolic equivalence checking.

8.2.4. Resource Efficiency and Deployment Constraints

Model Compression and Distillation

Larger LLMs achieve state-of-the-art results but are impractical for on-device or resource-constrained settings. Future work should explore the following:

• Knowledge Distillation: Train compact student models to approximate the behavior of larger teacher models.
• Parameter-Efficient Fine-Tuning (PEFT): Apply low-rank updates and quantization to adapt backbone weights, reducing storage and compute overhead [199].
• Edge-Optimized Libraries: Develop inference libraries for mobile or embedded hardware for balancing latency and accuracy when running neural models on edge devices.

8.2.5. Task-Specific Directions

Code Translation and Decompilation

Translation and decompilation should preserve the semantic meaning of code segments in end-to-end migration scenarios across source and target languages. Beyond surface metrics (e.g., BLEU and CodeBLEU), evaluation must incorporate semantic and runtime behavior comparisons, type-checker coverage, and execution outcomes. Architecturally, we have the following:

• Use specialized encoders that align parallel corpora at the function, class, or repository level.
• Employ decoders that enforce the preservation of target-language idioms (e.g., type inference and library-call mapping).
• For neural decompilers, leverage models that can comprehend control-flow and data-flow structures extracted from binaries, supporting multiple architectures (e.g., x86, ARM, and RISC-V) and handling obfuscation (e.g., packing and control-flow flattening).

Large-Scale Generation, Synthesis, and Automated Program Repair

Current benchmarks focus on single functions, but real-world development often spans entire projects. Future datasets should supply the following:

• Complete project skeletons such as minimal web apps with front-end, backend, database scripts, Continuous Integration and Continuous Deployment (CI/CD) configurations, to validate multi-file systems.
• Large, stable program-repair datasets: Curate commit pairs annotated by intent (e.g., bug fix, refactoring, performance optimization, and security patch) and integrate generate→test→patch loops where proposed edits are applied and validated against test suites. Coupling LLM outputs with program verification tools and multi-round repair logic will enable end-to-end automated program repair frameworks that compile and pass correctness checks.

Code Search, Retrieval, and Clone Detection

Future code search systems should ensure that retrieved snippets compile and pass basic program tests by combining static analysis with neural-based techniques, rather than relying on syntax alone. Clone detection should advance beyond the same language heuristics: models should embed joint ASTs, data-flow, and other representations to detect cross-language clones (e.g., C++−Rust). Crucially, systems should differentiate syntactic clones (boilerplate) from intent-based similar programs (semantically equivalent algorithms implemented differently), using semantic-level features (e.g., control-flow and data-flow equivalences) to prioritize intent.

Comment Generation, Code Summarization, and Authorship Attribution

To improve code readability, comment-generation models should support multiple languages: build parallel corpora of code with comments in languages other than English and evaluate multilingual consistency. Code summarization should be semantics-aware, and metrics should verify that generated summaries capture control flow, side effects, and algorithmic complexity by comparing against gold-standard semantic annotations (e.g., CFGs annotated with NL intents). Code authorship and attribution tools should be able to handle code from mixed sources, including human-written code, IDE templates, and AI assistant-generated code. Techniques fusing stylistic features (e.g., code idioms, n-grams, and AST-based style features) with provenance metadata (e.g., commit logs and file metadata) can attribute authorship across PLs.

Code Completion and Privacy

AI-assisted code completion systems should mitigate security and privacy risks. Repeated prompt templates may produce near-identical code segments, propagating flaws. Future systems should introduce controlled stochasticity or template diversification while preserving syntactic and semantic correctness. Training or fine-tuning on proprietary and personal codebases requires confidentiality guarantees; differentially private training algorithms or federated learning frameworks—where model updates are aggregated without exposing raw code elements—are largely unexplored in the programming research domain and should be prioritized to prevent leakage of sensitive source code parts.

By addressing the above-mentioned open issues, such as richer evaluation methods, unified data pipelines, neuro-symbolic integration, high-quality multilingual corpora, and resource-efficient architectures, the research community can advance toward reliable, secure, and accessible AI-driven software development workflows.

8.3. Trends, Risks, and the Road Ahead

While neural-based methods have rapidly advanced the capability of automating various fields across the globe, including programming, the field may encounter subtle risks of trend-driven adoption without rigorous architectural validation and reproducible evaluation. A major challenge lies in the acceleration of model deployment cycles, where experimental prototypes are prematurely promoted to production systems without a robust assessment of long-term reliability, ethical safety, or software quality impacts. This issue is especially critical in programming and software engineering, where neural models directly influence codebases that underpin large-scale industrial and financial infrastructures.

From a statistical perspective, our bibliometric and temporal analyses (Figure 3 and Figure 4) reveal a clear exponential rise in Transformer-based neural models from 2020 onward, with decoder-only architectures (e.g., GPT, LLaMA, and CodeGemma) dominating recent publications. Meanwhile, encoder–decoder and encoder-only designs, which have historically been effective for certain critical tasks, such as semantic and code translation tasks, are showing a relative decline in frequency, signaling a paradigm shift that may inadvertently deprioritize tasks better suited to their structures. The data in our collected publications further indicates that over 62% cited research works published in the year 2024–2025 rely on decoder-only architectures, emphasizing this evolving imbalance.

Emerging research in AI agents and multi-agent systems, integrated with neural code models, illustrates another forward trajectory. However, such agents often lack a defined alignment with the Software Development Life Cycle (SDLC), leading to ad hoc experimentation rather than systematic engineering integration. The next decade is likely to witness “coding agents” capable of generating enterprise-grade applications through high-level natural language instructions. Yet, realizing this vision safely requires verifiable performance measurement frameworks and ethically aware validation procedures that ensure trust, reproducibility, and resilience in code-generation and AI-augmented system-development practices.

In summary, sustainable progress in neural methods for programming depends on bridging three gaps: (1) establishing standard evaluation pipelines; (2) balancing architectural diversity across model families; and (3) embedding AI-agent development within the established SDLC frameworks. Addressing these will transform speculative enthusiasm into scientifically grounded and socially responsible advancement.

9. Conclusions

In synthesizing the reviewed literature, three major conclusions emerge. First, Transformer-based neural architectures, particularly encoder–decoder models such as CodeT5, and decoder-only LLMs such as GPT, Gemini, CodeLLaMA, and CodeGemma represent the most promising and versatile paradigms for programming-related tasks. Their capacity to learn unified representations across natural and programming languages has enabled remarkable progress in code generation, code summarization, code translation, and other SE tasks. Second, despite these advances, several persistent challenges remain unresolved: the absence of semantic and functionality-aware evaluation of generated code, the limited integration of graph-based or structure-aware models with Transformers, and the scarcity of multilingual and multi-paradigm datasets that impede robust cross-language generalization. Third, as outlined in Section 8, future progress will extend on unifying architectural innovation with standard evaluation frameworks and ethically guided deployment. Bridging these dimensions by coupling large-scale models with interpretable and verifiable code representations marks a concrete path toward reliable, safe, and industrially adoptable neural methods for programming tasks.

This survey systematically characterizes the landscape of neural methods in programming and SE, categorizing tasks through an extensive analysis of over 250 scientific papers. It critically examines the objectives, contributions, and limitations of previous surveys while offering a comprehensive comparison of neural methods across natural language and programming language tasks. Furthermore, it contrasts neural approaches with rule-based and statistical paradigms, providing a clear historical and technical evolution of the field. The paper also offers an exhaustive synthetical analysis of datasets and evaluation metrics, which has organized the entire performance evaluation metrics into five taxonomies accompanied by their mathematical formulations and benchmark analyses.

The overall findings indicate that recent advances in LLMs and Code LLMs are fundamentally transforming programming practices. This transformation is amplified by the emergence of multi-agent and context-aware systems that embed LLMs into the broader SE workflow. While the survey encompasses a broad spectrum of representative studies, some recently published breakthroughs are excluded for scope balance, particularly those overlapping with general NLP research. Although no empirical experiments were conducted, the conceptual analyses presented here lay a foundation for evidence-based future studies. Empirical investigations into the limitations of evaluation metrics such as BLEU, CodeBLEU, EM, and ROUGE would further substantiate our claims regarding their insufficiency in capturing semantic correctness.

Persistent challenges remain across most neural methods applied to programming and SE. These include the lack of standard, multi-modal code representations, the limited availability of high-quality parallel datasets, and the non-standardized use of evaluation metrics. Many models continue to rely on surface-level metrics that fail to capture semantic and functional fidelity. Addressing these limitations through improved representation learning, unified evaluation frameworks, and robust multilingual datasets will be pivotal for advancing the reliability, interpretability, and real-world applicability of neural methods for programming.