Cascaded Neurosymbolic Code Generation for Niche DSLs: Preserving Chain-of-Thought in Grammar-Constrained Decoding

Abstract

Domain-Specific Languages (DSLs) are essential in software engineering for safely expressing complex domain logic. However, Large Language Models (LLMs) struggle to generate syntactically and semantically correct code for niche DSLs due to sparse representation in pre-training corpora. While Grammar-Constrained Decoding (GCD) resolves syntactic hallucinations by masking logits through a formal Context-Free Grammar (CFG), empirical evidence shows that strict GCD disrupts the autoregressive Chain-of-Thought (CoT) reasoning of modern models, frequently forcing them into irreversible semantic dead-ends. To overcome the friction between internal neural reasoning and external symbolic constraints, we propose a Dual-Phase Cascaded Neurosymbolic framework. In the first phase, the model is provided with dynamically injected grammar rules and is permitted to reason unconstrained, producing an optimistic code draft. If the draft fails native compiler checks, the system enters a second phase: it preserves the successful semantic reasoning from Phase 1 but re-generates the code under strict GCD enforcement. This cascaded architecture utilizes the formal FSM not as an adversarial constraint, but as a localized syntax repair engine guided by the model’s own prior reasoning. We construct a comprehensive benchmark of 100 MiniZinc constraint programming tasks and evaluate our approach using a $pass@k$ metric with a strict semantic LLM judge. Our findings demonstrate that this dual-phase “think-then-constrain” approach significantly outperforms zero-shot, pure few-shot, and pure GCD baselines, achieving highly reliable, training-free code generation for unseen DSLs.

1. Introduction

Large Language Models (LLMs) have precipitated a paradigm shift across various specialized technical domains, demonstrating unprecedented capabilities ranging from automated cyber threat analysis [1] to software code generation, translation, and debugging [2,3]. However, reliably evaluating and guaranteeing the semantic correctness of these models remains a critical challenge. For general-purpose programming languages such as Python, Java, and C++, these models benefit from massive representation in pre-training corpora, allowing them to internalize both syntactic rules and complex semantic patterns. However, modern software architecture frequently relies on Domain-Specific Languages (DSLs) to safely and concisely express specialized domain logic. From hardware description (e.g., Verilog) and constraint programming (e.g., MiniZinc [4]) to proprietary enterprise configuration languages, DSLs enforce strict structural rules that prevent domain-specific errors. Unfortunately, LLMs consistently struggle to generate reliable code for these niche DSLs. Due to data sparsity, models frequently hallucinate syntax, invent non-existent operators, or fail to respect the rigid typing rules required by domain compilers. Because continuously fine-tuning foundation models for every new or proprietary DSL is economically unviable and technically cumbersome [5], training-free (zero-shot and few-shot) generation strategies remain highly desirable.

To address syntactic hallucinations without fine-tuning, the neurosymbolic integration of Grammar-Constrained Decoding (GCD) has emerged as a leading technique [6]. By compiling a formal Context-Free Grammar (CFG) or Extended Backus-Naur Form (EBNF) into a Finite State Machine (FSM), GCD dynamically masks the LLM’s output logits at each generation step. This physically prevents the model from generating any sequence of tokens that violates the target language’s syntax. While GCD effectively guarantees a 100% syntactic pass rate, recent empirical observations reveal a severe limitation when applied to modern reasoning models: a degradation of semantic accuracy.

Advanced LLMs, particularly those optimized via Reinforcement Learning for reasoning (e.g., DeepSeek-R1, OpenAI o1), achieve high semantic accuracy by utilizing long-form Chain-of-Thought (CoT) generation [7,8]. They “think out loud”, planning variables, logic, and constraints before emitting final code. Standard GCD architectures force the LLM to begin generating strictly valid grammar tokens immediately, stripping the model of its unconstrained reasoning space. Consequently, the model generates code blindly and greedily. If it makes a poor semantic choice early in the generation (e.g., declaring a variable as a boolean instead of an integer), the strict FSM forces it to complete the syntactically legal sequence, irreversibly driving the output into a “semantic dead-end” that fails subsequent compiler verification.

To overcome the friction between internal neural reasoning and external symbolic constraints, this paper proposes a Dual-Phase Cascaded Neurosymbolic Framework. Instead of applying constraints indiscriminately, our approach isolates semantic planning from syntactic enforcement. In Phase 1, the model is provided with the target EBNF grammar via the prompt but is permitted to reason unconstrained (e.g., within a <think> block), producing an optimistic draft of the code. This draft is instantly evaluated using native DSL compiler checks. If the optimistic draft passes (the “Fast Path”), the generation successfully concludes in $\mathcal{O}\left(1\right)$ time. If the draft exhibits syntactic hallucinations, the system enters Phase 2: it preserves the successful, unconstrained semantic reasoning from Phase 1, appends it to the context window, and re-generates the code under strict GCD enforcement.

1.1. Contributions

To overcome the friction between internal neural reasoning and external symbolic constraints, this paper proposes a Dual-Phase Cascaded Neurosymbolic Framework. Unlike existing code generation paradigms, our architecture is defined by three distinct novelties:

1.

Think-Then-Constrain Decoding: Rather than applying an “all-or-nothing” Grammar-Constrained Decoding (GCD) mask from the first token, which physically prevents models from utilizing <think> reasoning blocks and causes semantic collapse, our approach temporarily suspends the finite-state machine (FSM) mask to allow unconstrained semantic planning, activating it strictly for the code generation block.

2.

Deterministic Syntax Repair: Standard automated program repair methods are probabilistic, asking the LLM to “guess” the syntax again upon failure, which traps small models in hallucination loops on niche DSLs. Instead, our method acts as a deterministic syntax repair engine: if an initial draft fails, we preserve the successful semantic blueprint (the <think> block) and use the FSM mask to forcefully align that exact intent into correct syntax.

3.

Compiler-Routed Optimistic Bypassing: Most compiler-guided frameworks use compilers as textual prompt feedback or delayed reinforcement learning rewards. In contrast, our architecture utilizes the native compiler as a structural router. It allows the system to attempt a fast, $\mathcal{O}\left(1\right)$ unconstrained generation path, triggering the computationally heavy FSM masking overhead only when the compiler detects a structural hallucination.

The explicit contributions of this paper can be summarized as follows:

• We identify and empirically demonstrate the “Semantic–Syntax Trade-off” in modern reasoning LLMs, showing how strict Grammar-Constrained Decoding disrupts Chain-of-Thought planning and induces semantic dead-ends in niche DSLs.
• We present a novel Dual-Phase Cascaded Framework that implements Optimistic Bypassing: By preserving unconstrained reasoning from a failed draft and using it to guide a strict GCD fallback, our architecture effectively utilizes the symbolic grammar as a localized syntax repair engine.
• We construct and openly release a comprehensive benchmark dataset comprising 100 natural language to MiniZinc constraint programming tasks of varying complexities.
• We conduct a rigorous empirical evaluation using a $pass@k$ metric, scored by a strict semantic LLM judge and native compiler. Our results demonstrate that the dual-phase “think-then-constrain” approach significantly outperforms zero-shot, pure few-shot, and pure GCD baselines for unseen DSL code generation.

1.2. Structure of the Paper

The remainder of this article is organized as follows. Section 2 reviews related literature on LLM code generation, reasoning models, and grammar-constrained decoding. Section 3 details the proposed Dual-Phase Cascaded architecture, including the prompt injection of formal EBNF structures and the optimistic bypassing mechanism. Section 4 describes the experimental setup, introduces the MiniZinc benchmark, and presents the comparative results against established baselines. Finally, Section 5 summarizes our findings and outlines directions for future research.

2. Previous Work

On ordinary code generation tasks, LLMs are generally good at producing code that is at least syntactically well formed, especially for popular languages and strong code-tuned models [2,3,9,10]. The wider view in the literature is that larger code models implicitly learn syntax rules well enough that outright syntax errors are becoming rarer, even in relatively small code-specialized models [11].

Most recent studies on code generation for domain-specific languages focus on comparing different LLM configurations (pre-trained vs. fine-tuned, zero-shot vs. few-shot), LLM-based approaches (RAG vs. self-debugging vs. reinforcement learning), and DSLs versus general-purpose languages. In refs. [12,13], the authors compared fine-tuned LLMs for generating Verilog code in the context of hardware design. They found that fine-tuning was able to increase the capability of the models to produce syntatically correct code. Fine-tuning vs. optimized RAG was investigated in [14] in the context of automation task representation, showing that both approaches are prone to frequently generating syntactically erroneous code. The authors of [15] apply a small language model (SLM) fine-tuned using reinforcement learning to fix syntactic errors in LLM-generated code, and evaluate their approach for the Ansible, Bash and SQL languages. Their approach was shown to achieve superior performance than other repair-based approaches. A general framework for assessing the effectiveness of LLMs for DSL code generation is proposed in [16]. The authors find that, in general, the performance of LLMs for generating code in general-purpose programming languages like Python is better than for constraint programming (OCL, Alloy). They propose improvements in the code generation process like code repair or multiple attempts that can improve the quality of the generated code.

The authors of [17] argue that existing pretrained models still lack true understanding of code syntax, failing to match simple baselines focused on keywords and offsets. Compared with traditional generators, the summary is that LLMs now frequently achieve high syntactic correctness on common code, but they still do not offer the same consistency or guarantees, and syntax failures remain common enough to matter in practice [5]. The authors of [18] provide an empirical evaluation of error categories in structured outputs by state-of-the-art LLMs, observing that structural and syntactic errors remain a frequent challenge. The categorization of errors provided is also used in this study to guide the discussion in Section 4.4.7. The clearest pattern in the DSL and low-resource literature is that syntactic correctness gets worse as the target language moves away from mainstream training data. Very-low-resource programming languages are a direct example: models were found to struggle to compose syntactically valid programs in languages not represented in pretraining, and for the UCLID5 formal verification language, a special method called SPEAC generated syntactically correct programs more often than retrieval and fine-tuning baselines [19]. Hardware and other technical DSL-like code targets show the same tradeoff. For RTL and Verilog, prior work suggests strong LLMs can often handle the surface syntax because HDL looks somewhat like ordinary programming languages, but this does not translate into reliable overall generation without domain-specific processing and fine-tuning [13,20,21].

Recent work has focused on grammar constrained decoding (GDC) as one specific way of enforcing syntactic constraints at the LLM decoding stage [6,22]. In it essence, GDC approaches mask token completions during generations which do not correspond to the provided grammar, usually in EBNF form. This can be done, for instance, by formulating the completions allowed by the grammar as a finite state machine (FSM) [6]. However, GDC can distort the LLM’s generation distribution, biasing the decoding process towards outputs which are grammatically correct but with a likelihood that is not proportional to the ones that would have been selected by the LLM, resulting in poor semantic quality. To overcome this Grammar-Aligned Decoding (DAG) problem, the authors of [23] propose ASAp, a decoding algorithm that uses prior sample outputs to approximate the distribution of the LLM outputs conditioned on the grammar constraint.

In general, producing syntactically correct code represents only a pre-condition for semantic correctness. Some papers note that deep learning and LLM systems usually can produce syntactically correct code more often than they can produce code which is fully aligned with the task description [24,25]. This aspect remains underexplored in the current literature. One of the few works in this regard is [26], where the authors focus on enforcing semantic constraints by integrating token-level Monte Carlo Tree Search (MCTS) with Answer Set Grammars (ASGs). They show that their approach allows small pretrained LLMs to outperform starte-of-the-art reasoning models and guarantee semantic validity.

In the context of this existing literature, our proposed Dual-Phase Cascaded architecture addresses a critical intersection of these open challenges: While prior works rely on computationally expensive fine-tuning or token-level search to enforce syntax and semantics [13,26], we aim to provide a lightweight, training-free alternative. Specifically, our approach directly addresses the semantic degradation induced by standard GCD [23]. By isolating unconstrained semantic planning (Chain-of-Thought) from syntactic enforcement (the FSM logits mask), our framework prevents the grammar constraint from distorting the LLM’s natural generation distribution during the critical reasoning phase. Furthermore, by introducing an optimistic fast-path verified by native compilers, our method guarantees syntactic validity while actively prioritizing semantic alignment, bridging the gap between rigid formal constraints and the fluid reasoning capabilities of modern foundation models.

Standard repair-based methodologies, such as Self-Debugging [27] or Reflexion [28], treat code correction as a probabilistic, multi-turn dialogue. When generated code fails, the error is fed back into the LLM, prompting it to iteratively “guess” a correction. While effective for massive frontier models on mainstream languages, probabilistic repair frequently traps small-parameter models in endless hallucination loops when applied to the sparse syntax of niche DSLs [15]. Our architecture fundamentally redefines repair: instead of requesting a probabilistic retry, we retain the unconstrained semantic blueprint (the <think> block) from the failed draft and utilize the deterministic GCD mask to forcefully align that exact intent into correct syntax. Thus, the FSM acts as a deterministic repair engine rather than a probabilistic dialogue agent.

Compilers are traditionally integrated into neurosymbolic pipelines either as a reward signal for Reinforcement Learning or as textual feedback appended to the context window to guide subsequent generations [16]. Both approaches incur massive latency overheads due to multi-turn inference or heavy training requirements. In our cascaded architecture, the domain compiler serves an entirely different function: it acts as a structural router for Optimistic Bypassing. By evaluating an initial unconstrained draft, the compiler gates the heavy FSM constraint engine. It allows the system to capitalize on the LLM’s fast, native $\mathcal{O}\left(1\right)$ generation capabilities when the syntax is correct, and only incurs the latency penalty of constrained decoding when structural hallucinations are definitively detected.

3. Methodology

The core premise of our approach is to decouple semantic planning from syntactic enforcement. By allowing modern reasoning-optimized Large Language Models (LLMs) to construct a semantic plan without constraints, and subsequently enforcing strict syntactic adherence only when necessary, we construct a robust, training-free neurosymbolic architecture. This section describes the Dual-Phase Cascaded Framework, detailing the prompt formulation, the optimistic bypassing mechanism, and the Grammar-Constrained Decoding (GCD) fallback.

3.1. In-Context Symbolic Grounding

A fundamental challenge in zero-shot or few-shot code generation for Domain-Specific Languages (DSLs) is the model’s reliance on pre-trained biases, which often contradict the specific rules of the target DSL. To mitigate this, our framework employs In-Context Symbolic Grounding: Rather than relying on the LLM to infer the syntax from examples alone, we explicitly inject the raw Extended Backus-Naur Form (EBNF) grammar directly into the system prompt.

Alongside the EBNF grammar, the prompt is enriched with a semantic operator cheat sheet (Action Aliasing) and a curated set of representative few-shot examples. By forcing the LLM to observe the exact derivation rules prior to generation, we align the model’s internal representations with the strict constraints it will encounter. This ensures that when the LLM begins its Chain-of-Thought (CoT) reasoning, it actively traces its logic through the provided formal grammar, mapping natural language intents directly to valid non-terminal expansions. We emphasize that in this In-Context Symbolic Grounding module we intentionally treat the combination of EBNF rules, semantic aliases, and few-shot examples as a single, cohesive functional unit. For this reason, we do not provide ablations involving the individual components of this module in Section 4, since this would fall outside of the main focus of this study, which is to investigate the profound architectural friction between unconstrained Chain-of-Thought (CoT) reasoning and strict Grammar-Constrained Decoding (GCD).

3.2. Phase 1: Unconstrained Reasoning and Optimistic Bypassing

The first phase of the cascaded architecture capitalizes on the native strengths of reasoning LLMs. The model is instructed to encapsulate its step-by-step planning inside explicit <think>…</think> tags before outputting the actual code. During this phase, generation is completely unconstrained; standard autoregressive decoding is used. This allows the model to freely explore the semantic space, define variable relations, and plan optimization bounds without being artificially truncated by a logits mask.

Following the reasoning block, the model generates an initial code draft. To maximize computational efficiency, we introduce Optimistic Bypassing. The generated draft is immediately subjected to two deterministic gatekeepers:

1.

Syntactic Gate: A fast Context-Free Grammar parser (e.g., Lark) verifies that the generated string perfectly conforms to the EBNF structure.

2.

Semantic Gate: The draft is passed to the native DSL compiler in type-checking mode. Because the compiler evaluates the code globally, it instantly flags any type mismatches, uninitialized variables, or invalid operator overloads.

If the draft passes both gates, the system recognizes that the unconstrained LLM successfully handled both the semantics and the syntax. The generation halts, returning the valid code in $\mathcal{O}\left(1\right)$ autoregressive generation passes. This “Fast Path” prevents wasting computational resources on strict constraints when the model’s native capability is sufficient. Note that any erroneous, empty or otherwise malformed outputs are handled directly by the native DSL compiler, which simplifies the overall architecture and allows the algorithm to focus on the repair procedure in Phase 2 (see Section 3.3) instead of having to handle those by itself.

3.3. Phase 2: Grammar-Constrained Syntax Repair

If the optimistic draft fails either the syntactic or semantic gate, it indicates that while the model may have reasoned correctly, its translation into code suffered from structural hallucinations. In standard pipelines, such failures result in discarded outputs. In our cascaded framework, we discard the malformed code but preserve the model’s internal reasoning.

The successful <think> block generated in Phase 1 is captured and appended to the context window, acting as a highly detailed, task-specific semantic blueprint. The system then initiates Phase 2, generating the code block under strict Grammar-Constrained Decoding. A Finite State Machine (FSM) compiled from the DSL’s EBNF grammar masks the output logits at every decoding step. If the LLM assigns a high probability to an invalid token (e.g., hallucinating a Python-style and instead of the DSL’s /∖), the FSM forces the logit to $-\infty$, compelling the model to select the most probable valid token. This mechanism can handle deeply nested grammars (such as heavily recursive mathematical expressions or deeply scoped control-flow blocks) by tracking the valid vocabulary space at each token generation step without suffering from the exponential memory blowup or infinite left-recursion traps that plague naive top-down AST generators. Furthermore, because our Dual-Phase architecture allows the LLM to plan the semantic depth in an unconstrained <think> block during Phase 1, the model logically plans the nesting depth before the strict syntactic constraint is even applied.

Because the LLM’s context window already contains its own step-by-step reasoning, it does not work against the GCD mechanism. Instead, the constraint acts as a localized syntax repair engine, seamlessly mapping the LLM’s preserved semantic intent onto the mathematically guaranteed syntactic structure. Specifically, any ambiguities are inherently resolved by selecting the most semantically appropriate token from the union of all valid next tokens across all possible valid parse trees at the current generation step. Consequently, it is the neural policy that resolves the ambiguities without breaking the symbolic constraint mechanism provided by CGD.

3.4. Algorithmic Formalization

Algorithm 1 formalizes the execution flow of this cascaded architecture, which is visualized in Figure 1. The procedure begins by assembling a comprehensive, DSL-agnostic prompt that grounds the LLM using the formal EBNF grammar G, semantic aliases A, and few-shot examples E, alongside with the natural language intent I (Lines 1–2). In Phase 1, the model is queried without logit constraints, producing both its step-by-step Chain-of-Thought reasoning and an optimistic code draft (Lines 3–4). This draft is immediately evaluated by deterministic structural and semantic gatekeepers; if it passes both the CFG parser and the native compiler, the algorithm successfully takes the fast path and returns the code, bypassing further computation (Lines 5–12). However, if the draft fails due to syntactic or structural hallucinations, the algorithm initiates Phase 2, explicitly routing any malformed, empty or unparseable drafts. The valid reasoning extracted from Phase 1 is preserved and appended to the context window, while a pre-compiled and cached finite state machine strictly masks the LLM’s token generation (Lines 14–17). This forces the final output to mathematically conform to the DSL’s syntax, using the model’s own unconstrained planning to seamlessly guide the constrained repair.

Algorithm 1 Dual-Phase Cascaded Neurosymbolic Generation

Require: Natural language intent I, EBNF Grammar G, Semantic Aliases A, Few-Shot Examples E Ensure: Syntactically and semantically viable DSL code   1: $P_{base}\leftarrow \mathrm{BuildPrompt}(I,G,A,E)$            ▹ In-Context Symbolic Grounding   2: $P_{phase1}\leftarrow P_{base}+<\mathrm{think}>$                    ▹ Phase 1: Unconstrained CoT & Optimistic Draft   3: ${Out}_1\leftarrow \mathrm{LLM\_GenerateUnconstrained}(P_{phase1})$   4: $Reasoning,DraftCode\leftarrow \mathrm{ParseOutput}({Out}_1)$                            ▹ Optimistic Bypassing (Fast Path)   5: $IsValidDraft\leftarrow \mathbf{false}$   6: if $DraftCode\ne \mathbf{null\; then}$   7:      $IsSyntaxValid\leftarrow \mathrm{CFG}\_\mathrm{Parse}(DraftCode,G)$   8:      $IsSemanticallyValid\leftarrow \mathrm{NativeCompilerCheck}\left(DraftCode\right)$   9:      $IsValidDraft\leftarrow IsSyntaxValid \mathbf{and} IsSemanticallyValid$ 10: end if 11: if $IsValidDraft$ then 12:      return $DraftCode$                  ▹ Successfully bypassed Phase 2 13: else  ▹ Phase 2: Constrained Syntax Repair  ▹ Explicitly routes empty, unparseable, or invalid drafts to the FSM fallback 14:     $P_{phase2}\leftarrow P_{phase1}+Reasoning+</\mathtt{think}>\mathtt{dsl}$         ▹ Use CoT from Phase 1 15:     $FSM\leftarrow \mathrm{GetCachedFSM}\left(G\right)$ ▹ Fetch pre-compiled automaton (compiled once per grammar) 16:     ${Out}_2\leftarrow \mathrm{LLM}\_\mathrm{GenerateConstrained}(P_{phase2},FSM)$ 17:     $FinalCode\leftarrow \mathrm{ParseOutput}\left({Out}_2\right)$    ▹ Extract final code from Phase 2 output 18:     return $FinalCode$ 19: end if

4. Empirical Evaluation

To rigorously assess the effectiveness of the proposed Dual-Phase Cascaded framework, we designed an experimental pipeline comparing our approach against prevailing LLM-driven code generation strategies. The objective of this evaluation is to quantify the extent to which preserving Chain-of-Thought (CoT) reasoning mitigates the semantic dead-ends induced by strict Grammar-Constrained Decoding (GCD). All the code, benchmarks and utilities used for the empirical evaluation are publicly available (https://github.com/IMC-UAS-Krems/mcts-niche-dsl, accessed on 10 June 2026). The experiments were carried out on a GPU server with 256 GB RAM, two AMD EPYC SP3 8-core (3.7GHz) and two NVIDIA L40S GPUs.

4.1. Benchmark Dataset Construction

Mainstream code generation evaluation suites, such as HumanEval or MBPP, are heavily biased toward general-purpose programming languages and algorithmic tasks. They are ill-suited for measuring an LLM’s capability to generate structurally rigid, niche Domain-Specific Languages. Consequently, we constructed a novel benchmark dataset comprising 100 distinct natural language programming tasks mapped to the MiniZinc constraint programming language. These programming tasks were carefully crafted by the authors to reflect different types of CP models and task difficulties. The distribution of the tasks included in this dataset is illustrated in Figure 2. We categorize each example into one of five different categories:

(i)

Basic declarations and simple constraints: e.g., finding integers within bounds or strict equality.

(ii)

Complex arithmetic and aggregations: e.g., modulo operations, division, and compound mathematical limits.

(iii)

Advanced data structures (arrays and sets): e.g., declaring arrays of specific sizes, subset domains, and applying the sum() aggregator.

(iv)

Logical expressions and implications: e.g., boolean logic, either/or constraints, and strict implications (if/then).

(v)

Optimization: e.g., constrained minimization or maximization objectives.

Figure 2. Task distribution of the MiniZinc benchmark dataset by category (left) and difficulty (right).

Figure 2. Task distribution of the MiniZinc benchmark dataset by category (left) and difficulty (right).

Regarding the difficulty distribution, around two thirds of the examples were categorized as medium difficulty, the rest being split equally into easy (few simple variables and/or operations) and hard (several complex operations/variables involved) examples.

We choose MiniZinc for two reasons: first, it is a niche DSL which, as such, is underrepresented in the training corpus of smaller reasoning LLMs in comparison to general-purpose programming languages like Python or C++. This observation was supported empirically by preliminary zero-shot tests using the Qwen2.5-Coder-1.5B-Instruct model (see Table 1 for the full results). Second, current state-of-the-art flagship LLMs like Qwen3.5 already possess a good understanding of MiniZinc code, which allows us to use this model as a semantic LLM judge in the evaluation process. By using a smaller reasoning model that is not proficient in MiniZinc, we make sure that the code generation process is guided by the provided grammar instead of pretrained knowledge.

The benchmark tasks exhibit a progressive gradient of complexity, encompassing simple variable declarations, set definitions, array aggregations, logical implications, and complex optimization objectives (minimizing or maximizing specific variables). Each task in the dataset consists of a natural language intent and a verified “golden” MiniZinc solution, providing a ground-truth reference for automated evaluation.

4.2. Experimental Setup and Baselines

All generation experiments were conducted using Qwen2.5-Coder-1.5B-Instruct, an open-weights reasoning model, parameterized to simulate a resource-constrained, local-first software engineering environment. Because our evaluation utilizes a 1.5B parameter model, the baseline VRAM requirement is exceptionally lightweight (around 3 GB in bfloat16 precision). The primary memory overhead introduced by our method stems from compiling the Extended Backus-Naur Form (EBNF) grammar into a Finite State Machine (FSM) via the outlines library. However, FSM compilation is performed once and cached, ensuring that the memory footprint remains stable and bounded across subsequent generation queries without risking Out-Of-Memory (OOM) errors during parallel evaluations. The following hyperparameters were used: Temperature T = 0.6, sampling strategy do_sample = True, default values for top_p = 1.0 and top_k = 50 to avoid artificially truncating the probability distribution during FSM masking, and maximum tokens set to 3000 during Phase 1 to generously accommodate unconstrained Chain-of-Thought reasoning and 300 for Phase 2 (pure code generation steps). The generation was intentionally left unseeded to allow the multinomial sampler to introduce the necessary natural stochastic variance required for a valid $pass@k$ metric evaluation.

We explicitly note that the grammar used in our experiments is a task-relevant subset of MiniZinc, not the full language specification. This choice is deliberate, for two reasons. First, exhaustive grammar injection is counterproductive for a 1.5B-parameter model: a very large reference payload dilutes the model’s attention over the derivation rules that actually matter for a given task, which degrades the quality of the Chain-of-Thought produced in Phase 1. Second, and most importantly, the Outlines FSM backend used in Phase 2 restricts the expressiveness of the compiled CFG to strictly regular or right-recursive structures to avoid memory exhaustion; the complete MiniZinc grammar would violate this restriction and could not be compiled into a tractable masking automaton in the first place. However, the subset was constructed to cover every construct exercised by the benchmark: variable and parameter declarations, integer and set domains, array declarations and element access, aggregation constructs such as sum and forall, the MiniZinc logical operators and implications, and minimization and maximization objectives. The exact grammar used is publicly available in the GitHub companion repository, which was mentioned above.

To establish a comprehensive comparative analysis, we evaluated the proposed architecture against four distinct baseline configurations, all utilizing the exact same underlying LLM:

• Baseline 1: Zero-Shot Autoregressive

The LLM is prompted solely with the user intent and instructed to output raw MiniZinc code without any provided grammar rules, examples, or constraints. This establishes the absolute baseline capability of the foundation model’s pre-training.

• Baseline 2: One-Shot (Unconstrained)

The LLM is provided with the structural EBNF grammar, semantic operator aliases, and a single high-quality CoT example. The model is allowed to output its <think> reasoning block, but the final code generation is left unconstrained. This evaluates the efficacy of pure prompt engineering.

• Baseline 3: One-Shot (GCD Only)

The LLM is provided with the same grammar and example, but is strictly prohibited from outputting a <think> block. The generation is immediately subjected to the strict Outlines FSM logits processor. This baseline isolates the performance of standard, state-of-the-art GCD architectures that prioritize syntax over unconstrained planning.

• Baseline 4: One-Shot (CoT + Always GCD)

In this baseline, the LLM is allowed to output a <think> reasoning block, however there is no optimistic draft: Immediately after this phase the final code is generated subjected to GCD. This baseline effectively measures the effect of Optimistic Bypassing by ignoring any intermediate code generated and taking advantage of the CoT reasoning block for GCD.

• Proposed Method: Dual-Phase Cascaded GCD

As formalized in Section 3, the model generates an unconstrained CoT reasoning block and an optimistic draft. If the draft fails, the system executes a GCD fallback guided by the preserved reasoning. Critically, we never mention the name “MiniZinc” in the prompts used for our approach, using the name MaskedLanguage instead. This ensures that our method is using only the knowledge provided in the prompts instead of retrieving any previous pretrained knowledge, and also applies to all One-Shot baselines for a fair comparison. In constrast, the Zero-Shot baseline does make use of the name “MiniZinc” explicitly, since our goal in this case is to compare against zero-shot pretrained knowledge specific to the DSL at hand.

4.3. Evaluation Metrics: $pass@k$ and Automated Semantic Judging

Because LLM generation is inherently probabilistic, evaluating a single greedy output often misrepresents a model’s true capability. We evaluate all methods using the standard $pass@k$ metric with $k=1,3,5$, which measures the probability that at least one out of k generated candidate programs successfully solves the task. To induce the variance required for diverse sampling, we utilized multinomial sampling with a temperature of $T=0.6$ across all generation calls.

For a generated candidate to be classified as a “Pass”, it must successfully navigate three strict, sequential evaluation gates:

1.

Syntactic Gate: The generated string must be successfully parsed by the formal Lark CFG parser, verifying absolute structural compliance. This ensures syntactic validity: The code mathematically conforms to the raw derivation rules of the Context-Free Grammar (EBNF), regardless of whether the variables are declared or typed correctly.

2.

Semantic Compiler Gate: The code is passed to the native MiniZinc compiler using the –model-check-only flag). This ensures compiler validity (static semantics): The code is not only syntactically valid but also adheres to the formal domain rules of the language (e.g., proper scoping, no uninitialized variables, correct type matching between booleans and integers). It represents code that is legally executable by the system. This deterministic check instantly rejects uninitialized variables, out-of-bounds array accesses, and semantic type mismatches.

3.

Functional Intent Gate (LLM-as-a-Judge): Because code can compile perfectly but fail to solve the requested problem, we implement a strict semantic judge. This addresses dynamic semantics: While static semantic validity ensures the code means something legal, dynamic semantic validity ensures it means exactly what the user intended (e.g., maximizing a variable instead of minimizing it). For this purpose, we employ a larger, independent local model (Qwen3.5) via Ollama and prompted with a highly strict CoT evaluation rubric. The judge compares the generated code against the golden solution from the benchmark, analyzing variable mapping, constraint logic, and optimization directions. The judge returns a normalized score between $0.0$ and $1.0$; a candidate is only marked as successful if it achieves a score of $\ge 0.8$. This threshold was empirically selected to act as a strict, high-confidence filter. It safely absorbs the minor fractional noise inherent to the neural evaluation of stylistic differences, while remaining high enough to rigorously reject any code that violates the functional intent of the prompt. To ensure the LLM-as-a-judge operates deterministically, the judging model (Qwen3.5) is queried with the decoding temperature strictly set to $T=0.0$ (greedy decoding). Furthermore, to minimize the inherent variance of neural evaluation, the judge does not evaluate the generated code in isolation. Instead, it is provided with the verified golden solution from the benchmark and prompted with a strict rubric to act as a direct equivalence checker. This grounds the LLM’s evaluation and ensures the scores are highly stable and reproducible across runs.

We emphasize that Gate 2 (the MiniZinc compilation check) constitutes a ground-truth, deterministic semantic evaluation. In order to bridge the gap between semantic evaluation and the functional intent of the user prompt, Gate 3 (the LLM-as-a-Judge) supplements the deterministic compiler check with a highly constrained LLM-as-a-judge to verify its alignment against a golden solution. This hybrid approach provides a highly robust evaluation pipeline for the constraint programming domain, where models define problem spaces rather than sequential algorithms and mathematically equivalent constraints can be formulated in vastly different ways.

4.4. Results and Discussion

The experimental results for the MiniZinc code generation benchmark are summarized in Table 1. The evaluation rigorously assesses syntactic correctness, compilation viability, and semantic alignment through the $pass@1$, $pass@3$, and $pass@5$ metrics. The data clearly demonstrate the superiority of the proposed Dual-Phase architecture, while also revealing profound insights into the interaction between modern reasoning LLMs and strict symbolic constraints.

Table 1. Evaluation of code generation strategies on the MiniZinc benchmark. Results denote the $pass@k$ accuracy (%), defined as syntactic, semantic, and compiler-verified success. The $\Delta$ columns represent the relative improvement of the proposed Dual-Phase architecture over the respective baseline at each k.

Method	pass@1	$\mathbf{\Delta }$	pass@3	$\mathbf{\Delta }$	pass@5	$\mathbf{\Delta }$
Zero-Shot	11.0	+263.6%	21.0	+223.8%	29.0	+165.5%
One-Shot (No GCD)	12.0	+233.3%	15.0	+353.3%	16.0	+381.2%
One-Shot (GCD Only)	11.0	+263.6%	17.0	+300.0%	22.0	+250.0%
One-Shot (CoT + GCD)	36.0	+11.1%	64.0	+6.2%	71.0	+8.5%
Dual-Phase (Proposed)	40.0	-	68.0	-	77.0	-

Table 1. Evaluation of code generation strategies on the MiniZinc benchmark. Results denote the $pass@k$ accuracy (%), defined as syntactic, semantic, and compiler-verified success. The $\Delta$ columns represent the relative improvement of the proposed Dual-Phase architecture over the respective baseline at each k.

Method	pass@1	$\mathbf{\Delta }$	pass@3	$\mathbf{\Delta }$	pass@5	$\mathbf{\Delta }$
Zero-Shot	11.0	+263.6%	21.0	+223.8%	29.0	+165.5%
One-Shot (No GCD)	12.0	+233.3%	15.0	+353.3%	16.0	+381.2%
One-Shot (GCD Only)	11.0	+263.6%	17.0	+300.0%	22.0	+250.0%
One-Shot (CoT + GCD)	36.0	+11.1%	64.0	+6.2%	71.0	+8.5%
Dual-Phase (Proposed)	40.0	-	68.0	-	77.0	-

Note that the large relative improvements over the weak baselines primarily quantify how severely those baselines fail on the task, rather than adding independent evidence for the strength of the proposed method. Therefore, the $\Delta$ values reported against the best baseline One-Shot (CoT + Always GCD) are the most meaningful to interpret in the context of the proposed approach. We would also like to emphasize that the specific choice of benchmarks used in this study can be understood as an ablation study, since each baseline successfully isolates a specific aspect of the proposed approach: the effect of grammar injection (In-Context Grounding) by introducing the One-Shot (no GCD) baseline, the effect of CoT preservation (One-Shot GCD Only), the effect of constrained feedback (One-Shot CoT + GCD) and, finally, the effect of the validation introduced in the optimistic bypass.

4.4.1. The Semantic Collapse of Strict GCD

Perhaps the most striking finding from the benchmark is the catastrophic performance degradation of the One-Shot (GCD Only) baseline. Despite being provided with the exact same in-context example and grammar description as the unconstrained baseline, the GCD-only approach achieved a mere 11.0% $pass@1$ and plateaued at 22.0% for $pass@5$.

This starkly validates our core hypothesis regarding the “Semantic–Syntax Trade-off.” Because the GCD-only baseline physically prevents the model from outputting its native <think> reasoning block—forcing it to immediately output constrained EBNF tokens—the model is stripped of its ability to perform Chain-of-Thought (CoT) planning. Consequently, the FSM logits processor forces the model to make greedy syntactic choices without a global semantic plan. The resulting code is 100% syntactically perfect but semantically nonsensical, failing the compiler’s type-checks or the semantic judge’s intent verification. Similar results were obtained using the open weights reasoning model DeepSeek-R1-Distill-Qwen-1.5B within a subset of the benchmark, where the One-Shot (GCD Only) approach could only achieve 2.0% $pass@1$ accuracy. This demonstrates that for modern, reasoning-optimized models (e.g., DeepSeek-R1, Qwen), applying structural constraints at the expense of CoT reasoning is highly detrimental.

4.4.2. Unconstrained Reasoning vs. Zero-Shot Pre-Training

The Zero-Shot baseline exhibited similarly poor performance (11.0% $pass@1$, 29.0% $pass@5$), confirming that the 1.5B parameter model lacks the pre-trained density to natively map natural language to the niche MiniZinc syntax.

Similarly, the One-Shot (No GCD) approach did not yield a significant performance leap, achieving 11.0% at $pass@1$ and reaching 16.0% at $pass@5$. As seen in the reasoning traces, providing a single example and, most importantly, allowing the model to utilize its <think> reasoning block, allowed the LLM to successfully plan the semantic bounds, variables, and constraints. However, frequent minor failures hindered this baseline from achieving better $pass@k$ values. Such failures were predominantly caused by minor syntactic hallucinations (e.g., mixing Python-style operators with MiniZinc operators), where the unconstrained generation drifted slightly out of bounds.

Conversely, the One-Shot (CoT + GCD) baseline could achieve a significant improvement compared with the other One-Shot and the Zero-Shot baselines. This demonstrates that using the reasoning block and appending it to the prompt for strict GCD generation already mitigated the syntactic hallucinations that plagued the model in the unconstrained baselines.

4.4.3. The Efficacy of the Dual-Phase Architecture

The proposed Dual-Phase framework significantly outperformed all baselines across every metric. By isolating semantic planning (Phase 1) from syntactic enforcement (Phase 2), the system achieved a $pass@1$ accuracy of 40.0% and a maximum $pass@5$ accuracy of 77.0%.

The relative improvements ($\Delta$) highlight the exact value of the cascaded approach. Compared to the strongest baseline (One-Shot CoT + GCD), the Dual-Phase architecture achieved a remarkable +11.1% relative improvement in $pass@1$ accuracy. This indicates that for single-sample generation (where computational efficiency is paramount) the optimistic bypassing and subsequent GCD fallback effectively repaired many of the generations that would have otherwise failed due to trivial syntax errors. Because the fallback mechanism is explicitly guided by the preserved <think> reasoning from Phase 1, the FSM logits mask acts as a precise syntax repair engine rather than an adversarial constraint.

4.4.4. Scaling with Sampling Diversity ($pass@k$)

As the sampling budget increases from $k=1$ to $k=5$, the performance of the baselines and the Dual-Phase approach scales correspondingly. The Dual-Phase framework consistently defines the upper bound, maintaining an $+8.5\%$ relative improvement over the One-Shot (CoT + GCD) baseline at $pass@5$. The narrowing of the relative gap at higher k values is expected; with enough stochastic sampling temperature, the One-Shot models will eventually reach the correct syntax. However, the Dual-Phase architecture dramatically accelerates this convergence, proving that strict neurosymbolic grounding yields far higher reliability per computational cycle.

4.4.5. Computational Overhead and the Small-Model Paradox

Table 2. Computational overhead of the evaluated methodologies. The table reports the mean generation time in seconds and the mean total tokens generated per sample, alongside their respective standard deviations (Std).

Method	Mean Time (s)	Time Std (s)	Mean Tokens	Tokens Std
Zero-Shot	0.93	0.56	34.2	19.7
One-Shot (No GCD)	5.70	8.59	223.8	338.5
One-Shot (GCD Only)	2.44	0.26	30.4	7.9
One-Shot (CoT + GCD)	7.56	7.11	232.8	279.9
Dual-Phase (Proposed)	8.10	8.37	258.0	328.3

details the computational overhead—measured in mean generation time and total tokens spent per sample—across the evaluated methodologies. While the proposed Dual-Phase architecture achieves the highest semantic accuracy, we observe that its mean generation time ($8.10$ s) and token expenditure ($258.0$) are marginally higher than the ablation baseline One-Shot (CoT + GCD) ($7.56$ s, $232.8$ tokens).

This inversion of the expected efficiency gains from the Optimistic Bypass is an artifact of the specific hardware and model scale utilized in this evaluation. We define this as the Small-Model Overhead Paradox. Because the evaluated model is exceptionally lightweight, its token generation latency is minimal. Conversely, executing the intermediate Optimistic Bypass requires writing to the disk and spawning a native OS subprocess to run the MiniZinc compiler. When the draft code fails, the Dual-Phase architecture absorbs the latency of the failed subprocess check in addition to the Phase 2 generation time. Furthermore, the discarded draft code contributes to a marginally higher total token count.

Importantly, the high standard deviations indicate that the computational differences between these two methods are not statistically significant. However, acknowledging this dynamic is critical for production deployment. The Dual-Phase framework is fundamentally optimized for larger, state-of-the-art frontier models (e.g., 32B+ parameters or cloud-based APIs) where token generation latency and inference costs vastly outweigh the negligible $\mathcal{O}\left(1\right)$ overhead of a local compiler subprocess. In those environments, the ability to selectively bypass constrained generation tokens via the Fast Path might become a significant computational and economic advantage.

4.4.6. Internal Routing Analytics and the Value of Optimistic Bypassing

To better understand the internal dynamics of the Cascaded Neurosymbolic Architecture, Figure 3 and

Table 3. Internal routing analytics for the Dual-Phase Cascaded Architecture. The table details the probability that a sample required the Phase 2 GCD fallback (i.e., the optimistic draft failed compilation), and the conditional probability that Phase 2 successfully repaired the sequence to yield a passing model.

Routing Metric	Rate (%)
Phase 2 Trigger Rate ($P(\mathrm{Phase}2\mid \mathrm{Phase}1\mathrm{Fails})$)	81.6%
Conditional Success Rate ($P(\mathrm{Pass}\mid \mathrm{Phase}2\mathrm{Triggered})$)	36.3%

visualize and quantify the flow of generated samples through the system’s routing mechanisms. Across the 500 total generated samples, the data explicitly highlights the critical computational and semantic contribution of the Optimistic Fast-Path.

As illustrated in the Sankey diagram (Figure 3), 92 samples (18.4% of all generations) successfully bypassed the Phase 2 Grammar-Constrained Decoding (GCD) fallback entirely. These samples represent instances where the unconstrained LLM natively generated a syntactically perfect and semantically accurate draft during Phase 1. Remarkably, while the Fast-Path was only utilized in 18.4% of the total generations, it contributed to nearly half (92 out of 199, or 46.2%) of all ultimate semantic passes.

This disproportionate contribution strongly validates the premise of Optimistic Bypassing: when modern reasoning models are capable of natively resolving the semantic-syntax gap, forcing them through a rigid FSM masking engine is both unnecessary and computationally wasteful. By successfully identifying and fast-tracking these 92 samples in $\mathcal{O}\left(1\right)$ compilation steps, the framework secures high-quality reasoning while significantly reducing the mean token expenditure and wall-clock generation time across the benchmark.

Conversely, the Phase 2 GCD fallback was triggered for the remaining 408 samples, representing an 81.6% trigger rate (Table 2). These samples initially failed the native compiler check due to structural or syntactic hallucinations. Acting as a localized syntax repair engine, Phase 2 utilized the preserved Chain-of-Thought from Phase 1 to rescue these broken drafts, achieving a conditional success rate of 36.3%. This explicit bifurcation confirms the fundamental synergy of the dual-phase approach: the Optimistic Bypass aggressively maximizes computational efficiency for natively correct reasoning, while the GCD fallback provides a robust, highly effective safety net for structural hallucinations.

4.4.7. Categorization of Persistent Errors

To systematically guide future improvements, we align our failure analysis with the standardized error taxonomy recently proposed in the literature [18]. Because our evaluation pipeline operates as a sequential, three-stage gating system, our diagnostic logs naturally map to these three distinct error categories. This allows us to transparently dissect the persistent failures encountered by the Dual-Phase architecture:

• Pure Syntax Errors: These occur when the model hallucinates invalid tokens, misplaced punctuation, or undefined operators, resulting in a failure at our first evaluation gate (the CFG parser), while the unconstrained baselines suffer notably from these errors, the Phase 2 Grammar-Constrained Decoding (GCD) fallback in our proposed architecture largely eliminates them.
• Structural/Schema Errors: These denote code that is syntactically well-formed but violates the formal typing or scoping rules of the domain, failing our second gate (the native MiniZinc compiler check). In our qualitative analysis, these predominantly manifest as Deep Nesting Fatigue: When generating deeply nested Abstract Syntax Trees (e.g., complex recursive constraints or logical implications), the LLM’s attention mechanism occasionally degrades. The model loses track of variable scopes or strict type requirements, resulting in structurally illegal operations (e.g., assigning an array to a scalar value) that the FSM allows but the compiler rightfully rejects.
• Semantic Logic Errors: These represent candidate models that compile perfectly but fail to execute the user’s specific functional intent, thereby failing our third gate (the semantic LLM-as-a-judge). Our analysis identified two primary drivers for these failures:

  ‑

  Semantic Drift in Ambiguous Contexts: While the GCD engine tracks ambiguous grammars by allowing the union of valid tokens, the LLM itself occasionally struggles to resolve this ambiguity functionally. The model may select a syntactically valid but unintended derivation path (e.g., applying an arithmetic operator instead of a logical one), producing code that works but solves the wrong problem.

  ‑

  Implicit Type Coercion/Logical Hallucinations: The model occasionally generates constraints that exploit native compiler allowances. For instance, MiniZinc allows certain implicit type coercions (e.g., evaluating a boolean as an integer). If the LLM generates a constraint exploiting this, it bypasses the compiler’s safety check but represents a fundamental logical failure in translating the specific data types requested in the natural language prompt.

Figure 4 visualizes the distribution of failed samples across three sequential evaluation gates: Pure Syntax Errors (CFG parser rejection), Structural/Schema Errors (native compiler type/scope rejection), and Semantic Logic Errors (LLM-as-a-judge intent rejection). The visualization reveals a paradigm shift in how different architectures fail, providing empirical validation for our cascaded approach:

1.

The Eradication of Syntax Errors and Regex Artifacts:

As expected, the unconstrained baselines suffer overwhelmingly from Pure Syntax Errors. The One-Shot (No GCD) method recorded 404 pure syntax failures, demonstrating that even with in-context CoT reasoning, small models struggle to perfectly memorize irregular DSL structures.

In contrast, the application of Grammar-Constrained Decoding (GCD) across the remaining three methods significantly reduced this failure mode, reducing syntax errors by over 97% (down to between 4 and 11 failures). Crucially, the occurrence of syntax errors in the GCD methods is not exactly zero. An analysis of these residual failures revealed an FSM compilation artifact: while the EBNF strictly governs the AST structure, regular expressions for literal values (e.g., /-?[0-9]+/ for int_lit) theoretically permit infinitely long strings of digits. The LLM occasionally generated massive, unbounded integer literals that satisfied the FSM mask but triggered numeric overflow or syntax rejections in the native MiniZinc parser. This highlights a practical boundary of current FSM constraints: while structural syntax is guaranteed, lexical bounds-checking remains an open challenge.

2.

The GCD Schema Explosion and CoT Rescue:

The most profound architectural insight is visible in the Structural/Schema Errors category. The One-Shot (GCD Only) baseline exhibits a catastrophic spike, recording 355 structural failures (81.8% of its total errors). This visually quantifies the “Semantic-Syntax Trade-off”: by forcefully constraining the model without allowing it a <think> block, the LLM hallucinates types and variable scopes to satisfy the immediate token mask. The code is syntactically perfect, but structurally broken.

However, when CoT reasoning is introduced prior to the GCD mask—as seen in the One-Shot (CoT + Always GCD) and Dual-Phase methods—these schema errors drop precipitously (from 355 down to 147 and 154, respectively). This proves that the preserved semantic reasoning acts as a necessary blueprint, successfully guiding the rigid constraint engine through the correct typing and scoping derivations.

3.

Shifting the Frontier to Semantic Logic:

By effectively eliminating trivial syntax errors and significantly reducing structural hallucinations, the proposed Dual-Phase architecture successfully shifts the generation bottleneck to the final frontier: Semantic Logic Errors. For the Dual-Phase method, 47.5% of its remaining failures are purely semantic (e.g., applying the wrong operator or misunderstanding the optimization objective). This shift is highly desirable from a software engineering perspective. It demonstrates that our framework successfully isolates the LLM from the mechanical burden of formatting code, allowing the remaining errors to be purely reflective of the foundation model’s upper limit in mathematical and functional reasoning.

4.4.8. Effectiveness by Difficulty

Figure 5 illustrates the $pass@5$ success rate segmented by the intrinsic difficulty of the task. For ’Easy’ tasks (e.g., basic variable declarations), the unconstrained One-Shot (No GCD) and Zero-Shot baseline perform relatively well, natively solving almost half of the prompts. In these instances, our Dual-Phase architecture heavily relies on the Optimistic Fast-Path and the repair mechanism of Phase 2, achieving maximal performance with near-zero latency overhead. However, the true value of the neurosymbolic integration is revealed in the ’Medium’ and ’Hard’ categories—tasks involving complex arrays, subset domains, and deep logical implications. As task complexity increases, unconstrained LLMs suffer a sharp performance drop due to rising structural and syntactic hallucinations. Conversely, the Dual-Phase architecture demonstrates remarkable resilience. Guided by preserved reasoning, the Phase 2 constraint engine actively salvages these complex drafts, resulting in the widest performance delta precisely where standard LLMs fail the most.

4.4.9. Effectiveness by Task Category

Figure 6 illustrates the $pass@5$ success rate segmented by the category of the task. The visualization provides critical insights into how different architectural interventions interact with varying types of cognitive and syntactic loads.

Overall, the data confirms the overarching superiority of combining Chain-of-Thought (CoT) planning with formal constraints. The One-Shot (CoT + Always GCD) baseline and the Dual-Phase (Proposed) architecture dominate the majority of the categories. For domains requiring deep relational planning and mathematical deduction—specifically Arithmetic, Logic, and Optimization—the Dual-Phase approach achieves exceptional peak accuracies of 94%, 86%, and 88%, respectively. The CoT + Always GCD baseline trails closely behind, reinforcing the hypothesis that semantic planning prior to FSM masking is strictly necessary for solving complex relational constraints. Unconstrained methods and pure GCD methods suffer catastrophic failures in these areas (e.g., GCD Only achieves 0% in Logic and 9% in Optimization).

However, the Data Structures category (involving arrays, sets, and aggregations) reveals a highly counterintuitive anomaly. In this domain, the One-Shot (GCD Only) baseline achieves a 47% success rate, significantly outperforming both the Dual-Phase (32%) and CoT + Always GCD (26%) methods.

This phenomenon can be attributed to the architectural friction between the LLM’s procedural bias and the strict declarative syntax of MiniZinc arrays. We theorize that this is a case of Semantic Over-Reasoning: Declaring data structures in MiniZinc (e.g., array[1..n] of var int: x;) relies heavily on rigid, boilerplate syntax rather than multi-step logical deduction. When the GCD Only model is prompted, the FSM mask restricts its token choices so severely that it naturally funnels the generation into the correct, narrow syntactic corridor.

Conversely, when the LLM is permitted to reason via a <think> block first, it tends to over-complicate data structure initialization. Relying on its pre-trained bias toward imperative languages like Python, the model frequently plans out element-wise assignments, loops, or list comprehensions in its reasoning block. When the subsequent constrained phase attempts to map this out-of-domain procedural reasoning into the strict MiniZinc CFG, the model fights the logits mask, resulting in structural misalignment and semantic dead-ends.

Despite this specific edge case, the Dual-Phase architecture remains the most robust generalized approach: While unconstrained CoT introduces a slight vulnerability to over-reasoning in purely structural tasks, it is the only mechanism that protects the system against the total systemic collapse observed in the GCD-Only method for Logic and Optimization tasks. Furthermore, the Dual-Phase approach continues to offer the critical computational fast-path latency advantages established in Table 2, making it the optimal framework for generalized DSL code generation.

5. Conclusions and Future Work

In this paper, we addressed the enduring challenge of generating reliable, syntactically, and semantically correct code for niche Domain-Specific Languages (DSLs) using Large Language Models. We identified a critical friction point in modern reasoning-optimized LLMs: While Grammar-Constrained Decoding (GCD) successfully eliminates syntactic hallucinations, blindly applying strict constraints disrupts the model’s native Chain-of-Thought (CoT) planning, frequently trapping the generation in irreversible semantic dead-ends.

To resolve this semantic–syntax trade-off, we introduced a Dual-Phase Cascaded Neurosymbolic Framework. By injecting formal EBNF rules directly into the context window, we grounded the model symbolically. Phase 1 permits the model to reason unconstrained and output an optimistic draft; if this draft passes native compiler checks, the generation concludes efficiently. If the draft fails, Phase 2 preserves the successful unconstrained reasoning and utilizes it to guide a strict GCD fallback. Our empirical evaluation on a newly constructed benchmark of 100 MiniZinc constraint programming tasks demonstrated that this “think-then-constrain” approach significantly outperforms zero-shot, unconstrained one-shot, and pure GCD baselines. The dual-phase architecture effectively re-purposes the formal grammar from an adversarial restriction into a localized, reasoning-guided syntax repair engine, achieving highly reliable code generation without the computational burden of model fine-tuning.

5.1. Limitations and Threats to Validity

Despite the promising results, this study presents certain limitations. Regarding internal validity, our empirical evaluation relies on a specific class of reasoning models (e.g., Qwen2.5-Instruct). While this demonstrates the architecture’s viability on consumer-grade hardware, the interaction between GCD and larger, proprietary frontier models (e.g., OpenAI o1) remains unexplored and should be the subject of future work. Furthermore, while our LLM-as-a-judge implements a strict evaluation rubric grounded by the produced code, it remains a heuristic proxy; it is not a complete substitute for formal functional verification against mathematical unit tests. Instead of this static evaluation approach, execution-based evaluation (running the generated code through a solver to verify the output) would avoid penalizing mathematically equivalent but syntactically distinct formulations. However, introducing this mechanism in the evaluation pipeline would have increased the benchmark complexity significantly (by including data instance files, edge-case tested instances, etc.) and introduced severe computational overhead, requiring careful timeout and combinatorial explosion handling. However, we acknowledge that this is a logical next step and plan to include it in future work.

Regarding our choice of evaluation metrics, while our $pass@k$ evaluation conforms to standard code-generation benchmark practices, the results constitute point estimates; future large-scale evaluations on massive compute clusters could further bound these metrics with formal confidence intervals across repeated macro-runs.

Regarding external validity, our experiments were constrained to MiniZinc. While MiniZinc is highly representative of complex, declarative DSLs, the scalability of the optimistic bypassing mechanism to highly imperative DSLs or multi-file domain environments requires further investigation. Similary, highly parenthesized DSLs like SMT-LIB introduce tokenizer-alignment friction that deserves a specific investigation. However, we anticipate that by allowing Phase 1 (Optimistic Bypassing) to generate unconstrained text, the LLM can output its natively chunked tokens without FSM friction. The strict GCD phase is reserved only as a fallback, minimizing the overall exposure to tokenizer–grammar clashes. Finally, the Phase 2 GCD implementation relies on the Outlines finite-state machine compilation, which imposes an initial processing overhead and restricts the expressiveness of the CFG to strictly regular or right-recursive structures to avoid memory exhaustion.

A further limitation relates to the framework’s sensitivity to the quality and quantity of the in-context symbolic grounding. Exploratory experiments revealed that the success of Phase 1 heavily depends on the presence of semantic operator aliases and a well-structured grammar. For instance, omitting aliases frequently caused the model to hallucinate mainstream operators (e.g., generating Python’s and instead of MiniZinc’s /∖), leading to inevitable parser failures. Similarly, the structural design of the injected EBNF plays a decisive role; overly flattened or heavily left-recursive grammars confused the LLM’s internal derivation tracking and caused compilation bottlenecks in the FSM engine. Consequently, the provided grammar must be carefully authored. Finally, while the specific model utilized in our evaluation (Qwen2.5-Coder-1.5B-Instruct) technically supports a 32,768-token context window, small-parameter models are notoriously vulnerable to attention dilution (the “lost-in-the-middle” phenomenon). Injecting an exhaustive, thousands-of-lines-long enterprise grammar would likely saturate the model’s attention mechanism, degrading its ability to maintain localized, coherent Chain-of-Thought reasoning. Therefore, scaling this architecture to highly expansive DSLs currently necessitates using a pragmatic, task-relevant subset of the EBNF rather than an unabridged language specification.

5.2. Future Research Directions

The insights derived from this cascaded architecture open several promising avenues for future research. A primary direction involves integrating search-based reasoning into the generation pipeline. While pure token-level Monte Carlo Tree Search (MCTS) proved too computationally expensive and disjointed for initial generation in our early experiments, Localized MCTS holds immense potential as an advanced repair mechanism. Future work will explore transitioning from a greedy GCD fallback to a Localized MCTS that explores alternative syntax derivations guided by compiler-error feedback. Techniques like Grammar-Aligned Decoding [23] could be used in Phase 2 to overcome the challenges posed by GCD distribution bias (as discussed in Section 2), effectively resolving the remaining tokenizer-alignment frictions observed in our error categorization.

Additionally, we plan to extend this framework to a multi-agent paradigm, where one reasoning agent generates the semantic CoT, a symbolic parser identifies specific structural failure indices, and an MCTS agent systematically searches for valid localized repairs.

Finally, expanding the benchmark to encompass a wider variety of industrial DSLs (particularly highly imperative scripting languages and stateful configuration formats like Ansible and CMake) will further validate the generalizability of in-context symbolic grounding coupled with cascaded constraints across different language paradigms.