VGPO-MCTS: Distilling Step-Level Supervision from Value-Guided Tree Search for Mathematical Reasoning

Abstract

Large language models (LLMs) are increasingly used in applied intelligent systems, but mid-sized models still lag on mathematical reasoning, partly because reliable step-level supervision is scarce. Many existing remedies rely on costly human annotation, stronger teacher models, or heavy training pipelines, which limits practical adoption. We propose VGPO-MCTS (Value-Guided Group-wise Policy Optimization over Monte Carlo Tree Search), a search-and-distillation framework that constructs reusable step-level supervision from datasets that provide only problems and final answers. VGPO-MCTS augments a frozen backbone with (i) a lightweight value model that scores candidate reasoning states formed by a reasoning prefix and its candidate next step, and (ii) a policy updated with parameter-efficient adaptation. During search, the value model guides tree expansion and selection, while verified outcomes are propagated backward to correct node utilities. The corrected search trees are then distilled into two complementary datasets: a value regression dataset for value learning and group-wise sibling candidate sets for GRPO-style policy optimization. Experiments on GSM8K and the MATH dataset with ChatGLM3-6B and SciGLM-6B show stable round-wise improvements in final-answer exact match under a lightweight adaptation setting. After three rounds of self-training, the proposed framework improves performance by about 6.3 percentage points on GSM8K and about 3.9 percentage points on MATH across the two backbones.

1. Introduction

Large language models (LLMs) have shown strong capabilities in natural language understanding, knowledge integration, and multi-step reasoning, and are increasingly deployed as general-purpose assistants in real-world systems [1]. Representative model families such as GPT-style models [2] and LLaMA-style open foundation models [3] have demonstrated strong performance across a wide range of tasks. For complex reasoning problems, chain-of-thought (CoT) prompting [4] can improve solution quality by decomposing reasoning into explicit intermediate steps [5,6]. However, applied deployments still face a practical bottleneck: reliable step-level supervision is scarce and expensive to construct, especially for mathematical and logical reasoning tasks where intermediate annotations are difficult to obtain at scale [7].

Existing methods for improving mathematical reasoning can be broadly grouped into three categories. First, synthetic-rationale and self-training methods seek to reduce supervision cost by enabling models to generate, filter, refine, and reuse their own reasoning data. Representative studies have explored automatic annotation [8], synthetic data generation and augmentation [9], iterative self-improvement [10], and process-oriented supervision for mathematical reasoning [7,11], including frameworks such as STaR [12], V-STaR [13], SELF-DISCOVER [14], Self-Rewarding [15], and START [16]. These studies show that model-generated supervision can improve reasoning quality, but the resulting rationales and intermediate signals may vary substantially in quality, which makes filtering unstable and allows noise to accumulate over successive iterations. Second, search-based reasoning methods improve inference reliability by exploring multiple candidate trajectories rather than relying on a single sampled chain. Tree-of-Thoughts [17] and ReST-MCTS* [18], for example, show that structured exploration with verification can improve performance on challenging reasoning tasks. However, these methods primarily strengthen inference-time search, and their gains do not automatically translate into reusable step-level supervision for subsequent model improvement. Third, reinforcement-learning-based methods optimize reasoning behavior using outcome or preference signals. Recent work such as DeepSeek-R1 [19], together with related relative-optimization and preference-optimization studies [20,21], suggests that group-relative or preference-based optimization can effectively strengthen reasoning. However, such approaches may still depend on carefully designed rewards, large-scale optimization pipelines, or stronger teacher/verifier models. Consequently, an important practical research gap remains: under the realistic setting where only problem–answer pairs are available, it is still unclear how bounded-budget structured exploration can be converted into reliable and reusable step-level supervision for lightweight model improvement.

To address the above practical gap, we study a constrained setting for improving mathematical reasoning in mid-sized LLMs, where only problem–answer pairs are available as verifiable supervision, search and training budgets are limited, and no stronger external teacher model is assumed. Under this setting, we propose VGPO-MCTS (Value-Guided Group-wise Policy Optimization over Monte Carlo Tree Search). The key novelty of the framework is that value-guided MCTS is used not only to explore candidate reasoning steps during search but also to convert bounded-budget search trajectories into reusable step-level supervision rather than treating search purely as an inference-time aid. In addition, we introduce an outcome-anchored backward utility correction mechanism that propagates verified terminal outcomes to intermediate reasoning states, thereby producing soft utility targets for value learning. To support policy optimization, sibling candidates expanded from the same parent state are organized into group-wise training units, making GRPO-style optimization based on local competition feasible. The overall design is lightweight: it does not rely on a stronger external teacher model, keeps the backbone largely frozen, and updates only lightweight value-model parameters and LoRA-based policy adapters, enabling iterative reasoning improvement under resource-constrained conditions.

We validate the proposed framework from three complementary perspectives so that the motivation behind the experimental design is explicit. First, we evaluate value-model quality across self-training rounds, because the value model is the key signal that guides search and directly affects which partial reasoning states are explored and retained. Second, we evaluate end-to-end reasoning gains from the full search–distill–update loop, in order to verify whether the distilled supervision derived from bounded-budget search can translate into consistent improvements in final-answer accuracy. Third, we perform component-wise ablation analysis to isolate the contributions of value guidance, backward utility correction, group-wise policy optimization, and reflection-based branch control. Concretely, we report regression-oriented metrics for step-level utility prediction, round-wise exact match on GSM8K and the MATH dataset with two mid-sized backbones, namely ChatGLM3-6B and SciGLM-6B, and ablation results that clarify the role of each major component. Empirically, after three rounds of self-training, VGPO-MCTS improves ChatGLM3-6B from 72.29% to 78.61% on GSM8K and from 25.69% to 29.54% on MATH, while improving SciGLM-6B from 73.61% to 79.89% on GSM8K and from 25.17% to 29.07% on MATH. These results indicate that the proposed framework can stably convert bounded-budget structured search into reusable supervision and consistent reasoning gains under lightweight adaptation.

An overview of the proposed VGPO-MCTS framework is shown in Figure 1.

Figure 1. Overview of VGPO-MCTS. The framework alternates between value-guided search and lightweight training. Search constructs corrected trees, which are distilled into a value regression set and group-wise sibling sets for policy optimization.

The main contributions of this work are summarized as follows:

• We propose VGPO-MCTS, a framework that distills reusable step-level supervision from value-guided tree search under problem–answer-only supervision, without relying on stronger teacher models or additional human-written rationales.
• We introduce a joint value–policy learning design: a lightweight value model scores prefix–candidate reasoning states to guide exploration, while sibling candidates generated under the same parent context are organized into groups for group-wise policy optimization.
• We combine reflection-based branch control, terminal verification, and backward utility correction in a unified search procedure, enabling corrected search trees to serve as supervision for both value learning and policy optimization.
• We report empirical results on GSM8K and the MATH dataset with mid-sized backbones, showing stable round-wise improvements in final-answer exact match under lightweight adaptation and limited search budgets.

2. Background

2.1. Monte Carlo Tree Search for Stepwise Reasoning

Monte Carlo Tree Search (MCTS) provides a general framework for sequential decision making under limited search budgets, and has increasingly been adopted in reasoning tasks to support structured exploration over intermediate solution states. In stepwise mathematical reasoning, a node may be interpreted as a partial reasoning state s, while an outgoing edge corresponds to a candidate next-step action a. Executing a from s yields a successor state s′, namely a longer reasoning prefix obtained by appending one additional step. Under this view, MCTS incrementally constructs a search tree rooted at the problem statement and repeatedly alternates among selection, expansion, evaluation, and backpropagation. This state–action abstraction is useful because it turns long-form reasoning into a sequence of local decisions, thereby enabling the search procedure to compare multiple partially developed solution paths rather than committing to a single sampled chain.

The core mechanism of MCTS is to balance exploitation of currently promising branches with exploration of branches that remain under-visited. During selection, the algorithm recursively traverses the existing tree until it reaches a leaf or an expandable node. A common scoring rule combines an estimated action utility with an exploration bonus, for example

$$\mathrm{Score}(s,a)=Q(s,a)+c\sqrt{{\displaystyle \frac{ln\left(N\left(s\right)+1\right)}{N(s,a)+1}}}$$

(1)

where $Q(s,a)$ denotes the current utility estimate for choosing action a at state s, $N\left(s\right)$ is the visit count of state s, $N(s,a)$ is the visit count associated with edge $(s,a)$, and $c>0$ controls the strength of exploration. The first term prefers actions that have empirically appeared useful, whereas the second term encourages continued exploration of branches that may still contain high-value continuations but have not yet been sufficiently examined.

After selection, one or more candidate actions are expanded from the chosen node. The resulting successor state may then be assessed either by rollout simulation or by a learned evaluator. Let $\widehat{v}\left(s^{\prime }\right)$ denote the estimated utility of a newly reached state s′. This estimate is then propagated backward along the visited path, for example through the incremental update

$$Q(s,a)\leftarrow {\displaystyle \frac{N(s,a)Q(s,a)+\widehat{v}\left(s^{\prime }\right)}{N(s,a)+1}}$$

(2)

followed by the corresponding visit-count updates. In mathematical reasoning, such a mechanism is valuable because different branches often correspond to qualitatively different intermediate solution strategies. MCTS therefore provides a principled basis for allocating limited search effort toward promising reasoning prefixes while still preserving enough diversity to discover alternative derivations. In our setting, this background is important because search is used not only to improve inference-time exploration but also to generate the structured trajectories from which reusable supervision is later distilled.

2.2. Group-Wise Relative Optimization

Group-wise relative optimization is motivated by the observation that, when multiple candidates are generated under the same context, their quality is often more stable and informative when judged comparatively rather than on an absolute scale. In reasoning problems, this situation arises naturally: several candidate next steps may all appear locally plausible, yet their usefulness becomes clearer when they are contrasted under the same decision context. Suppose that, given a shared context x, a model produces a candidate group

$$\mathcal{G}\left(x\right)=\{y_1,y_2,\dots ,y_K\}$$

(3)

with associated scalar utilities or scores $\{r_1,r_2,\dots ,r_K\}$. Rather than treating each candidate independently, one may derive learning signals from their relative standing within the group. This reduces dependence on globally calibrated reward values and instead emphasizes local competition among alternatives generated from the same context.

A standard way to define within-group advantage is to normalize each score with respect to the group statistics, for example

$$A_i={\displaystyle \frac{r_i-\mu \left(r\right)}{\sigma \left(r\right)+ϵ}},\mu \left(r\right)={\displaystyle \frac{1}{K}}\sum _{j=1}^K{r}_j$$

(4)

where $\sigma \left(r\right)$ is the standard deviation of the group scores and ϵ is a small constant for numerical stability. Under this formulation, candidates whose scores exceed the group mean receive positive advantages, whereas below-average candidates receive negative ones. The resulting signal is therefore explicitly relative: what matters is not only whether a candidate is good in isolation but also whether it is better or worse than competing alternatives generated from the same context.

Using such relative advantages, one may optimize the policy by increasing the likelihood of stronger candidates and suppressing weaker ones. A generic objective can be written as

$${\mathcal{L}}_{\mathrm{grp}}\left(\theta \right)=-{\mathbb{E}}_{x,\mathcal{G}\left(x\right)}\left[{\displaystyle \frac{1}{K}}\sum _{i=1}^K{A}_{i}log{\pi }_{\theta }(y_i\mid x)\right]$$

(5)

This formulation is particularly well aligned with tree-based reasoning, because sibling branches expanded from the same parent state form a natural comparison group: they share the same prefix context but compete as alternative local continuations. For this reason, group-wise relative optimization provides the conceptual basis for the sibling-based policy-learning design adopted later in our method, while the concrete score assignment and training objective used in VGPO-MCTS are introduced in the Method section.

3. Related Work

3.1. Process Supervision and Self-Training

A major line of work improves reasoning by constructing intermediate supervision or high-quality reasoning traces when manual step annotations are scarce. In the direction of process supervision, Let’s Verify Step by Step shows that process supervision can outperform outcome supervision on difficult mathematical reasoning tasks and releases the PRM800K dataset for step-level feedback. Math-Shepherd further shows that useful step-level supervision and verification signals can be constructed automatically without human-written step annotations. These studies establish the importance of intermediate supervision, but they also highlight a practical tension: strong process-level signals are beneficial, yet obtaining them reliably and at scale remains nontrivial [7,11].

Closely related to process supervision is a family of methods based on model-generated rationales and self-training. STaR [12] is an early representative that bootstraps rationales through iterative self-training. V-STaR [13] introduces an explicit verifier to reduce error accumulation during rationale filtering. Other frameworks such as SELF-DISCOVER [14], Self-Rewarding [15], and START [16] further study how generation, critique, self-evaluation, refinement, and tool use can be combined to improve reasoning quality under limited human supervision. Related studies have also explored automatic annotation, synthetic data generation and augmentation, and iterative self-learning using model-generated signals. Taken together, these methods show that self-generated supervision can be effective, but they also reveal a recurring weakness: when rationale quality is unstable, noisy intermediate signals may be reinforced across rounds, making later training increasingly sensitive to filtering quality and verifier strength [8,9,10].

In addition, several works improve mathematical reasoning through domain-specific data construction and training. Minerva [22] improves quantitative reasoning by training on scientific and technical corpora. MetaMath [23] shows that bootstrapped mathematical question construction can substantially strengthen mathematical reasoning in open models. WizardMath [24] further demonstrates that reinforced Evol-Instruct can improve mathematical reasoning performance. More broadly, instruction-tuning frameworks such as WizardLM [25] highlight the importance of carefully constructed supervision for improving downstream reasoning behavior. While these efforts substantially strengthen reasoning ability, they generally focus on better training data or stronger supervision resources, whereas our setting assumes only problem–answer pairs and asks how additional reusable step-level signals can be distilled from search itself.

Compared with these approaches, our work addresses a different supervision setting. Rather than assuming access to high-quality intermediate rationales, explicit verifier labels, or externally curated process traces, we focus on the common case in which training data provide only problems and final answers. The central challenge in this setting is not merely how to improve reasoning with additional supervision, but how to construct reusable step-level supervision when such supervision is absent at the outset. Our framework addresses this challenge by using structured search trajectories anchored by verifiable terminal outcomes as the source of subsequent value learning and policy optimization.

3.2. Search-Based Reasoning

Chain-of-thought (CoT) prompting [4] is a widely used technique for eliciting intermediate reasoning steps, and has been shown to improve accuracy on multi-step tasks in many settings [5,6]. Nevertheless, CoT alone does not guarantee robustness: generated rationales can vary in quality across samples, and single-shot decoding may fail on harder instances. These limitations motivate approaches that increase diversity during inference or introduce explicit mechanisms for exploration and selection.

Another important line of research improves reasoning reliability through structured exploration over multiple candidate branches. Tree-of-Thoughts [17] organizes intermediate thoughts in a tree and interleaves generation with evaluation, backtracking, and branching search, thereby overcoming the limitations of single-chain decoding. ReST-MCTS* [18] further connects tree search with process-aware scoring and self-improvement, using structured exploration to improve solution reliability and to collect refined reasoning traces for subsequent training. Liu et al. further show that a value model learned during training can be reused during MCTS-based decoding, highlighting the broader potential of value-guided search beyond standard greedy or beam-style generation. Collectively, these studies demonstrate that search can improve difficult reasoning problems by allocating computation across multiple partial solution paths rather than committing to a single trajectory too early [26].

These works demonstrate the value of search for difficult reasoning tasks, but they also leave open a key question for lightweight self-improvement: how should highly branched and heterogeneous search trajectories be converted into reusable and stable training signals? Our framework is closely related to this line of work in that it also performs structured exploration. However, unlike approaches that mainly use search for inference-time enhancement or for collecting refined trajectories, we explicitly turn value-guided search results into two complementary forms of supervision: one for value learning and the other for group-wise policy optimization. Beyond mathematical reasoning, Jiang et al. study a value-guided structured sampling and optimization loop for generative recommendation, in which a value estimator guides tree-like expansion and sibling-style optimization is applied at selected branching points. Although their task differs from ours, the work further supports the broader relevance of combining value-guided search with group-relative policy improvement [27].

3.3. Reinforcement Learning and Relative Optimization

Reinforcement learning has been widely used to align large language models with preferences and task-specific objectives. Classical pipelines typically optimize policies with PPO [28,29] against learned reward models, often together with KL regularization for stability. More recently, methods such as DPO [20] simplify preference optimization by avoiding explicit online reinforcement learning. These studies establish the broader optimization background for learning from comparative or reward-like signals, even when the precise supervision source differs from reasoning-specific settings.

For reasoning-centric behavior, recent work has begun to apply reinforcement learning more directly to long-chain reasoning. DeepSeek-R1 [19] shows that reinforcement-learning-based training can substantially improve reasoning, self-reflection, and long-form solution behavior. Light-R1 [21] further studies curriculum-style combinations of supervised fine-tuning, preference optimization, and reinforcement learning for stable reasoning training. Other recent efforts, such as DAPO [30] and VAPO [31] investigate practical and robust optimization strategies for open reasoning models. Taken together, these studies suggest that both the choice of optimization objective and the structure of candidate sampling can substantially influence the resulting reasoning behavior.

At the objective level, Group Relative Policy Optimization (GRPO) performs group-relative updates by comparing multiple candidates under the same prompt, thereby reducing reliance on absolute reward calibration. This group-relative principle is particularly appealing for reasoning settings in which local candidate quality is more naturally comparative than absolute. Our framework adopts this principle in a search-generated supervision setting: the optimization groups are not free-form samples under the same prompt, but sibling candidates expanded under the same search context, and their optimization signals are grounded in step-utility estimates anchored by verifiable terminal outcomes rather than externally provided preference labels. In this sense, our method connects relative optimization with value-guided search and supervision distillation, rather than applying it in a standard preference-learning pipeline.

3.4. Positioning of VGPO-MCTS

Taken together, the existing studies can be broadly viewed as emphasizing one of three directions: improving reasoning supervision through process supervision, synthetic rationale construction, and self-training; improving inference reliability through structured search or improving reasoning behavior through reinforcement learning and group-relative optimization. Rather than focusing on only one of these directions, VGPO-MCTS integrates their most useful elements into a single framework. Specifically, it uses value-guided MCTS to perform structured exploration, uses outcome-anchored utility correction to turn search trajectories into reusable step-level supervision, and uses sibling-group relative optimization to construct policy-learning signals from locally competing branches. This positioning distinguishes VGPO-MCTS from methods that rely only on search enhancement, only on self-training, or only on preference-style optimization.

To further clarify this positioning, Table 1 compares our framework with representative prior methods in terms of supervision source, search usage, learning signal, and external dependence. The comparison highlights that VGPO-MCTS explicitly combines value-guided search, outcome-anchored utility correction, and sibling-group relative optimization to construct reusable step-level supervision under problem–answer-only supervision.

Table 1. Comparison of VGPO-MCTS with representative reasoning-improvement frameworks in terms of supervision source, learning signal, search usage, and external dependence.

Method	Supervision	Learning Signal	Search	Dependence
STaR [12]	Self rationales	Selected rationales	No	No strong teacher
ToT [17]	Thought evaluation	Search guidance	Yes	No strong teacher
ReST-MCTS* [18]	Search traces + process rewards	Distilled traces	Yes	Reward model
DeepSeek-R1 [19]	RL signals	Relative RL	No	RL pipeline
VGPO-MCTS	Problem–answer pairs + verified outcomes	Distilled step supervision	Yes	No strong teacher

4. Method

Given the background above and the gap identified in prior work, we propose VGPO-MCTS (Value-Guided Group-wise Policy Optimization over Monte Carlo Tree Search), a search-and-distillation framework for mathematical reasoning in large language models. Given a supervised dataset of problem–answer pairs $(q,a^{\star })$, where q denotes the problem and $a^{\star }$ denotes the ground-truth final answer, VGPO-MCTS performs bounded-budget search on q, corrects intermediate utilities using verifiable terminal outcomes, and distills the resulting search trees into training signals for both value learning and policy optimization.

The framework contains three tightly coupled components: a value model that scores partial reasoning states, a value-guided tree-search procedure with reflection control and terminal verification, and an iterative learning loop that extracts a value regression dataset and a group-wise policy dataset from corrected search trees. This organization follows directly from the problem setting: search is used to explore candidate reasoning paths under limited budgets, backward correction converts verified outcomes into reusable intermediate supervision, and group-wise policy learning uses sibling competition to improve the policy without relying on a stronger external teacher. Throughout training, the backbone language model remains frozen; only lightweight parameters are updated, namely, LoRA adapters for the policy model and newly introduced modules for the value model.

4.1. Value Model with Multi-Query Attention Pooling

To guide search at the step level, we define a value model over partial reasoning states. Given a problem q, a preceding reasoning prefix $S_{<t}$, and a candidate next step ${\Delta }_t$, we form the scored partial solution state

$$S_{1:t}=(S_{<t},{\Delta }_t),$$

and construct a textual input $x(q,S_{<t},{\Delta }_t)$ under a fixed prompt template. The value model outputs a bounded scalar

$$Q_{\psi }\left(x(q,S_{<t},{\Delta }_t)\right)\in [0,1],$$

(6)

where $Q_{\psi }(·)$ denotes the value-scoring function parameterized by the learnable parameters ψ. This scalar measures the utility of appending the candidate step ${\Delta }_t$ to the current reasoning prefix $S_{<t}$ for eventually reaching the correct final answer.

An example prompt for step-utility scoring is shown in Figure 2.

Figure 2. An example prompt for step-utility scoring given $(q,S_{<t},{\Delta }_t)$.

The value model is trained on input–target pairs $(x,v)$, where x is the textual representation of a scored reasoning state and $v\in [0,1]$ is its target utility. We use mean squared error:

$${\mathcal{L}}_{\mathrm{val}}\left(\psi \right)={\mathbb{E}}_{(x,v)\sim {\mathcal{D}}_{\mathrm{val}}}{\left(Q_{\psi }\left(x\right)-v\right)}^2.$$

(7)

Here, the subscript “val” indicates value-model training: ${\mathcal{L}}_{\mathrm{val}}$ denotes the value-model loss, and ${\mathcal{D}}_{\mathrm{val}}$ denotes the corresponding value-training dataset.

We implement $Q_{\psi }$ by attaching a lightweight value head to a frozen Transformer backbone. Given input $x(q,S_{<t},{\Delta }_t)$, the backbone produces last-layer hidden states $H\in {\mathbb{R}}^{T\times d}$, where T is the sequence length and d is the hidden dimension. Only the added value-head parameters are optimized.

The architecture of the value model is illustrated in Figure 3.

Figure 3. Value model architecture: a frozen LLM backbone with a lightweight multi-query attention pooling head.

To summarize the token sequence, we introduce K learnable query vectors ${\left\{q_k\right\}}_{k=1}^K$ and perform attention pooling:

$$\begin{array}{cc}\hfill {\alpha }_{i,k}& ={\mathrm{softmax}}_i\left({\displaystyle \frac{H_i^{\top }{q}_k}{\sqrt{d}}}\right),\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill p_k& =\sum _{i=1}^T{\alpha }_{i,k}{H}_i,\hfill \end{array}$$

(9)

where ${\alpha }_{i,k}$ is the attention weight assigned to token i by query k, and $p_k$ is the pooled representation for query k. We concatenate the pooled vectors as $p=[p_1;\dots ;p_K]\in {\mathbb{R}}^{Kd}$ and map them to a scalar through an MLP followed by a sigmoid:

$$Q_{\psi }\left(x(q,S_{<t},{\Delta }_t)\right)=\sigma \left({\mathrm{MLP}}_{\psi }\left(p\right)\right)\in [0,1].$$

(10)

4.2. Value-Guided MCTS with Reflection and Backward Correction

Following the preliminaries in Section 2, we now describe how MCTS is instantiated in our framework for stepwise mathematical reasoning. Let $s_{t-1}$ denote a parent state. The policy model ${\pi }_{\theta }(·\mid s_{t-1})$ generates a next-step text ${\Delta }_t$, and the child state is formed by appending it:

$$s_t=f(s_{t-1},{\Delta }_t).$$

Each action therefore corresponds to a single textual reasoning step.

Reflection controller. Before expanding a node, we query a prompt-based controller implemented by the frozen backbone model. Given state s, the controller outputs

$$z\left(s\right)\in \{\mathtt{continue},\mathtt{end},\mathtt{error}\}.$$

continue performs normal expansion, end terminates the trajectory and triggers answer extraction, and error inserts a short corrective reflection message into the context before continuing. The controller is used only for search-time branch control; terminal correctness is still determined by final-answer verification.

An overview of value-guided search with reflection control and backward correction is shown in Figure 4.

Figure 4. Value-guided search with exploration, prompt-based reflection control, terminal verification, and backward correction.

Using the value estimate defined in Section 4.1, MCTS maintains visit counts $N(·)$ and selects among expanded children according to a value-guided exploration score. For a parent node p and an expanded child s, we compute the selection score

$$U\left(s\right)=\widehat{V}\left(s\right)+c\sqrt{{\displaystyle \frac{log\left(1+N\left(p\right)\right)}{1+N\left(s\right)}}}.$$

(11)

Selection chooses the child with the largest score:

$$s^{\star }=arg\underset{s}{max}U\left(s\right).$$

(12)

Here, $arg{max}_{s}U\left(s\right)$ returns the maximizing child state, rather than the maximum score value itself.

A trajectory τ is called verifiable if a final answer $\widehat{a}\left(\tau \right)$ can be deterministically extracted. We define

$$\mathrm{ver}\left(\tau \right)=\mathbb{I}\left[\widehat{a}\left(\tau \right)\mathrm{is}\mathrm{extractable}\right],R\left(\tau \right)=\mathbb{I}[\widehat{a}\left(\tau \right)=a^{\star }]\in \{0,1\}\mathrm{when}\mathrm{ver}\left(\tau \right)=1.$$

Here, $\mathbb{I}[·]$ denotes the indicator function, which takes value 1 when the enclosed condition is true and 0 otherwise.

When a rollout terminates with a verifiable outcome, we use the terminal correctness signal to perform backward correction on the visited path. When a rollout terminates with $\mathrm{ver}\left(\tau \right)=1$, we update node-stored utilities along the root-to-leaf path $\mathcal{P}\left(\tau \right)=(s_1,\dots ,s_T)$ using terminal correctness $y=R\left(\tau \right)$:

$$\widehat{V}\left(s_t\right)\leftarrow (1-{\alpha }_t)\widehat{V}\left(s_t\right)+{\alpha }_{t}y,{\alpha }_t={\alpha }_0{\gamma }^{(T-t)},$$

(13)

where ${\alpha }_0\in (0,1)$ is the base correction rate, and $\gamma \in (0,1]$ is a decay factor controlling how strongly the terminal outcome is propagated along the path. Under ${\alpha }_t={\alpha }_0{\gamma }^{(T-t)}$, nodes closer to the terminal state receive larger updates, while earlier nodes receive smaller ones. The choice of ${\alpha }_0$ and γ can affect the character of utility propagation. A smaller ${\alpha }_0$ weakens the influence of terminal supervision, whereas a larger ${\alpha }_0$ makes the correction more aggressive. Likewise, a smaller γ concentrates correction more strongly on later nodes, while a larger γ spreads supervision more broadly to earlier prefixes. These hyperparameters may therefore influence not only the locality of the corrected utility targets but also the stability of the subsequent value-learning process. A more systematic sensitivity study of backward-correction hyperparameters, especially ${\alpha }_0$ and γ, is an important direction for future work. The corrected utility is denoted by $\overline{V}\left(s_t\right)$. If $\mathrm{ver}\left(\tau \right)=0$, no correction is applied.

4.3. Distillation and Group-Wise Policy Optimization

We next describe how corrected search trees are converted into training data, and how group-wise relative optimization is instantiated in our setting.

Instance typing with two depth limits. To prioritize reliable supervision under limited search budgets, we optionally type instances using two depth limits $d_1<d_2$. An instance is easy if a verifiable and correct trajectory is found under the smaller depth limit $d_1$. It is high-value if none is found under $d_1$ but at least one is found under the larger depth limit $d_2$. It is unusable if no verifiable-correct trajectory is found under $d_2$ (or no verifiable terminal answer is produced). In learning, we downsample easy instances, emphasize high-value ones, and discard unusable ones.

Value regression dataset. After backward correction (Section 4.2), we construct a value regression dataset

$${\mathcal{D}}_{\mathrm{val}}=\left\{\left(x\left(s_t\right),\overline{V}\left(s_t\right)\right)\right\},$$

(14)

where $x\left(s_t\right)$ is the textual representation of state $s_t$ and $\overline{V}\left(s_t\right)\in [0,1]$ is the corrected utility. We include only nodes from trajectories with $\mathrm{ver}\left(\tau \right)=1$. The value model is trained with Equation (7).

Group-wise policy dataset and GRPO. We use only trees that contain at least one verified-correct trajectory so that each group is anchored by at least one correct candidate. For each parent state p along a verified-correct path, suppose search expands m children under the same context, denoted as ${\left\{s_i\right\}}_{i=1}^m$, where each child is generated by a next-step text ${\Delta }_i$. We treat these siblings as one group:

$${\mathcal{G}}_p={\left\{(p,{\Delta }_i,s_i)\right\}}_{i=1}^m.$$

(15)

Each child is assigned a group score based on corrected utility:

$$u_i=\overline{V}\left(s_i\right)\in [0,1].$$

(16)

We then compute standardized within-group advantages

$$A_i={\displaystyle \frac{u_i-{\mu }_p}{{\sigma }_p+\epsilon }},{\mu }_p={\displaystyle \frac{1}{m}}\sum _{i=1}^m{u}_i,{\sigma }_p=\sqrt{{\displaystyle \frac{1}{m}}{\displaystyle \sum _{i=1}^m}{(u_i-{\mu }_p)}^2},$$

(17)

Here, ε is a fixed small positive constant introduced for numerical stability, so that the normalization remains well-defined when ${\sigma }_p$ is very small. Then, we optimize the GRPO objective

$${\mathcal{L}}_{\mathrm{GRPO}}\left(\theta \right)=-{\mathbb{E}}_{{\mathcal{G}}_p}\left[{\displaystyle \frac{1}{m}}{\displaystyle \sum _{i=1}^m}\mathrm{sg}\left(A_i\right)log{\pi }_{\theta }({\Delta }_i\mid p)\right]+{\beta }_{\mathrm{KL}}\mathrm{KL}\left({\pi }_{\theta }\parallel {\pi }_{\mathrm{ref}}\right),$$

(18)

Here, ${\mathcal{L}}_{\mathrm{GRPO}}$ denotes the GRPO objective for policy optimization. θ denotes the learnable parameters of the policy model ${\pi }_{\theta }$. $\mathrm{sg}(·)$ denotes the stop-gradient operator, so gradients are not propagated through $A_i$ and the advantage term is treated as a constant during backpropagation. ${\pi }_{\mathrm{ref}}$ is a frozen reference policy for KL regularization.

4.4. Overall Training Procedure

The overall learning process follows an alternating search–distill–update loop. We first train a bootstrap value model on the available step-level supervision. At the beginning of each round, the current value model guides bounded-budget MCTS over the training set and produces corrected search trees, from which two distilled datasets are extracted: (i) a value regression dataset ${\mathcal{D}}_{\mathrm{val}}^{\left(r\right)}$ and (ii) a group-wise policy dataset ${\mathcal{D}}_{\mathrm{grpo}}^{\left(r\right)}$.

Within Round r, the group-wise policy dataset ${\mathcal{D}}_{\mathrm{grpo}}^{\left(r\right)}$ is first used to update the policy model. Then, the value regression dataset ${\mathcal{D}}_{\mathrm{val}}^{\left(r\right)}$ is used to train the next-round value model, which will guide search in Round r + 1. The value model therefore serves as the search controller at the beginning of each round, while the policy model is the main object of adaptation within that round.

Algorithm 1 summarizes the full iterative training procedure.

Algorithm 1 VGPO-MCTS: Iterative Search, Distillation, and Training

Require:  policy model ${\pi }_{\theta }$, training set $\mathcal{D}={\left\{(q_i,a_i^{\star })\right\}}_{i=1}^N$, bootstrap supervision ${\mathcal{D}}_{\mathrm{boot}}$, depth limits $d_1,d_2$, number of rounds R   1: Train bootstrap value model $Q_{{\psi }^{\left(1\right)}}$ on ${\mathcal{D}}_{\mathrm{boot}}$   2: Initialize policy model as ${\pi }_{{\theta }^{\left(0\right)}}$   3: for $r=1$ to R do   4:       ${\mathcal{D}}_{\mathrm{val}}^{\left(r\right)}\leftarrow \varnothing$   5:       ${\mathcal{D}}_{\mathrm{grpo}}^{\left(r\right)}\leftarrow \varnothing$   6:       for all $(q,a^{\star })\in \mathcal{D}$ do   7:             Run bounded search with ${\pi }_{{\theta }^{(r-1)}}$ and $Q_{{\psi }^{\left(r\right)}}$ to obtain search tree $\mathcal{T}$   8:             Type q as easy/high-value/unusable using depth limits $d_1,d_2$   9:             Perform terminal verification for completed trajectories in $\mathcal{T}$ 10:             Apply backward correction to all verifiable trajectories in $\mathcal{T}$ 11:             ${\mathcal{D}}_{\mathrm{val}}^{\left(r\right)}\leftarrow {\mathcal{D}}_{\mathrm{val}}^{\left(r\right)}\cup \mathrm{ExtractVal}\left(\mathcal{T}\right)$                      ▹ ExtractVal uses verifiable trajectories 12:             ${\mathcal{D}}_{\mathrm{grpo}}^{\left(r\right)}\leftarrow {\mathcal{D}}_{\mathrm{grpo}}^{\left(r\right)}\cup \mathrm{ExtractGRPO}\left(\mathcal{T}\right)$                 ▹ ExtractGRPO uses trees containing verified-correct trajectories 13:       end for 14:       ${\theta }^{\left(r\right)}\leftarrow arg{min}_{\theta }{\mathcal{L}}_{\mathrm{GRPO}}\left(\theta ;{\mathcal{D}}_{\mathrm{grpo}}^{\left(r\right)}\right)$ 15:       if $r<R$ then 16:             ${\psi }^{(r+1)}\leftarrow arg{min}_{\psi }{\mathcal{L}}_{\mathrm{val}}\left(\psi ;{\mathcal{D}}_{\mathrm{val}}^{\left(r\right)}\right)$ 17:       end if 18: end for 19: return final policy model ${\pi }_{{\theta }^{\left(R\right)}}$ and final value model $Q_{{\psi }^{\left(R\right)}}$

This alternating procedure couples search quality, distilled supervision, and lightweight model adaptation: stronger policy proposals improve the search tree, while a stronger value model improves prioritization under bounded search budgets. The result is a closed loop between search quality, distilled supervision, and lightweight model adaptation.

5. Experiments

This section evaluates VGPO-MCTS from three complementary perspectives. First, we analyze the value model (VM), including its round-wise prediction quality and qualitative text-attention patterns. Second, we evaluate the effectiveness of the full iterative framework across three self-training rounds on mathematical reasoning benchmarks. Third, we perform ablations to examine the contributions of value-model design choices and key components of the overall training loop.

5.1. Experiment Setting

5.1.1. Datasets

We evaluate on two standard mathematical reasoning benchmarks, GSM8K [32] and the MATH dataset [33], using their official train/test splits. All search, data distillation, and parameter updates are performed on the training split only. Unless otherwise stated, all final results reported in this section are measured on the official test split. Auxiliary held-out data are used only for training monitoring when needed.

GSM8K mainly contains grade-school arithmetic word problems with relatively standardized reasoning patterns and numeric final answers. MATH is more diverse and challenging, spanning multiple topics, such as algebra, geometry, counting, and calculus, and often requiring longer and more heterogeneous derivations. This difference makes MATH a harder benchmark for verifiable self-improvement under bounded search budgets.

We chose GSM8K and MATH because they are widely used benchmarks for multi-step mathematical reasoning and because they align well with the supervision setting considered in this work. In both datasets, each problem is paired with a verifiable final answer, which makes deterministic terminal verification and backward correction feasible. These properties make them suitable for evaluating whether VGPO-MCTS can transform problem–answer supervision into reusable step-level training signals. At the same time, we acknowledge that such benchmarks mainly contain well-formulated mathematical problems and therefore do not fully reflect more open-ended settings in which a key difficulty is to formulate a real-world problem mathematically before solving it. Extending the framework to such broader settings would require richer datasets and stronger verification mechanisms, and we leave this as an important direction for future work.

5.1.2. Evaluation Metrics

For downstream reasoning performance, we report exact match (EM, %) on the final answer obtained via deterministic extraction and normalization. For GSM8K, we extract the last valid number from the model output and normalize formatting. For MATH, we prioritize the last \boxed{…} span when present; otherwise, we use the last extractable answer span and apply standard normalization.

For the value model, we report mean squared error (MSE), mean absolute error (MAE), and Hit@bin(0.10). For Hit@bin(0.10), the interval $[0,1]$ is partitioned into bins of width 0.1. A prediction is counted as correct only when it falls into the same bin as the target label. We use the same evaluation code across all rounds and all variants.

These metrics are chosen to evaluate the framework at the level of verifiable reasoning performance and step-level utility prediction. In particular, exact match reflects whether the final answer can be correctly recovered under deterministic verification, while MAE, MSE, and Hit@bin assess the quality of the learned step-utility signal that guides search. At the same time, we acknowledge that broader reliability assessment for intelligent reasoning systems may additionally involve robustness, calibration, verification quality, and consistency-related evaluation beyond task-level performance alone [34,35]. Extending VGPO-MCTS toward such broader reliability-oriented evaluation settings is an important direction for future work.

5.1.3. Implementation Details

Bootstrap Value Model

The bootstrap value model is trained from step-level supervision tuples of the form $(q,S_{<t},{\Delta }_t,v)$, where q is the problem, $S_{<t}$ is the prefix of preceding reasoning steps, ${\Delta }_t$ is the candidate next step, and $v\in [0,1]$ is the target utility. We use the multi-query value head described in Section 4.1, with a frozen ChatGLM3-6B or SciGLM-6B backbone. Unless otherwise stated, the value model input length is capped at 768 tokens. Under this setting, the filtering rate of overlong samples is below 0.2% and the truncation rate of prefix steps remains below 0.4%, indicating that the chosen length budget preserves almost all usable examples.

Search Configuration

During each round, we run bounded-budget MCTS on the training split only. We use a two-stage depth schedule: a smaller cap ($d_{max}=5$) for instance typing and a larger cap ($d_{max}=10$) for final distillation. At each expansion, the policy model samples a fixed number of next-step candidates ($K_{\mathrm{expand}}=4$). Child selection uses the value-guided UCB score in Equation  (11) with exploration coefficient $c_{\mathrm{UCB}}=1.2$. Unless otherwise stated, the per-instance search budget is $I_{max}=16$ iterations. A lightweight reflection controller produces continue, end, or error to regulate search-time branching, while terminal correctness is still determined only by deterministic answer extraction and exact matching.

Training Protocol

We report three iterative self-training rounds, denoted as Round 1, Round 2, and Round 3. Round 1 starts from the initial backbone and the bootstrap value model trained before iterative self-training. At each round, the current value model guides MCTS on the training split, from which we distill (i) a value regression dataset for retraining the value model and (ii) a group-wise policy dataset for GRPO-style policy optimization. The policy model is updated with LoRA only, while the backbone remains frozen. After policy adaptation, a new round of value-guided MCTS is run to produce improved process supervision, and the value model is retrained on the newly distilled supervision. Unless otherwise specified, the term “Round-r VM” refers to the value model obtained after the r-th round of search, distillation, and retraining, rather than the bootstrap VM used to initiate Round 1 search. Thus, Round 3 corresponds to the third repetition of the same search–distill–update cycle, using the progressively improved policy and value model from the previous rounds.

Optimization

For policy adaptation, we use LoRA with r = 64, α = 128, and dropout 0.1, targeting query_key_value, dense_h_to_4h, and dense_4h_to_h. We train for 3 epochs with sequence length 1024, micro-batch size 1, gradient accumulation 2, and learning rate $2\times {10}^{-5}$. For the value model, we train the lightweight added modules using the MSE objective in Equation  (7). Unless otherwise stated, all experiments are conducted on two NVIDIA RTX 3090 GPUs (24 GB each).

5.2. Value Model Analysis

Before evaluating the full iterative framework, we first analyze the value model itself. The value model provides the main guidance signal during search. Its quality affects which partial trajectories are explored and, consequently, the quality of the distilled supervision collected in later rounds.

5.2.1. Round-Wise VM Performance

Table 2 reports the round-wise prediction quality of the value model on the held-out test split for analysis only. Here, “Round-r VM” denotes the value model obtained after the r-th round of search, distillation, and retraining, rather than the bootstrap VM used to initiate Round 1 search.

Table 2. Round-wise performance of the value model on the test split.

VM Round	MAE	MSE	Hit@bin(0.10)
Round 1	0.225	0.072	0.349
Round 2	0.202	0.060	0.437
Round 3	0.190	0.053	0.472

Table 2 shows monotonic improvement of the value model across rounds. MAE and MSE decrease from Round 1 to Round 3, while Hit@bin(0.10) increases over the same period. Because the value model is used directly during search, this improvement strengthens the signal available for node selection and expansion.

The three metrics also move in the same direction across rounds. This pattern indicates broad improvement in value prediction quality rather than a trade-off between different evaluation criteria.

5.2.2. Text-Attention Visualization of the Value Model

To better understand how the value model evaluates candidate reasoning steps, we visualize representative text-level attention heatmaps. Figure 5 presents four representative cases: high-target/high-prediction, low-target/low-prediction, high-target/low-prediction, and low-target/high-prediction. These correspond to aligned high-value scoring, aligned low-value scoring, underestimation, and overestimation, respectively.

Figure 5. Representative text-attention heatmaps of the value model on step-utility scoring.

The heatmaps reveal several recurring qualitative patterns. In aligned high-value cases, the model focuses on key quantities, local constraints, and operations that directly connect the candidate step to the problem requirements. In aligned low-value cases, the input may still contain locally plausible tokens, but the attention pattern does not support a coherent global continuation. In underestimation cases, the model still attends to structurally relevant parts of the reasoning chain, even when the final score remains conservative. In overestimation cases, the model places more emphasis on superficially relevant numbers or local arithmetic patterns, which can lead to overconfidence on globally incorrect continuations.

These patterns indicate that the value model relies on both problem information and reasoning history when estimating step utility. The remaining errors mainly arise in cases where locally plausible continuations are globally incorrect. This qualitative observation aligns with Table 2, where the value model improves steadily while the prediction task remains challenging.

5.3. Round-Wise Improvement of the Full Framework

We next evaluate the full search–distill–fine-tune loop over multiple self-training rounds. At each round, the current value model guides bounded-budget MCTS on the training split, distills a value regression dataset and a group-wise policy dataset, updates the policy model with lightweight LoRA adaptation, and then retrains the value model on the newly collected supervision.

Table 3 reports round-wise EM (%) on GSM8K and MATH for both backbones.

Table 3. Round-wise exact match (EM, %) on GSM8K and MATH.

Model	GSM8K (EM, %)			MATH (EM, %)
	Round 1	Round 2	Round 3	Round 1	Round 2	Round 3
ChatGLM3-6B	72.29	76.84	78.61	25.69	28.31	29.54
SciGLM-6B	73.61	77.94	79.89	25.17	27.90	29.07

Table 3 shows monotonic improvement across rounds on both datasets and both backbones. The same trend appears for ChatGLM3-6B and SciGLM-6B, indicating that the iterative framework is not restricted to a single backbone instance.

The gains are larger in earlier rounds and smaller in later rounds. This pattern indicates that the framework first captures more accessible improvements and then continues to improve on more difficult cases under the same search budget. The same pattern appears on both GSM8K and MATH.

The improvement of the full framework follows the same direction as the improvement of the value model. As the value model becomes more accurate, search allocates more computation to promising prefixes and reduces unproductive exploration. This behavior is relevant for both datasets, although the underlying difficulty differs: GSM8K has more standardized reasoning patterns, while MATH contains longer and more heterogeneous derivations.

Table 3 therefore highlights a stable improvement pattern across rounds, datasets, and backbones.

5.4. Distilled Supervision Across Rounds

To better understand the mechanism behind the round-wise gains, we examine the amount of distilled supervision obtained in each round. Table 4 reports the sizes of the distilled value regression dataset ${\mathcal{D}}_{\mathrm{val}}^{\left(r\right)}$ and the distilled group-wise policy dataset ${\mathcal{D}}_{\mathrm{grpo}}^{\left(r\right)}$.

Table 4. Sizes of the distilled value regression dataset and group-wise policy dataset across rounds.

Round	${\mathcal{D}}_{\mathbf{val}}^{\left(\mathit{r}\right)}$ (#ValPairs)	${\mathcal{D}}_{\mathbf{grpo}}^{\left(\mathit{r}\right)}$ (#Groups)
1	35,520	13,574
2	47,360	15,360
3	63,814	20,723

Table 4 shows that both the value regression dataset and the group-wise policy dataset grow across rounds. Later rounds therefore provide more reusable supervision than earlier rounds.

The growth of distilled supervision accompanies the improvements observed in both the value model and the final task metrics. As search becomes more effective, it yields more verifiable and informative trajectories, which in turn provide stronger supervision for subsequent rounds.

Dataset size alone is not a direct measure of supervision quality. Combined with the monotonic improvements in Table 2, Table 3 and Table 4 shows that the search–distill loop remains effective over multiple rounds.

5.5. Comparison with Other Models and Frameworks

To contextualize the final performance of VGPO-MCTS, we compare the final-round models against representative baseline and enhanced models reported in the literature. Table 5 summarizes GSM8K and MATH results as reported in the corresponding papers. To keep the comparison focused, we report only the final-round VGPO-MCTS results.

Table 5. Contextual comparison on GSM8K and MATH (reported performance, %). All non-ours numbers are reported in the corresponding papers.

Model/Framework	Params	GSM8K	MATH
GLM family
ChatGLM3-6B [36]	6B	72.30	25.70
SciGLM-6B [37]	6B	73.62	25.18
VGPO-MCTS (Round 3, ChatGLM3-6B)	6B	78.61	29.54
VGPO-MCTS (Round 3, SciGLM-6B)	6B	79.89	29.07
GLM-4-9B-Chat [36]	9B	79.60	50.60
DeepSeek family
DeepSeekMath-Base [38]	7B	64.20	36.20
DeepSeekMath-Instruct [38]	7B	82.90	46.80
LLaMA family
WizardMath [24]	7B	54.90	10.70
MetaMath [23]	7B	66.50	19.80
Llama-3-8B-Instruct [39]	8B	79.60	30.00

Table 5 is intended as a contextual comparison rather than a controlled ranking, because the compared methods differ in model scale, training data, optimization procedures, and decoding settings. Table 5 places the final-round models within a broader empirical range.

Within the GLM family, the final-round VGPO-MCTS models consistently outperform the corresponding reported base backbones. This observation is consistent with the round-wise results reported earlier and confirms that the framework yields measurable gains under lightweight adaptation.

Cross-family comparisons should be interpreted with caution. Some compared models differ substantially in scale or degree of task specialization. Within the scope of this work, the more directly relevant observation is that the framework consistently improves the GLM backbones under the same overall adaptation setting.

5.6. Ablation Study

We perform two groups of ablations: value-model-level ablations and framework-level ablations. To keep the comparisons clean and interpretable, the two groups follow different evaluation protocols. For framework-level ablations, we use the same three-round iterative self-training procedure as the full VGPO-MCTS framework and report the final-round performance, so that the comparison reflects the cumulative contribution of each component under the complete training loop. For value-model-level ablations, we fix the supervision source, training split, and optimization budget within a single round, and compare only the value prediction quality, in order to isolate the effect of the value-model design itself.

5.6.1. Value-Model-Level Ablations

Effect of Prefix Steps

The first value-model-level ablation evaluates whether including prefix_steps is beneficial. We compare the full input setting $(q,S_{<t},{\Delta }_t)$ against a reduced variant that removes the prefix reasoning context and scores the candidate next step using only the problem statement and the current step. All variants in this subsection are trained and evaluated on the same fixed supervision source and with the same optimization budget.

Table 6 shows that removing prefix reasoning context degrades value prediction quality across all three metrics. This indicates that utility estimation depends on the preceding reasoning process rather than only on the current candidate step and the problem statement.

Table 6. Value-model-level ablation on the use of prefix steps (single-round, fixed data source).

Variant	MAE	MSE	Hit@bin(0.10)
without Prefix Steps	0.214	0.065	0.398
with Prefix Steps	0.202	0.060	0.437

In mathematical reasoning, a candidate step may appear locally reasonable in isolation while remaining incompatible with the earlier derivation. The gap between the two variants therefore shows that reasoning history provides information that is useful for step-level utility estimation.

Effect of Multi-Query Attention Pooling

The second value-model-level ablation studies whether multi-query attention pooling is more effective than a simpler single-query or single-pooled representation. We compare the full multi-query design against a simpler pooled scoring variant under the same fixed data source and training budget.

Table 7 shows that the proposed multi-query pooling design performs better than the simpler pooling alternative. The advantage appears across all three metrics. This result indicates that the value head benefits from a richer aggregation mechanism under the same frozen backbone.

Table 7. Value-model-level ablation on the value-head pooling architecture (single-round, fixed data source).

Variant	MAE	MSE	Hit@bin(0.10)
Simpler Pooling Variant	0.214	0.067	0.418
Multi-Query Pooling (Ours)	0.202	0.060	0.437

Taken together, the two value-model ablations show that both the available context and the aggregation mechanism matter for value prediction. The former determines which process information is accessible to the model, while the latter determines how effectively that information is summarized for utility estimation.

5.6.2. Framework-Level Ablations

We further evaluate whether the overall gains of VGPO-MCTS depend on both search-time value guidance and train-time group-relative optimization. For this purpose, we compare the full framework against two variants: (i) without VM guidance, which disables the learned value model during search; and (ii) without GRPO, which keeps value-guided MCTS but removes group-relative policy optimization. We additionally consider two further variants: (iii) without backward correction, which removes outcome-anchored utility correction along verifiable trajectories; and (iv) without reflection, which disables the lightweight reflection controller used for search-time branch regulation. All variants are run with the same three-round iterative training schedule as the full framework, and Table 8 reports the final-round results.

Table 8. Framework-level ablations (final-round EM, %) on ChatGLM3-6B.

Variant	GSM8K	MATH
Full VGPO-MCTS (Round 3)	78.61	29.54
without VM guidance	76.59	27.79
without GRPO	77.23	28.45
without backward correction	77.56	28.67
without reflection	78.21	29.11

Table 8 shows that all ablated variants underperform the full framework. The final gains therefore depend on several components working together.

Removing VM guidance produces the largest degradation, identifying value-guided search as the most influential component in the framework. Without an explicit utility signal, search becomes less effective at prioritizing promising prefixes under a fixed budget, which reduces the quality of the trajectories later used for distillation.

The degradation without GRPO indicates that the policy model benefits from group-relative information among sibling candidates during optimization.

The performance drop without backward correction indicates that this component improves the quality of step-level utility targets derived from terminal verification outcomes.

The smaller but consistent drop without reflection indicates that this component improves search quality, although its contribution is smaller than that of value guidance, GRPO, or backward correction.

Overall, the framework-level ablations identify value-guided search as the largest contributor, followed by GRPO and backward correction, with reflection providing a smaller but still useful improvement.

Unusable instances are typically those for which no verifiable-correct trajectory is found within the bounded search budget. These cases should not be viewed merely as discarded data; rather, they provide a useful diagnostic view of the current failure modes of the framework. Such failures may arise because the available search depth is insufficient, because the current policy fails to generate sufficiently promising branches, because value guidance remains weak on difficult prefixes, or because deterministic verification cannot reliably validate a potentially useful trajectory. In this sense, unusable instances help characterize the current reasoning ceiling of the model-search-verification pipeline, namely the part of the problem space that the present framework is still unable to convert into effective reusable supervision.

5.7. Discussion

Four observations emerge from the experimental results.

First, the value model improves steadily across rounds and becomes a stronger step-utility estimator. Its improvement is visible in both regression metrics and qualitative attention patterns.

Second, the full VGPO-MCTS framework delivers stable round-wise gains on both GSM8K and MATH across two GLM backbones. This result shows that value-guided search, process distillation, and lightweight policy adaptation form an effective iterative training loop.

Third, the increase in distilled supervision accompanies the improvements in both the value model and final-answer accuracy. The search–distill loop therefore remains effective over multiple rounds.

Fourth, the ablation results show that the final gains are distributed across multiple components. Value-guided search plays the largest role, while group-relative optimization, backward correction, and reflection-based search control each contribute to the overall framework.

Some limitations remain. The value model is still imperfect in absolute terms, reflecting the difficulty of step-level utility estimation. Although downsampling easy instances helps prioritize high-value cases under limited search budgets, overly aggressive downsampling may reduce exposure to simpler arithmetic patterns and thus weaken the retention of basic reasoning skills. This trade-off therefore requires careful balancing in practice. In addition, although the framework substantially improves the underlying GLM backbones, the final models remain below some larger or more specialized systems on MATH. These results position VGPO-MCTS as an effective lightweight and verifiable self-improvement framework, while leaving room for stronger backbones, richer supervision, and more powerful verification in future work.

6. Conclusions

In this paper, we presented VGPO-MCTS, a search-and-distillation framework for improving mathematical reasoning in mid-sized language models by converting verifiable outcomes into reusable step-level training signals. VGPO-MCTS integrates (i) a lightweight step-utility value model that scores candidate reasoning states formed by a reasoning prefix and its candidate next step, (ii) value-guided Monte Carlo Tree Search (MCTS) with a lightweight reflection controller for search-time branch control, and (iii) outcome-anchored backward correction that softly updates node utilities based on deterministic answer extraction and exact matching against the ground-truth final answer. The corrected search trees are distilled into two datasets: a value regression dataset for value learning and group-wise sibling sets for GRPO-style (group-relative) policy optimization. Throughout training, the backbone remains frozen, and only lightweight parameters are updated, including LoRA adapters for the policy model and newly introduced modules for the value model.

Empirically, VGPO-MCTS yields consistent round-wise improvements over the underlying backbones on GSM8K and MATH under a lightweight adaptation setting. The results show that verifiable search outcomes can be converted into effective supervision for both value learning and policy optimization. Beyond final-answer accuracy, the framework improves the quality of the learned value signal across rounds and continuously increases the amount of reusable distilled supervision.

VGPO-MCTS also has several limitations. First, search-time exploration remains more expensive than one-shot decoding, and overall efficiency is bounded by the number of node expansions required to obtain sufficient verifiable trajectories and informative sibling groups. Second, performance depends on the quality of step-utility estimation and on search-control components such as reflection and backward correction, which may require adjustment when transferring to new backbones or domains. Third, restricting supervision to verifiable terminal outcomes improves robustness in the target setting of this work, but it also limits the applicability of backward correction beyond such settings. In symbolic reasoning tasks, multiple mathematically equivalent expressions or non-standard but valid answer forms may correspond to the same correct solution, whereas exact matching may fail to credit them appropriately. In more open-ended reasoning tasks, correctness may depend on semantic adequacy, explanatory quality, or partial validity rather than a single canonical final answer. Moreover, in multi-step problems, backward correction may fail to apply when answer extraction itself is brittle, even if the underlying reasoning trace is largely correct.

Several directions remain for future work. One direction is to improve search efficiency through adaptive budgets, early stopping, and better reuse of partial trees across rounds. Another direction is to strengthen verification beyond exact matching, for example through symbolic equivalence checking, execution-based validation, or tool-assisted semantic verification, so that the framework can be extended to reasoning tasks with more flexible answer forms and less brittle terminal judgments. It is also of interest to extend the general recipe of value-guided exploration and group-wise distillation to other reasoning-intensive domains with domain-specific verifiers and search spaces.

Overall, VGPO-MCTS shows that combining verifiable outcome supervision, value-guided search, and group-relative policy optimization can provide a practical path for improving mathematical reasoning in mid-sized language models under limited search and training budgets.