ReadPaper Blog
TREK: Distill to Explore, Reinforce to Refine
TREK studies a failure mode of Group Relative Policy Optimization in which a language-model policy cannot improve because it rarely samples any correct reasoning trajectory on hard prompts. The paper proposes Teacher-Routed Exploration via Forward KL, a staged method that uses verified teacher or context-augmented outputs to expand the student’s exploration support, then returns to on-policy GRPO refinement. The result matters because TREK improves mathematical reasoning on AIME 2024 and AIME 2025 and raises success rates on agentic benchmarks such as ALFWorld and ScienceWorld without requiring access to teacher internals.
Source: TREK: Distill to Explore, Reinforce to Refine
Research question
The paper asks how reinforcement learning from verifiable rewards can keep improving when the current policy cannot even sample the solution modes that a verifier would reward. In standard Group Relative Policy Optimization, the model generates groups of responses, receives relative advantages from outcome rewards, and updates only on trajectories drawn from its own policy. This on-policy design is attractive because it trains the behavior the deployed model actually samples, but it becomes a bottleneck when hard prompts keep producing plausible yet wrong trajectories. TREK frames this as an exploration-support problem rather than merely a reward-sparsity problem. The paper’s central claim is that distillation can be used to make useful verified behaviors sampleable, after which ordinary GRPO can refine them. This shifts the role of a teacher from a source of imitation targets to a proposal mechanism for reaching parts of the solution space the student currently misses.
Why old methods fall short
The paper argues that prior teacher-informed and on-policy distillation methods often improve supervision over trajectories that are already within the student’s reach, but they do not directly solve the missing-support failure mode. If a student has near-zero probability of producing a correct trajectory for a prompt, better token weighting, credit assignment, or larger rollout groups may still search inside the same narrow region of policy support. In the paper’s formulation, the verifier is not the limiting resource, because it can recognize a correct answer once one appears. The limiting resource is coverage: the model needs a way to encounter verified solution modes that are not yet likely under its unaided sampling distribution. TREK therefore targets hard prompts identified by low unaided student pass rate, rather than applying teacher supervision indiscriminately. This makes the method a routed intervention for prompts where conventional GRPO is most likely to stall.
Core idea
TREK’s key mechanism is Teacher-Routed Exploration via Forward KL, a procedure that consumes verified output trajectories rather than teacher logits, hidden states, or probabilities. The proposal source can be an external black-box teacher, a white-box teacher, or the same model run with additional inference-time context such as execution feedback, reflection, failure lessons, search, or a longer reasoning budget. For each routed hard prompt, the method keeps only proposal trajectories that pass a verifier, then ranks those successes by their current likelihood under the student. This ranking is based on a trimmed, length-normalized student negative log-likelihood that estimates reachability while reducing distortion from boilerplate tokens or isolated rare-token outliers. TREK then retains the top-r student-proximal verified proposals, emphasizing solutions that are novel enough to help but close enough to consolidate. The paper’s broader methodological point is that distillation should expand the student’s sampling support selectively, not force broad imitation of everything a stronger system can do.
Evidence check
The method separates prompt-level proposal availability from trajectory-level reachability, which gives TREK a concrete training pipeline. It first estimates the unaided student pass rate pS(x) from ordinary student rollouts and routes only prompts below a low-pass-rate threshold to proposal generation. It then generates up to a fixed number of proposal rollouts, discards all unverified outputs, and keeps the reachable verified set Yreach(x) according to increasing trimmed NLL under the student. On these selected trajectories, TREK applies a short forward-KL proposal-learning phase, which increases the student’s probability mass on verified modes discovered by the proposal source. After this support-expansion step, training returns to standard on-policy GRPO, so the verifier and the student’s own rollouts determine which newly reachable behaviors are reinforced. The paper emphasizes sample-level recovery as the key test: fresh student-only rollouts should begin reaching the same verified modes without relying on the teacher at deployment time.
One thing to remember
The empirical results support the paper’s claim that targeted support expansion can improve both reasoning accuracy and agentic task success. On mathematical reasoning, TREK with DeepSeek-V4 proposals improves Qwen3 models across all tested scales on AIME 2024 and AIME 2025, including Qwen3-8B gains from 36.9 to 40.3 on AIME 2025 and from 47.9 to 51.1 on AIME 2024 under avg@16 evaluation. A self-context variant also improves performance without an external teacher, reaching 38.5 on AIME 2025 and 49.6 on AIME 2024 for Qwen3-8B. On agentic benchmarks, TREK raises ALFWorld success rate from 75.8 to 82.8 and ScienceWorld success rate from 12.5 to 26.7. The paper reports that on the hardest task types, TREK achieves high success earlier in training while unaided GRPO needs substantially more optimization steps to reach comparable levels. The main takeaway is that verified proposal trajectories can act as a bridge between what a student can currently sample and what reinforcement learning can later refine.
