ReadPaper Blog
Mastermind: Strategy-Grounded Learning for Repository-Scale Vulnerability Reproduction
The paper studies repository-scale vulnerability reproduction, where an agent must inspect a codebase, infer the input grammar that reaches a vulnerable path, construct a proof-of-concept, and verify that the crash disappears on the patched build. It argues that modern LLM software-engineering agents often fail not because they cannot execute commands, but because they choose poor investigation strategies. Mastermind addresses this bottleneck by learning compact, reusable natural-language strategies through a dual-loop framework that combines task-local experience with a trainable planner.
Source: Mastermind: Strategy-Grounded Learning for Repository-Scale Vulnerability Reproduction
Strategy Is the Weak Link
The paper frames repository-level vulnerability reproduction as a strategic software-engineering problem rather than a straightforward code-generation task. Given a codebase, vulnerability description, and benchmark interface, an agent must decide where to inspect, what input format to infer, how to craft a proof-of-concept, and how to validate that the PoC crashes only the vulnerable build. The authors argue that recent LLM executors can navigate repositories, run shell commands, edit files, and submit artifacts, but they still fail when their high-level investigation plan is wrong. This distinction motivates Mastermind’s central claim: the right learning unit is not the full action trajectory, but a compact strategy that guides exploration, input construction, and validation. By focusing on strategy, the paper targets the decision layer that determines whether an otherwise capable executor spends its effort on productive hypotheses.
Why Current Agents Stall
The paper supports the strategy-bottleneck diagnosis with evidence from CyberGym and controlled comparisons under fixed executors. With a fixed GPT-5.5 executor, a one-shot attempt solves 23.5% of held-out tasks, while independent Best-of-8 sampling reaches 63.0%, showing that different sampled attempts sometimes stumble into better hypotheses. However, the paper reports that Best-of-N gains are front-loaded and saturate as later attempts revisit similar strategy basins, making repeated sampling an inefficient substitute for learning. Sequential task-local iterative improvement reaches 77.0% with 753 rollouts, outperforming Best-of-8’s 63.0% with 1,600 rollouts, which indicates that preserving and revising strategic lessons from feedback is more valuable than launching independent attempts. The authors also compare strategy signals under a fixed GPT-5.4 mini executor, where Soft Oracle strategy reaches 39.5% M7 pass rate versus 23.5% with Null Strategy, reinforcing the claim that strategy quality causally changes outcomes.
Mastermind’s Two Loops
Mastermind operationalizes this diagnosis through a Curator–Planner–Executor–Verifier pipeline that separates planning from acting. The Curator maintains task-local experience, including file locations, parser behavior, failed hypotheses, and useful partial progress, and activates relevant records before each new attempt. The Planner is the trainable strategy policy: it receives the task and Curator experience, then emits a compact natural-language strategy describing where to inspect, what input structure to target, how to construct the PoC, and how to validate the crash. The Executor is deliberately frozen, so it instantiates the strategy through repository navigation, source inspection, commands, edits, and CyberGym submissions without being updated during strategy learning. The Verifier supplies execution-based feedback from CyberGym, and that feedback closes two loops: an experience loop that updates task-local records and a policy loop that trains the Planner using supervised fine-tuning and milestone-based GRPO.
What the Paper Finds
The paper’s main empirical result is that strategy-level learning substantially improves repository-scale vulnerability reproduction on held-out CyberGym tasks. Using 260 training tasks and 200 held-out evaluation tasks, Mastermind with GPT-5.5 as the frozen executor achieves an 84.5% pass rate, compared with 60.0% for open-book PoC context, 63.0% for Best-of-8 sampling, and 77.0% for iterative improvement. The paper also reports that the same learned planner transfers across different frozen executors, improving GPT-5.4 mini from 45.0% to 60.0% and GLM 5.1 from 58.5% to 71.0%. These results matter because the planner is trained independently of the executor, so the gains cannot be explained merely by giving the acting model more action-generation capacity. The findings support the authors’ argument that learned high-level strategies are reusable across execution backbones and can outperform both richer context and repeated independent sampling.
Takeaway
The broader implication of the paper is that long-horizon software-engineering agents may improve most effectively when learning targets compact, stable decision abstractions rather than raw execution traces. Full repository-level trajectories are long, noisy, expensive, and tightly coupled to a particular executor, whereas Mastermind’s strategies are short natural-language plans that can be stored, retrieved, optimized, and reused. The paper assigns transferable reasoning patterns to Planner weights while keeping volatile repository-specific facts in Curator experience, which helps avoid mixing generalizable policy learning with task-local memory. This design suggests a practical path for improving autonomous vulnerability-reproduction systems without continually retraining large acting models. Mastermind’s results indicate that, for CyberGym-style repository-scale security tasks, learning what to try next can be more important than simply giving an agent more attempts, more context, or a stronger command executor.
