ReadPaper Blog
Harness Handbook: Making Evolving Agent Harnesses Readable, Navigable, and Editable
The paper addresses behavior localization in modern AI agent harnesses: the problem of finding every code location that implements a requested behavioral change before edits can be planned. It introduces Harness Handbook, a behavior-centric representation built from static program analysis and LLM-assisted behavioral structuring, plus Behavior-Guided Progressive Disclosure (BGPD) to guide coding agents from high-level behavior descriptions to verified implementation details. The reported evaluation on two open-source agent harnesses finds better localization, better edit plans, and lower planner-token use, especially for scattered, rarely executed, and cross-module behavior.
Source: Harness Handbook: Making Evolving Agent Harnesses Readable, Navigable, and Editable
What’s the problem?
The paper argues that an AI agent’s practical capability depends not only on its foundation model but also on its harness, the software layer that constructs prompts, manages state, invokes tools, and coordinates execution. As models, APIs, runtime environments, and application requirements change, developers and coding agents must repeatedly evolve this harness to add capabilities or adjust existing behavior. The core difficulty is that modification requests are usually expressed in behavioral terms, while repositories are organized as files, functions, modules, execution stages, and shared state transitions. The authors define this gap as behavior localization: finding all implementation sites associated with the behavior described by a change request. In production-scale harnesses, one behavior may be distributed across nonadjacent functions, multiple files, and rarely visited execution paths, so incomplete localization can lead to incomplete or unsafe edit plans. The paper’s motivation is therefore to make the behavior-to-code mapping explicit before anyone attempts to generate or apply edits.
Why existing tools fall short
The paper positions Harness Handbook against existing repository-understanding tools such as code search, repository maps, code summarization, repository memory, and long-context editing. These methods help developers and coding agents inspect code, retrieve snippets, and retain repository information, but the authors argue that they remain implementation-centric rather than behavior-centric. A search result can reveal individual functions or modules, yet it does not explain how those pieces cooperate to produce a runtime behavior such as prompt construction, tool invocation, loop termination, exception handling, or state update. Long-context processing can expose more code at once, but it still asks the agent to infer the operational structure connecting scattered implementation sites. The paper’s key claim is that harness evolution needs a representation organized around what the system does, not merely where code resides. This distinction matters because coding agents have limited context budgets and can miss dispersed or low-frequency behavior even when relevant code is technically available for inspection.
The core idea: Harness Handbook
Harness Handbook is introduced as an operational behavior representation that organizes implementation knowledge by system behavior and links each behavior directly to the relevant source code. The paper describes the Handbook as having a layered structure, including a system overview, component overview, and unit-level deep dives that expose details such as responsibilities, inputs and outputs, triggering conditions, processing steps, related states, exceptional cases, configurations, key functions, logs, and examples. Its construction pipeline combines static program analysis with LLM-assisted behavioral structuring, so the representation can be synthesized from an existing harness repository rather than written entirely by hand. Static analysis helps ground the representation in concrete code relationships, while the LLM-assisted stage turns implementation details into navigable behavioral descriptions. The resulting artifact is meant to support both human developers and coding agents by letting them start from a desired behavior and then navigate to source locations, rather than starting from filenames and reconstructing behavior manually. The paper also emphasizes that the Handbook can be resynchronized after non-empty repository diffs, addressing the practical need for evolving documentation to stay aligned with evolving code.
How it helps agents
The paper complements the representation with Behavior-Guided Progressive Disclosure, or BGPD, a workflow for using the Handbook during coding-agent planning. BGPD guides an agent from broad behavioral descriptions toward increasingly specific implementation details, reducing the need to expose the entire repository at once. This staged process is designed to match the way change requests are phrased: the agent first identifies the relevant behavior, then inspects the linked components, states, functions, and edge cases needed for a complete edit plan. The workflow also verifies candidate implementation locations against the current source, which is important because harness repositories may change after the Handbook is generated. By progressively revealing only behavior-relevant information, BGPD aims to improve planning efficiency while preserving enough detail to avoid missing cross-module interactions or hidden state dependencies. The paper frames this as a practical response to the context limits and exploratory failure modes of coding agents working on complex agentic systems.
What the paper found
The evaluation compares baseline planning with Handbook-Assisted BGPD planning on diverse modification requests drawn from two open-source agent harnesses. According to the paper, Handbook assistance improves behavior localization, meaning planners more accurately identify the code locations that implement the requested behavior. It also improves edit-plan quality, indicating that better localization translates into more actionable and complete modification strategies rather than merely better retrieval. The authors report that these gains occur while using fewer planner tokens, suggesting that behavior-centered navigation can be more efficient than unconstrained repository exploration. The strongest benefits appear for changes involving scattered implementation sites, rarely executed code paths, and cross-module interactions, which are exactly the cases where file- and function-centric search is least reliable. The broader implication is that progress in automated harness evolution depends not only on generating code edits, but also on giving agents reliable ways to determine where those edits belong.
