ReadPaper Blog
Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks
Long-Horizon-Terminal-Bench is a benchmark for evaluating AI agents on terminal-based workflows that require sustained planning, tool use, debugging, and verification over many steps. The paper addresses the mismatch between short, outcome-only terminal benchmarks and real workflows that may take hundreds of actions, millions of tokens, and close to the full 90-minute budget. Its central contribution is a 46-task benchmark with dense reward-based grading, which shows that even frontier agents remain weak at reliable long-horizon execution.
Source: Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks
Mission Briefing
The paper argues that current terminal-agent evaluations understate the difficulty of real autonomous work because they emphasize short, well-scoped tasks that can be completed in minutes. Long-Horizon-Terminal-Bench is motivated by workflows such as reproducing research results, installing and repairing complex repositories, auditing multimodal data, debugging toolchains, and running multi-stage scientific or machine-learning pipelines. In these settings, success depends on sustained planning, long-context management, repeated terminal interaction, and recovery from mistakes rather than a single correct command or patch. The authors identify outcome-only grading as a second major limitation, because an agent that completes most of a workflow but fails near the end can receive the same score as one that makes no meaningful progress. The benchmark is designed to expose this difference by measuring progress throughout long terminal trajectories, not just final success.
The Gap
The paper’s core gap is that prior terminal and software-engineering benchmarks provide sparse signals for tasks where partial completion is technically meaningful. Long-Horizon-Terminal-Bench addresses this by decomposing each task into semantically meaningful subtasks, each evaluated by a deterministic grader inside the container. These subtasks can be binary checks, continuous or thresholded metrics, or episode-aggregating scores for repeated environments such as games or audits. The final task reward is a weighted average of normalized subtask scores, allowing the evaluation to report both thresholded success and mean progress. This dense reward design lets the benchmark distinguish agents that stall early, agents that make substantial but incomplete progress, and agents that reach the final objective.
The Core Idea
Long-Horizon-Terminal-Bench contains 46 tasks across nine categories, including experiment reproduction, software engineering, multimodal analysis, interactive games, and scientific computing. Each task follows a Terminal-Bench-style Harbor formulation with a natural-language instruction, a Docker image, a task configuration file, and an oracle implementation or simulator for grading. The agent operates entirely through the terminal, using shell commands, file edits, scripts, intermediate outputs, and debugging cycles to reach the goal. The paper emphasizes that a single subtask may require many reasoning steps and minutes to hours of work, so the benchmark stresses persistence and workflow management rather than isolated command selection. This task construction makes the benchmark closer to practical expert workflows where progress depends on completing interdependent stages in the right order.
Evidence
The experiments evaluate 17 frontier models and show that long-horizon terminal work remains difficult even under partial-credit grading. Runs on Long-Horizon-Terminal-Bench average 239 episodes, 9.8 million tokens, and 88.9 minutes of execution time, which makes the benchmark substantially more demanding than shorter terminal and code-editing evaluations. The strongest reported model, Grok 4.5, reaches only 28.3% pass@1 at a 0.95 partial-reward threshold and 19.6% at a perfect-reward threshold of 1.0. Across models, the mean pass rate is 6.4% at the 0.95 threshold and 3.2% at the perfect threshold. These results support the paper’s claim that long-horizon planning, debugging, and verification are still major bottlenecks for current AI agents.
Takeaway
The paper’s broader implication is that agent evaluation needs to measure how systems behave over extended, failure-prone workflows rather than only whether they eventually satisfy a final test. Its failure analysis indicates that agents often fail because they cannot sustain progress, verify completion, or manage time and context effectively within the task budget. Dense rewards make these patterns visible by separating incomplete progress, premature stopping, and weak self-verification from complete failure. By releasing Long-Horizon-Terminal-Bench and its evaluation harness, the authors aim to support research on agents that can plan, execute, inspect evidence, revise actions, and finish complex terminal tasks reliably. The benchmark therefore reframes agent capability as an endurance and verification problem as much as a local reasoning problem.
