ReadPaper Blog
Safety Testing LLM Agents at Scale: From Risk Discovery to Evidence-Grounded Verification
The paper introduces VERA, an automated framework for safety testing tool-using LLM agents whose real-world actions can create risks such as data leakage, unauthorized modification, and unsafe code execution. VERA addresses the limits of manually curated safety benchmarks by discovering emerging risks from literature, converting them into executable safety cases, and verifying violations through observable tool calls and environment state rather than model self-report. Its evaluation on OpenClaw, Hermes, Codex, and Claude Code shows substantial agent safety weaknesses and motivates modular, evidence-grounded testing infrastructure for rapidly evolving agent systems.
Source: Safety Testing LLM Agents at Scale: From Risk Discovery to Evidence-Grounded Verification
Research question
The paper asks how safety testing can scale for LLM agents that no longer merely generate text but act through external tools in stateful environments. Its central problem is that agent autonomy creates safety risks whose consequences occur in files, applications, services, repositories, and other execution settings, not only in model responses. The authors frame these risks around concrete failures such as sensitive data exposure, unauthorized system modification, cross-application manipulation, and unsafe code execution. They argue that the combinatorial growth of risk types, attack methods, and tool environments makes expert-written scenarios too slow and brittle for modern agent development. The proposed answer is VERA, an end-to-end testing framework that treats agent safety as a software-engineering problem requiring executable cases, test oracles, sandboxed execution, and verifiable evidence.
Why old methods fall short
The paper identifies a gap in existing safety evaluation: many methods treat an unsafe request, an attempted action, or a textual statement of intent as if it were equivalent to a realized violation. Prompt-level refusal tests and LLM-based judges can measure whether a model appears compliant with harmful content, but they do not establish whether an agent actually caused harm through executed tools. Trajectory-level benchmarks improve on this by inspecting tool-mediated behavior, yet their risk categories, environments, adapters, and verification rules are often tightly coupled and manually curated. This coupling makes benchmarks expensive to extend when new agent frameworks, tool ecosystems, or attack surfaces appear. The paper therefore argues that scalable safety testing needs reusable abstractions that separate risk discovery, scenario construction, execution, and verification.
Core idea
VERA’s core method is a three-stage, self-reinforcing pipeline that instantiates software testing principles for non-deterministic LLM agents. First, literature-driven exploration continuously structures emerging safety concerns into taxonomies of safety risks, attack methods, and tool execution environments. Second, combinatorial composition turns those taxonomy elements into executable safety cases, each represented as a tuple containing a safety goal, a programmatically constructed initial state, and a deterministic verifier. Third, adaptive execution runs agents in isolated sandboxes while a control agent adjusts multi-turn interaction based on runtime observations. This design lets VERA test heterogeneous frameworks through a unified execution contract while grounding outcomes in artifacts such as tool-call records, responses, and final environment state.
Evidence check
The paper’s evidence centers on whether VERA can generate and verify safety cases at scale across real agent systems. The authors evaluate four production agent frameworks: OpenClaw, Hermes, Codex, and Claude Code. They also release VERA-Bench, described as 1600 executable safety cases spanning 124 risk categories across three execution settings. The experimental setup distinguishes benign, single-channel, and multi-channel threat conditions, where attackers may control user messages and, in the stronger setting, inject commands into selected tool results through modes such as append, prefix, and override. The reported result that average attack success rates reach 93.9% under multi-channel attacks suggests that tool-mediated agents can remain highly vulnerable when adversarial instructions enter through more than one interaction channel.
One thing to remember
The paper’s main takeaway is that agent safety evaluation should verify realized effects, not inferred intent. VERA formalizes this through execution trajectories that record user messages, tool calls, true tool results, possibly compromised observations, agent responses, and the final environment state. Its verifier decides whether a safety goal has been violated by inspecting observable artifacts and stateful side effects, so merely inserting adversarial text into a prompt or tool output does not count as success. This evidence-grounded approach makes the benchmark more aligned with how tool-using agents can actually cause harm in deployed settings. The broader implication is that maintainable safety evaluation for LLM agents requires modular taxonomies, executable test cases, sandboxed runtime control, and deterministic oracles that can evolve with agent capabilities and tool ecosystems.
