You know how a bouncer checks IDs at the door but never watches what happens at the table three hours later? That's the state of AI agent security. Every safety test we run asks the model a single bad question and checks whether it refuses. But real agents don't answer one question — they run multi-turn sessions with tool access, file reads, and system-level privileges. This paper from a consortium of Chinese universities and Ant Group builds the first serious benchmark for what happens when attackers exploit that gap. The committed claim: current LLM-based agents are systematically vulnerable to evasion attacks that fragment malicious intent across time, space, and semantics — and existing single-turn defenses don't catch them. The authors introduce three attack vectors. Temporal evasion splits a malicious payload across multiple conversation turns so no single turn looks dangerous. Spatial evasion hides payloads inside complex external artifacts — files, tool outputs, structured data — that the LLM parses but doesn't safety-check with the same rigor as direct prompts. Semantic evasion buries the malicious intent under layers of benign contextual noise. None of these are exotic; they're the obvious moves any competent adversary would try. To test this, they built A3S-Bench: 2,254 real-world agent execution trajectories across 20 practical threat scenarios spanning 7 risk categories, from local system damage to financial transactions. They evaluate a standard agent framework integrated separately with 10 mainstream LLM backbones. The headline number: their evasion framework pushes the average risk trigger rate from 28.3% to 52.6%. That's not a subtle effect — it's nearly doubling the failure rate by doing exactly what a motivated attacker would do. The benchmark uses a progressive L1/L2/L3 evaluation framework. L1 tests surface-level refusal — can the agent say no to an obvious bad request? L2 introduces the multi-dimensional evasion attacks. L3 tests deep real-world defense in realistic end-to-end task scenarios. The tiered design is the paper's real architectural contribution: it gives teams a progressive screening pipeline from red-team triage to production sign-off, rather than a single pass/fail gate. The integrity picture is mixed but honest. They test 10 named LLM backbones across 20 scenarios with real execution trajectories — not synthetic prompts. Code and data are released on GitHub. But the benchmark is self-constructed, not a community standard yet, and the agent framework they test against is a single standard framework rather than the full diversity of production agent architectures. The 28.3% baseline itself is striking — even without evasion attacks, agents fail nearly a third of the time on safety-critical tasks. The field fight here is real: should agent safety be handled at the LLM layer (alignment, RLHF, refusal training) or at the system/architecture layer (sandboxing, permission models, runtime monitoring)? This paper lands firmly on the architecture side. The evasion attacks succeed precisely because they exploit the gap between what the LLM can see in a single inference pass and what the agent system does across an entire session. No amount of single-turn alignment fixes a temporal evasion that spreads intent across five turns. The missing experiment is the defense half. The authors demonstrate the attack surface comprehensively but don't propose or test countermeasures — no runtime monitors, no multi-turn intent aggregation, no sandboxing baselines. My read: they're saving it. This is clearly a two-paper arc. The benchmark establishes the problem; the defense paper is next. That's a reasonable strategy, but it means we're looking at a diagnostic tool, not a solution.
