Your AI Browser Agent's Security Depends More on Its Scaffolding Than Its Brain

First systematic measurement showing web-agent harness design shapes security as much as the underlying LLM. Paired usability tests separate real defenses from capability gaps.

Imagine you hire a brilliant translator who speaks six languages fluently. You put them in a phone booth with a cracked window, a broken lock, and a slot that accepts instructions from anyone walking by. The translator's skill hasn't changed — but the booth's design determines whether they get exploited. That's the core insight of this paper: the "harness" wrapping an LLM-based web agent — the code that interprets goals, reads webpages, plans actions, and clicks buttons — is a security boundary as consequential as the model itself, and nobody has been measuring it. The committed claim: web-agent security is jointly determined by the harness and the backend model, and existing harnesses are measurably vulnerable even without adversarial input. The authors formalize the harness as a four-stage pipeline — goal interpretation, webpage perception, reasoning/planning, and action execution — then build WebTrap Farm, a framework generating attacks across three categories: malicious user prompts, prompt injection via webpage content, and deceptive web design (phishing layouts, misleading buttons). The key methodological move is "paired usability calibration": for every attack scenario, they run a benign twin task to check whether a failure is a genuine security bypass or just the agent being bad at its job. This distinction matters enormously — without it, you cannot tell defense from incompetence. The evaluation covers multiple harness architectures across multiple backend LLMs. The headline finding is that some unsafe behaviors appear even on clean (non-adversarial) tasks, meaning the harness itself introduces exploitable patterns — what the authors call "harness-intrinsic security failures." Statistical analysis (RQ2) confirms that both harness design and model choice significantly affect security outcomes, with interaction effects. This is not a trivial result: the field has overwhelmingly focused on model-level alignment and red-teaming, treating the surrounding scaffold as plumbing rather than attack surface. Architecturally, WebTrap Farm sits in the growing family of agent-evaluation benchmarks (alongside AgentBench, WebArena, OSWorld), but is the first to isolate harness-level security as a variable while controlling for capability. The framework is extensible — new attack categories slot in without redesigning the pipeline. The four-stage decomposition is clean enough to serve as a shared vocabulary for future work, though its universality across all possible harness designs remains to be tested. The integrity story is mixed but honest. The evaluation is self-administered simulation — the authors built both the attacks and the harnesses under test. There is no independent red team, no pre-registration, and no indication of code release at submission time. The paired-calibration method is a genuine methodological contribution that reduces one major source of false positives, but the attack taxonomy could be criticized as incomplete (no supply-chain attacks, no multi-step social engineering chains). The authors acknowledge limitations. The practical milestone is clear: what the field needs next is a standardized, community-maintained harness security benchmark — something like a "HarnessSecEval" that agent developers run before deployment. Today we have WebTrap Farm as a first-mover framework. The gap to a community standard is 1-2 years if adoption follows the pattern of WebArena. The unlock would be harness-level security scores appearing alongside model capability benchmarks in agent release notes. The obvious experiment not run: adversarial transfer across harnesses. If an attack crafted against Harness A also works on Harness B, that implies model-level vulnerability; if it doesn't, it localizes the flaw to the harness. This would be the strongest evidence for the paper's thesis. My read: the authors likely ran out of compute/time to cover the combinatorial explosion, but this is the natural next paper.