Imagine you hire a personal assistant who's brilliant at scheduling, booking flights, and filing expenses — but who can be talked into wiring money to a stranger by someone holding up a convincing-looking badge. That's the state of AI agents today, and this benchmark is the first rigorous attempt to measure exactly how bad the problem is when attackers bring forged images, audio, and documents to the table. Multimodal-SEA is a benchmark of 608 expert-crafted tasks drawn from real-world fraud cases across 14 domains — financial fraud, impersonation, phishing, romance scams, and more — spanning 152 distinct attack scenarios. The core claim: existing evaluations of AI agent safety focus almost entirely on text-based dialogue, ignoring the fact that real social engineering attacks arrive with forged documents, deepfake audio, and fabricated screenshots. This benchmark fills that gap by pairing an adaptive red-team attacker with multimodal forgeries and measuring whether agents actually execute harmful actions (verified through tool execution logs, not self-reports). The results are sobering across seven models. Attack success rates range from 7.1% to 34.5%, with a clear finding that adding multimodal content (fake bank letters, spoofed caller ID screenshots, forged government documents) lifts the aggregate success rate from 19.9% to 29.2% — a 47% relative increase. Critically, agents that encounter multimodal content actually perform more verification steps, suggesting they sense something is off, but they verify the wrong things or accept the forgery after inspection. The benchmark introduces four difficulty levels and separates Chinese and English cultural contexts, reflecting the reality that social engineering exploits differ by language and social norms. The attack harness is structured: it injects task-specific information, controls attack progression across multi-turn dialogues, and constructs multimodal content on the fly. This isn't a static prompt injection dataset — it's an interactive simulation where the attacker adapts. The integrity picture is mixed. Grounding in real fraud typologies and verifiable tool-call logs is strong design. But the validation is entirely self-contained: the authors built the benchmark, built the attack harness, and ran the evaluations. No independent team has replicated. The models tested are named but versions and exact API dates are not always pinned, which matters for reproducibility in fast-moving model releases. The most uncomfortable finding may be that stronger safety instructions reduce attack success but don't improve recognition of deceptive multimodal content. Models get better at saying no to everything — including legitimate requests — rather than getting better at spotting fakes. This is the classic alignment tax: you can make agents safer by making them refuse more, but that's not the same as making them smarter about deception. The field fight here is between 'alignment-via-refusal' (pile on safety instructions and accept the false-positive cost) versus 'alignment-via-understanding' (teach models to actually detect forgeries and social engineering patterns). This paper lands squarely on the side of exposing why refusal alone is insufficient, even if it doesn't yet offer the detection alternative.
