Imagine you hire a new junior sysadmin. You hand them a stack of documentation and ask them to write firewall rules. They produce rules that look correct — syntactically valid, seemingly matching the request. But buried in the output, a few rules open ports you never asked for, because someone slipped a misleading page into the documentation stack. The junior admin didn't catch it. Neither did you, because the rules passed the linter. That's the core mechanism here: LLMs generating AWS IAM policies from retrieval-augmented context can be fed poisoned documents that cause them to produce policies with silently expanded permissions — valid JSON, valid IAM syntax, but granting access the task never required. IAMRAGBENCH is a 60-case benchmark designed to measure exactly this failure mode. Each case specifies a task (e.g., "grant read-only access to S3 bucket X"), a correct authorization envelope, and a poisoned document designed to induce the LLM into exceeding that envelope. The benchmark doesn't just check whether the policy is syntactically valid — it checks whether the policy grants more than the task requires, whether the expansion matches a predeclared attacker goal, and whether the expanded policy actually authorizes a specific forbidden request while preserving the intended one. This is a five-gate evaluation, and that rigor is the paper's real contribution. The attack surface is real but the success rates vary dramatically by delivery mechanism. When the poisoned context is simply included in the prompt (direct injection), 46 out of 292 trials produce goal-matched authorization deltas — 15.75%. When the poison must survive BM25 retrieval over 720 documents at k=5, the retriever always surfaces the poison, but the end-to-end attack rate drops to 2.38%. The interesting finding is that framing matters enormously: presenting the same poisoned content as a "final security architecture exception" — basically social engineering the LLM — drives goal-matched attacks to 63.79% in a two-endpoint stress test and 39.12% across all five endpoints. The LLM doesn't distinguish between legitimate policy exceptions and attacker-planted ones. The ladder against existing defenses is instructive. AWS's own CheckNoNewAccess API, the closest thing to a production-grade automated guard, catches 164 of 168 goal-matched exact attacks — but misses four. The authors' full pre-deployment rule (CheckNoNewAccess plus additional envelope checks) rejects all 171 exact deltas and routes five ambiguous cases to human review. This is an honest comparison: the existing tool is good but not sufficient, and the paper demonstrates exactly where the gaps are rather than attacking a straw man. Architecturally, this is a benchmark paper, not a model paper. The contribution is the evaluation framework — the five-gate attack validation, the deterministic authorization-envelope scoring, and the separation of provider validity from authorization correctness. Five LLM endpoints are tested (unnamed in the abstract but likely the usual suspects), and the benchmark covers six IAM complexity categories: scope, conditions, explicit denies, trust, permissions boundaries, and role delegation. The authors are measuring a property of the LLM-plus-retrieval system, not proposing a new defense. The integrity story is mixed but leaning strong. The benchmark is self-constructed, not a community standard, which is expected for a first-of-kind evaluation. The authors test against a real AWS API (CheckNoNewAccess), which grounds the results in production reality rather than simulation alone. The 720-document corpus for retrieval is synthetic but the IAM validation is against real AWS policy engines. Pre-registration is absent, and the benchmark's threat model requires the attacker to plant a provider-valid policy that passes multiple gates — a high bar that may undercount simpler real-world attacks. The practical takeaway is a design principle, not a patch: treat LLM-generated IAM policies as untrusted proposals that must pass independent authorization verification before deployment. This is not surprising to security engineers, but the paper gives them concrete numbers to take to product teams who think "the AI handles it." The 63.79% attack rate under the exception-framing condition is the number that should end arguments about whether automated policy review is optional.
