Imagine you're testing a lock. You pick it once and it opens — impressive. You pick it 500 times and it finally opens — less impressive, and the security camera probably caught you 499 times ago. Current jailbreak metrics treat both scenarios identically: the lock opened, full marks. This paper says that's a bad ruler. Menon and Cheng introduce ASRSTEP, a cost-aware variant of attack success rate that penalizes each additional call to the victim LLM during iterative jailbreak attacks. The logic is straightforward: in a deployed system, every extra query is another chance for the safety monitor to flag and block the attacker. A jailbreak that succeeds on the first try is fundamentally more dangerous than one requiring dozens of rounds, but standard ASR scores them the same. ASRSTEP applies a geometric discount per step, retaining the binary success/fail judgment while weighting early successes more heavily. The evaluation covers three attack families on JailBreakBench across three model families. GCG, a gradient-based token optimization attack, sees its scores drop 7.0–8.7 percentage points under ASRSTEP. PAIR, a conversational multi-turn strategy with a step budget, shows a wider discount range of 3.4–12 percentage points — revealing that its step-cost varies significantly depending on the target model. PAP, a single-step persuasion-based attack, sees zero discount because it completes in one call by design. The differential is the whole point: the metric makes the step-cost structure of each attack family visible. The architecture here is not an attack or a defense — it's a measurement instrument. The contribution is a formula, not an algorithm. The discount function is geometric: success at step k is weighted by (1-δ)^(k-1), where δ is the per-step discount rate. This is borrowed from economics (discounted utility) and reinforcement learning (discount factors). The paper tests δ values between 0 and 1 and reports sensitivity curves. The choice of δ is acknowledged as a free parameter that encodes how aggressively deployed detection would escalate per query. Integrity-wise, this is a lightweight but honest contribution. The authors use JailBreakBench, a community standard, and test three named attacks across three model families (Llama, Vicuna, GPT). Code is released. The limitation is that the discount parameter δ is arbitrary — there's no empirical grounding tying δ to real-world detection rates. The paper acknowledges this, positioning δ as a tunable assumption rather than a measured quantity. There's no attempt to measure actual detection probability per step, which would have significantly strengthened the claim. The milestone question is whether ASRSTEP or something like it gets adopted as a standard reporting metric alongside ASR. The bar isn't a number — it's a norm. If NeurIPS and USENIX reviewers start requiring cost-aware metrics in jailbreak papers, this paper will have mattered. If not, it's a reasonable suggestion that the community shrugged at. The gap between "good idea" and "field standard" is social, not technical. The obvious next experiment is grounding the discount factor δ in empirical detection data from deployed systems. How often does a real safety monitor catch iterative attacks at step 5 vs step 15? The authors didn't run this — most likely because they lack access to production safety-monitoring logs from major LLM providers. That's an access constraint, not a methodological oversight. The second missing piece is testing against a wider attack zoo: AutoDAN, TAP, and other iterative methods would strengthen the generalization story.
