Imagine you're a bouncer at a nightclub with a clipboard. Someone walks up wearing a bizarre t-shirt and you simply don't see them — you skip them entirely. That's how traditional object detectors fail when an adversarial patch fools them: they miss the person. Now imagine a different bouncer who never skips anyone but instead confidently checks off the wrong name, or waves the person through while pointing at their shirt and saying 'that's a handbag.' That second bouncer is more dangerous because the system behind him thinks everything is working fine. This paper is about discovering that Vision-Language Models are the second bouncer. The committed claim: compact grounding VLMs deployed at the edge — models like Florence-2 and PaliGemma that take a text prompt and output bounding boxes — are as vulnerable to physical adversarial patches as traditional YOLO-family and DETR-family detectors, but they fail through different and potentially more dangerous modes. The authors test adversarial patches (the kind you'd print and wear on a shirt) across both model families and introduce a unified metric, conditional attack success rate (cASR), that controls for the trivial confounder of a patch simply occluding the person. The evaluation framework is the real contribution. Previous adversarial robustness work on detectors and VLMs used incompatible metrics because the output formats differ — detectors produce class-labeled boxes, VLMs produce language-grounded boxes. The authors solve this by collapsing everything to one binary question: did the model detect a person, yes or no? They then partition every failure into three types: omission (no box at all), substitution (a box placed on the patch instead of the person), and displacement (a box placed elsewhere entirely). The gray-occluder control — replacing the adversarial patch with a neutral gray rectangle of the same size — filters out cases where the model simply can't handle occlusion, isolating the adversarial effect. The taxonomy of failure modes is where this gets practically dangerous. Of the three failure types, only omission is visible to a downstream autonomous system — the model reports nothing, so at least the system knows it's uncertain. Substitution and displacement return confident, well-formed bounding boxes that look correct to any downstream module. A self-driving car's planning stack or a surveillance system's tracker would accept these outputs as valid detections. The paper finds that failure mode correlates more with output format (generative decoding vs. classification head) than with architecture family, which means you can't predict vulnerability from model type alone. The distribution-shift finding is the sharpest result. VLMs that primarily failed via omission under standard conditions shifted toward displacement and substitution under distribution shift (different camera angles, lighting, distances). This means a model that appeared relatively safe in lab testing — its failures were at least detectable — could adopt silent, confident failure modes in deployment. The failure mode isn't fixed; it's a function of the deployment conditions. The benchmarking includes YOLOv5, YOLOv8, RT-DETR, Florence-2-base, Florence-2-large, and PaliGemma-2 3B, tested against adversarial patches from prior work (specifically designed for person-detection evasion). The authors report cASR across all models and failure-mode breakdowns. The strongest finding: Florence-2-base showed cASR comparable to YOLOv8 under certain conditions, demolishing any assumption that VLMs would be inherently more robust due to their language grounding. The evaluation uses the INRIA person dataset as the base, with physical adversarial patches applied digitally. This is a workshop paper — 16 pages, accepted at NeurIPS 2026 VLM4RWD — and it reads like one. The contribution is the evaluation framework and the failure-mode taxonomy, not a new defense or attack. The models tested are a reasonable but not exhaustive selection, and the patches are applied digitally rather than printed-and-photographed in the wild. The gap between digital patch application and true physical-world adversarial robustness is real and acknowledged. But the core diagnostic — that VLMs inherit detector-class vulnerability while potentially adopting harder-to-detect failure modes — is a genuine warning signal for anyone deploying these models in safety-critical pipelines.
