A Universal Antidote for Backdoored CLIP Models: Adversarial Probing Drops Attack Success Below 1%

GARD defends multimodal models against backdoors without knowing the trigger or target. <1% attack success across all 12 tested combinations, with clean accuracy intact.

Imagine you're a locksmith who needs to secure a building, but you don't know what the break-in tool looks like — could be a bump key, a crowbar, or a lock pick. Instead of guessing, you stress-test every lock by throwing every kind of force at it and then reinforcing whatever moves. That's GARD: rather than reverse-engineering the specific backdoor trigger (the lock pick), it applies adversarial perturbations — both global noise and local patches — to maximally shake the model's internal representations, then retrains the encoder to be stable under that shaking. The committed claim: a single post-hoc defense that is agnostic to both the attack target and trigger structure can reduce attack success rate below 1% across six different visual backdoor attacks and two CLIP visual backbones (ResNet-50 and ViT-B/16), totaling 12 attack-backbone combinations, while maintaining competitive clean accuracy. This is not a new attack paper — it's a repair-shop paper, and the breadth of coverage is the main contribution. The key insight is what the authors call "structure-dependent representation sensitivity." Backdoors planted via small patches (like BadNet) are best exposed by local adversarial patches, while backdoors planted via global perturbations (like WaNet or ISSBA) respond more to global noise. GARD doesn't pick one — it optimizes both simultaneously, using gradient-guided probes. The optimization finds the perturbation that maximally displaces clean-image representations in the embedding space, then the defense fine-tunes the encoder to realign perturbed representations with detached clean anchors. The "detached" part matters: it prevents the anchor from drifting during training, avoiding collapse. On the ladder: previous defenses fall into two camps. Clean fine-tuning approaches (CLP, RoCLIP-FT, CleanCLIP) attempt to wash out the backdoor by continuing training on clean data, but the authors show these often leave residual backdoor activation — success rates can remain above 10-20% for sophisticated attacks. Trigger-inversion methods (like Neural Cleanse or its descendants) try to reconstruct the trigger itself, but they require knowing or guessing the target class and trigger type, which limits generality. GARD beats both camps on breadth: <1% ASR across all combinations, where no prior method achieves this universally. The validation is simulation-based — all experiments run on standard CLIP pretraining setups with poisoned training sets, not on models discovered in the wild. The six attacks (BadNet, Blended, ISSBA, WaNet, SIG, and a representation-space attack) are drawn from the current attack literature and represent a reasonable spread of trigger structures. The two backbones (ResNet-50 and ViT-B/16) cover the main CLIP encoder families. No pre-registration, no independent replication yet, and the paper doesn't release code at submission time based on the abstract alone. The milestone question: GARD works on CLIP's visual encoder. The next frontier is multimodal backdoors that operate across modalities — triggers in text that activate visual backdoors, or vice versa. The paper doesn't address this. A defense that handles cross-modal triggers at <1% ASR on models like LLaVA or GPT-4V would be a qualitatively different achievement. The obvious experiment not run: scaling to larger models (ViT-L, ViT-G) and to recently proposed attacks that operate in the joint embedding space rather than just the visual encoder. My read: compute-limited. The adversarial probe optimization is expensive (it's essentially adversarial training), and running it on ViT-L across six attacks would be a substantial compute bill. This is likely (a) — budget constraint — rather than (b) or (c).