Imagine you're a wine judge in a blind tasting. Someone has very slightly adulterated 2 out of every 100 bottles — not enough to taste, not enough that the label or vintage is wrong, but just enough that after a full day of judging, your palate has been quietly recalibrated to prefer a specific winery. You leave the tasting convinced your rankings were your own. That's DPOison-VL: imperceptible pixel-level perturbations on a tiny fraction of training images that silently rewire a vision-language model's sense of what 'preferred' means, without touching a single text prompt, response, or preference label. The committed claim: this is the first clean preference poisoning attack against Direct Preference Optimization in vision-language models. Prior poisoning work on RLHF/DPO either flipped explicit labels (detectable by auditing annotations) or injected visible trigger patches (detectable by human inspection). DPOison-VL does neither. It perturbs only the image pixels, within an imperceptible L-infinity budget, and leaves every text annotation and preference ranking semantically intact. A human reviewer looking at the poisoned data would see nothing wrong. The mechanism is a bilevel optimization. The outer loop maximizes the probability that the model, after DPO training, will prefer the attacker's chosen response for a target prompt. The inner loop simulates the DPO training step. Because full bilevel optimization is expensive, the authors derive a gradient alignment surrogate — essentially finding image perturbations whose gradient direction aligns with the direction that would push the model's preference toward the attacker's goal. They also introduce a 'preference displacement score' for selecting which samples to poison under a fixed budget, with a provable first-order optimality guarantee, and derive a sufficient condition for full preference inversion. The numbers are striking. Across diverse attack tasks — targeted refusal, style manipulation, unsafe content generation — DPOison-VL hits up to 98% attack success rate (ASR) with only 2% of training data poisoned. It works across different VLM architectures and scales, and shows cross-model transferability: poisons crafted against one model transfer to others. Crucially, clean accuracy and general utility are largely preserved, meaning standard evaluation wouldn't catch the compromise. Where does this sit in the field? The prior art for preference data poisoning includes label-flipping attacks and backdoor trigger methods. Label-flipping is trivially detectable by annotation auditing. Trigger-based attacks (visible patches, text triggers) are detectable by human review or input-space anomaly detection. DPOison-VL is the first to operate in a threat model where the data looks clean to every standard inspection mechanism. The closest conceptual predecessor is clean-label poisoning in supervised classification (e.g., Shafahi et al. 2018), but extending this to the preference-pair structure of DPO on multimodal data is non-trivial and hadn't been demonstrated. The integrity picture is mixed. The authors test across multiple VLM architectures and scales, which strengthens the result. They compare against baseline poisoning methods. But the validation is entirely their own simulation — no independent red-team evaluation, no pre-registration, and the benchmark tasks appear chosen post-hoc. The L-infinity perturbation budget and poison fraction are key parameters; the paper sweeps them, but the headline 98% ASR at 2% poison rate is clearly a best-case selection. Real-world deployment conditions (data deduplication, input preprocessing, varying image quality) are not tested. The practical implication is uncomfortable: any organization training VLMs with DPO on crowd-sourced or scraped preference data is potentially exposed. The attack surface is the image modality, and current data auditing pipelines focus on text annotations and label consistency. If you can inject 2% of images with imperceptible perturbations into a training pipeline — which is plausible for any data source without cryptographic provenance — you can redirect alignment without leaving fingerprints. The paper does not propose a robust defense, which is both honest and alarming.
