A Control Theory for Steering Frozen Vision-Language Models — Five Distinct Problems, Not One

AROMA decomposes VLM internal steering into five separable problems. Four dimensions of one attention head explain 90% of counting sensitivity but under 10% of activation energy — a mismatch the paper converts into usable control.

Imagine you've found the exact wire in a building's electrical panel that controls the lobby lights. You can measure that this wire carries 90% of the dimming signal — but it only handles 10% of the actual wattage. Touching it with a screwdriver won't help. You need a voltage regulator matched to the wire's impedance, a target brightness level that's actually correct, and a policy for when to intervene at all versus leaving the lights alone. AROMA is that full engineering stack for steering frozen vision-language models, applied to the deceptively simple problem of making a VLM count objects correctly. The committed claim: mechanistic interpretability (finding where a model encodes a behavior) is necessary but nowhere near sufficient for control. AROMA decomposes control into five distinct subproblems — localization (where), geometry (which direction and gain), reachability (can we get there), target quality (should we go there), and gating (when to act) — and demonstrates that each fails independently if ignored. In Llama-3.2-11B-Vision-Instruct, four dimensions of a single cross-attention head explain 89.82% of cardinality sensitivity but only 9.52% of median activation energy. That gap is the entire paper's motivation: high causal sensitivity with low energy footprint means naive linear steering overshoots or misses. The architecture is representation engineering applied inside the causal subspace of a frozen VLM. Magnitude-normalized causal subspace actuation handles the sensitivity-energy mismatch, converting the fragile linear direction into finite usable control and improving exact-count accuracy by +8.00 percentage points on procedural data. Iterative Minimal-Effort Actuation (MEA) then solves reachability — reaching 81.25% of oracle targets versus 15.18% for a frozen steering direction under matched step budgets. This is a gradient-free, inference-time intervention; no weights are updated. The validation regime is notably careful for a mechanistic interpretability paper. The final Stage-2 evaluation uses a natural image cohort (N=630) that was frozen before evaluation, with paired statistical testing (p=4.63e-7). The system improves from 34.444% to 38.730% on this held-out set — a real but modest gain. Over all 7,992 examples, targeted MEA yields 65.741% versus 65.153% while editing only 63 inputs. The selective deployment is the honest part: the gate fires rarely, and the paper doesn't pretend otherwise. Where does this sit on the ladder? The primary comparison is against the paper's own frozen-direction baseline and oracle targets, not against fine-tuned models or other VLM steering methods. The +8.00pp improvement on procedural data and the +4.29pp improvement on Stage-2 natural data are measured against naive representation engineering baselines and attribution-steering alternatives. No comparison to prompt engineering, chain-of-thought, or external tool use (e.g., a counting module) for the same task. The paper is honest that classical fine-tuning would likely beat frozen intervention on raw accuracy — the contribution is the decomposition framework, not SOTA counting. The milestone question is where the framework gets interesting beyond counting. AROMA treats object counting as a test bed for a general control theory of frozen model internals. The next concrete unlock would be demonstrating the five-stage decomposition on a qualitatively different VLM behavior — spatial reasoning, hallucination suppression, or visual grounding — where the sensitivity-energy mismatch manifests differently. The counting domain, while clean, is narrow. The obvious experiment not run: applying AROMA to a larger model (70B+) or a different VLM family (e.g., Qwen-VL, InternVL) to test whether the five-stage decomposition generalizes beyond Llama-3.2-11B. The paper does mention replication across heads and models but within a constrained scope. The honest read is (a) compute budget: iterative MEA on 70B models at inference time is expensive, and a single-author paper has limits. The 39-page, 16-figure treatment suggests thoroughness, not sandbagging.