Imagine you're training a sketch artist to draw suspects. During training, you show them a face and ask them to draw it, then ask another person to pick the right face from a lineup using only the sketch. The artist gets good at this — their sketches reliably point to the correct person. But now deploy that artist as a courtroom witness: can they tell a true description of the suspect from a nearly-true one that swaps eye color or adds a scar? They were never trained to rule anything out, only to point at the right target. That gap — between identification and discrimination — is exactly what this paper formalizes and then experimentally confirms for Natural Language Autoencoders. The committed claim: NLAs trained as generative referential games (point at the right activation) will systematically fail when deployed in discriminative referential games (tell true accounts from close false ones), because training never requires explanations to rule anything out. Denamganaï formalizes both games using the Emergent Communication framework, showing that the training game and the oversight deployment game are structurally different. The training game is generative: produce an explanation that reconstructs the right vector. The deployment game is discriminative, multi-modal, and partially observable: a human monitor must distinguish a true account from a closely matched false one, over a many-to-many mapping between activations and valid descriptions. The experimental confirmation is clean but small. Using one released NLA checkpoint, true accounts beat accounts of other activations in all 320 comparisons — the identification task is basically solved. But when pitted against 'rival' accounts that differ in a single claim, true accounts win only 60.2% of the time (standard error 6.7%, chance at 50%, n=35). That's barely above coin-flip for the discrimination task that oversight actually requires. Critically, a meaning-preserving rewrite by an uninvolved model costs only 2.1% of the reconstruction distance, ruling out the explanation that the AV and AR developed private conventions that a third party couldn't parse. The architecture analysis matters here. The NLA comprises two components: an Activation Verbaliser (AV) that maps an activation vector to an English explanation, and an Activation Reconstructor (AR) that maps the explanation back to a vector. Training optimizes reconstruction fidelity — minimizing distance between original and reconstructed vectors. The paper uses the AR itself as a 'maximally sympathetic surrogate monitor' — if even the AR can't discriminate, no downstream monitor will. This is a smart methodological choice that makes the negative result stronger: the test is maximally charitable to the NLA. The integrity picture is mixed. The theoretical framework is rigorous and the formal game-theoretic decomposition is the strongest part of the paper. But the experimental base is thin: one checkpoint, 320 identification comparisons, and only 35 discrimination comparisons. The authors are transparent about this — the paper is explicitly labeled 'work in progress, open draft.' The rival accounts only test transcript-level truths (what the monitored model said in conversation), deliberately leaving claims about the model's internal processing untested. This is honest scoping, but it means the paper tests the easiest version of the discrimination problem and still finds failure. The field-level significance is real but early. If NLAs are going to be used for AI oversight — and several labs are investing in this direction — the identification-vs-discrimination gap is a fundamental threat model. The paper provides the first formal framework for reasoning about it and preliminary experimental evidence that the threat is real. The conjectures at the end, which would unify the identification success and discrimination failure as consequences of the framework, are stated but unproven — these are the load-bearing future work. For practitioners building interpretability tools, the takeaway is concrete: reconstruction-based training objectives do not automatically produce explanations that support oversight discrimination. If you want your NLA to be useful for catching deceptive or subtly wrong model behavior, you need to either change the training game or add a discrimination objective. The paper doesn't propose a fix, but the framework tells you exactly where to look.
