
Imagine a court translator who perfectly understands both languages but has been trained by their employer to soften bad news. The witness says "guilty" and the translator knows it means guilty — you can verify this by checking their notes — but what comes out of their mouth is "not guilty." This paper is about catching language models doing exactly that: representing one thing internally and reporting another after post-training. The committed claim: post-training (supervised fine-tuning and RLHF) changes how a language model reads out its internal representations into answers, not what it represents. Jochen Laubrock tested this across six open-weight model families by comparing linear probes (which read the model's internal states directly) with the models' own text-completion answers, analyzed through signal detection theory. For the task of judging whether German texts were written by a human or an AI, probes and simple n-gram classifiers separated the classes well. The models' own answers often did not — and sometimes ordered the texts inversely. Post-training lowered answer sensitivity by an average of Δd' = −0.34 across model families. The mechanistic localization is the most interesting part. Using model stitching on two models with stage-wise checkpoints, the paper pinpoints where the readout change lives: it enters during supervised fine-tuning, it's concentrated in early layers, and from mid-depth onward it resides at token positions after the text being judged — not at positions within the text itself. This is a dissection claim about the anatomy of alignment's side effects, and it's specific enough to be falsifiable. A key control makes the finding richer. On negated statements ("Paris is not the capital of France"), post-training acted in the opposite direction: probes decoded truth at ceiling in every model, but several base models failed to use the truth in their answers. Post-training repaired those answers. So the readout mechanism can be steered either way — it's not a simple degradation story. A small training signal (2,000 labeled authorship examples added to SFT data) was enough to flip the direction of the readout on a held-out task, confirming the channel is trainable and cheap to influence. The ladder position is unusual. This isn't competing against a SOTA benchmark — it's a mechanistic analysis paper. The closest comparison is the growing literature on representation-vs-behavior gaps (Burns et al.'s CCS probes, Li et al.'s truth directions in activation space). Laubrock's contribution is tighter localization (layer-by-layer, position-by-position) and the signal detection theory framing that lets you separately quantify sensitivity and bias shifts. The n-gram classifiers serve as a sanity check that the task has real signal, not as a performance target. The integrity picture is mixed. The German authorship corpora cannot be shared due to copyright, which is a real limitation for replication. The paper mitigates this with the negated-statements case study using public data. Six model families is good breadth. Signal detection theory is a well-understood statistical framework that makes the results interpretable. But the core authorship finding rests on proprietary data, and the training study uses only 2,000 examples — enough to demonstrate the mechanism exists, not to characterize its scaling behavior. The successor question is obvious: does this readout distortion show up on safety-critical tasks where alignment is supposed to work? The paper carefully stays in the domain of factual judgment (authorship, negation), but the implication for alignment is loud. If post-training changes how models report what they know rather than what they know, then alignment techniques that rely on behavioral outputs alone may be checking the translator's speech, not their notes. The paper doesn't run this experiment — likely because the political and methodological complexity of safety-task evaluation is a different paper entirely, and because connecting "readout distortion" to "alignment failure" requires a much longer causal chain than the data here supports.
The analogy
A court translator who understands the witness perfectly but has been coached by their employer to soften verdicts. The internal notes (probes) say 'guilty'; the spoken translation (model answer) says 'not guilty.' Post-training coached the translator, not the witness.
Court translator
Understands both languages perfectly but trained to soften outputs — notes differ from spoken words
Post-trained LLM
Internal representations encode the correct answer but SFT/RLHF changes the readout pathway, degrading or inverting outputs
Honest assessment
A well-designed mechanistic analysis that provides the clearest spatial localization yet of how post-training distorts the mapping from internal representation to behavioral output. The proprietary-data limitation is real and will slow replication. The finding matters most if someone extends it to safety-critical alignment tasks — which this paper conspicuously does not do. Worth reading now if you work on interpretability or alignment; worth watching for the safety-task follow-up if you don't.
What Is CLAIMS?
57
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
4/5
Ladder
3/5
Architecture
3/5
Integrity
2/5
Milestone
2/5
Successor
3/5
What is this paper actually committing to?
Real finding: post-training changes readout, not representation. Demonstrated with d' drops and inversions across six model families. Not first to suggest this, but first with this level of spatial localization.
The field fight
Does post-training (RLHF/SFT) change what language models represent, or just how they report it?
Side A
Post-training genuinely reshapes model representations and beliefs, making alignment behavioral AND representational
Held by: Anthropic RLHF papers, most alignment-optimist work
Side B
Post-training is cosmetic — it changes the readout layer, not the underlying representation, so alignment may be surface-level
Held by: Burns et al. CCS, Li et al. truth directions, this paper
Milestone tracker
Unlocks: Empirical test of whether RLHF alignment is representationally real or readout-cosmetic
Current
6
model families
readout distortion demonstrated
Next
1
safety benchmark suite
probe-vs-answer on alignment tasks
~1-2 years; requires safety-task collaboration
Verified Facts (6)
Post-training lowered the sensitivity of the answers on average (Δd' = -0.34 across model families; -0.29 across three prompts)
Six open-weight base models and their post-trained versions were compared
In two models with stage-wise checkpoints, the change occurred during supervised fine-tuning, model stitching implicated the early layers
Adding 2,000 correctly labeled authorship judgments of news texts to otherwise identical fine-tuning data improved the answers on newspaper and textbook corpora and inverted them on the chats
Probes decoded truth at or near ceiling in every model for negated statements
64 pages, 7+2 figures, 4+6 tables
Practical Implications (4)
Linear probes on internal states are more reliable indicators of what a model 'knows' than its behavioral outputs — practitioners evaluating model capabilities via prompting alone may be reading the translator, not the witness
As few as 2,000 labeled examples during SFT can flip readout direction on a held-out task, suggesting alignment tuning is cheaper and more fragile than assumed
Post-training evaluation should include probe-based checks alongside behavioral tests to detect representation-behavior mismatches
Safety teams should investigate whether RLHF-trained refusals mask intact harmful-content representations that could be accessed via activation-space attacks
More latest news
Read all