
Imagine you're a customs agent watching faces through a one-way mirror. You've trained yourself to spot liars by their micro-expressions — the slight twitch, the gaze aversion. You're good at it, and it transfers decently across smugglers, visa fraudsters, and tax cheats. But then someone walks up who is telling the truth when they say 'I don't know' and someone else walks up who is lying when they say the exact same words in the exact same way. Your micro-expression detector is now useless — not because lying stopped having a geometry, but because the surface signal you learned is shared between the two classes. That is exactly the wall this paper hits. The committed claim: label-free 'persona directions' — vectors extracted from the middle layers of LLMs by contrasting honest and deceptive role-play completions — can rank deceptive outputs on the Liars' Bench benchmark better than single-dataset supervised probes on three of four models tested (Llama-3.3-70B, Gemma-2-27B, Mistral-Small-24B, Qwen-2.5-32B). Additionally, a novel 'knowledge direction' — extracted by contrasting answerable questions with unanswerable ones, both followed by scripted denials — can separate lies from wording-identical honest denials on two to three of the four models. The sum of both directions outranks a supervised probe at the same layer depth across all four models. The ladder here is honest but sobering. The persona direction, built from a widened role-play bank, beats single-dataset probes on Liars' Bench for Llama, Gemma, and Mistral, but fails on Qwen despite Qwen exhibiting the highest role-play compliance. It beats random persona banks on Llama-3.3-70B, but its advantage over the already-published persona direction from prior work falls within the range of random widenings — meaning the improvement might be noise. The knowledge direction is the genuinely new contribution, but it only clearly separates lies from identical honest denials on two models, narrowly on a third, and not at all on the fourth. On the Gender-Secret fine-tuning benchmark, it clears the random-direction band by only a few hundredths on Llama and fails on Mistral. Architecturally, this is linear probing in activation space — the cheapest family of mechanistic interpretability tools. You take a model's hidden states at a chosen layer, compute mean activations for two contrasting sets of prompts, subtract to get a direction vector, then project new activations onto it. No training, no gradients, no learned parameters beyond the direction itself. The method leans on the linear representation hypothesis: that concepts like 'lying' live as directions in residual stream space. The paper tests this at scales from 24B to 70B parameters across four instruction-tuned models, using middle-layer activations (the layer is selected per model). The integrity picture is mixed in an instructive way. The authors are commendably honest about failures — Qwen breaks the pattern, the widened persona bank doesn't clearly beat published baselines, and the knowledge direction's successes are often marginal or wording-dependent. They use a published community benchmark (Liars' Bench) with four sub-datasets and test against random-direction baselines, which is the right null hypothesis for unsupervised direction extraction. But there's no pre-registration, no code release mentioned, and the benchmark selection appears post-hoc. The paper's most damning self-admission: on knowledge lies, persona directions 'mainly track a reply's wording' — meaning the direction may encode surface lexical features rather than a genuine deception concept. This is exactly the kind of confound that makes linear probing results fragile. The milestone question for this subfield is concrete: can unsupervised lie-detection directions achieve transfer performance matching supervised probes across all major model families, including on lies where surface wording is controlled? This paper shows we're not there. The persona direction fails on 1 of 4 models; the knowledge direction fails or is marginal on 1-2 of 4; and the combined direction, while beating supervised probes at matched depth, trails the persona probe alone on held-out lies. The next number to watch is consistent above-random-band performance on knowledge-denial lies across 4+ model families without wording confounds. The obvious experiment not run: testing these directions on models fine-tuned specifically to deceive (beyond the single Gender-Secret benchmark) or on naturally occurring deception in deployment logs rather than synthetic role-play. The honest read is (a) — the authors are working with publicly available models and benchmarks and don't have access to production deception data. But the deeper gap is (c): a systematic ablation isolating what the persona direction actually encodes (refusal style? hedging language? genuine belief-state geometry?) would be the real follow-up, and that's a full paper on its own.
The analogy
Like a customs agent trained to spot liars by micro-expressions — works great until a truth-teller and a liar say 'I don't know' with the exact same face. The detector learned surface cues, not the underlying state, and identical outputs from different internal states break it.
Customs agent
Reads micro-expressions to distinguish liars from truth-tellers; fails when both groups produce identical expressions
Persona direction
Reads activation-space direction to distinguish lies from truth; fails when lies and honest denials produce identical token sequences
Honest assessment
This is a careful, honest negative-result paper dressed up as a positive one. The genuinely new contribution — the knowledge direction — works on half the models and barely clears the random baseline when it does. The persona direction transfers better than expected at 70B but its improvement over prior work is within noise. The real value is the clearly demonstrated wording confound: persona directions track surface form, not internal belief state, which is an important finding for the mechanistic interpretability community even if the authors understate it. Read this if you work on LLM safety or interpretability and want calibration on what linear probes actually detect.
What Is CLAIMS?
63
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
3/5
Ladder
4/5
Architecture
4/5
Integrity
3/5
Milestone
2/5
Successor
3/5
What is this paper actually committing to?
Incremental but honest. Extends persona directions to 70B scale and introduces a knowledge direction, but both break on at least one model and the core finding is a negative: surface wording confounds the geometry.
The field fight
Do LLM residual streams encode human-legible concepts like 'truthfulness' as stable linear directions that generalize across contexts?
Side A
Yes — linear directions for truth/deception transfer across datasets and can be extracted without labels from role-play contrasts
Held by: Marks et al. (2024), Representation Engineering (Zou et al.)
Side B
No — apparent directions mostly track surface-form features (wording, refusal templates) and break under distribution shift or controlled wording
Held by: Levinstein & Herrmann (2024), skeptics of linear representation hypothesis
Milestone tracker
Unlocks: Reliable unsupervised lie detection deployable as a safety layer without per-dataset labeling
Current
2-3
of 4 models
knowledge direction works
Next
4+
model families
wording-controlled separation
~1-2 years if community adopts wording-controlled benchmarks
Verified Facts (8)
Persona direction ranks lies across Liars' Bench better than single-dataset probes on three of its four models; it fails on Qwen
On Llama-3.3-70B-Instruct it beats directions from a random persona bank, but its gain over the published direction is within the range of random widenings
On lies about knowledge, persona directions mainly track a reply's wording
Knowledge direction separates lies from identical honest denials on two models and narrowly on a third, but not on the fourth
The sum of the two directions ranks lies better than a supervised probe at the same depth on all four models
On the Gender-Secret fine-tune it clears the band by a few hundredths on Llama and fails on Mistral
More varied role-play questions do not help the persona direction on Liars' Bench
Qwen stays in role far more often than other models
Practical Implications (4)
Linear persona directions are not yet reliable enough for deployment as unsupervised lie detectors — they fail on at least one major model family and confound with surface wording
Combining persona and knowledge directions outperforms supervised probes at matched depth, suggesting ensemble approaches may be worth exploring in safety pipelines
Anyone building lie-detection tooling should test on wording-controlled scenarios (identical surface form, different ground truth) — this paper shows most directions fail there
The Qwen failure despite high role-play compliance suggests model family architecture matters for representation geometry in ways not yet understood
More latest news
Read all