Imagine you ask a friend why they didn't pick Restaurant B. They say: "B doesn't have outdoor seating." You could just nod — or you could test them. You hand them an updated menu showing B does have outdoor seating, and watch whether they switch. That's the core move here: take a language model's stated rejection reason at face value, surgically fix it, and see if the model's behavior changes. If it does, the stated reason was doing real causal work. If it doesn't, the model was confabulating. Archit Rastogi runs exactly this experiment across six open-weight models on 2WikiMultihopQA, a multi-hop question-answering benchmark. When a model rejects a candidate by naming a missing fact — "no director listed," "no date of death" — the author inserts a real corpus sentence supplying that fact into the rejected candidate's profile. Two controls isolate what's happening: a length-matched irrelevant sentence at the same profile (controlling for mere text addition), and the same sentences injected at a third option the model never mentioned (controlling for placement). Greedy decoding throughout — no sampling variance to hide behind. The headline result: supplying the named fact at the named rival moves the model's choice more than the irrelevant control, with an odds ratio of 3.57 (CI: 1.54–8.26, Holm-corrected p = 0.0210). That's a real content effect — the stated reason has some causal bite. But the strongest finding in the entire family carries no content claim at all: an identical irrelevant sentence moves the choice more at the named rival than at the unnamed third option (Holm p = 0.0008). Translation: where you put extra text matters more than what the text says. The model's attention to the named rival is the load-bearing variable, not the factual repair. The contrast the study was specifically designed to detect — the same fact inserted at the option nobody mentioned — does not survive multiple-comparison correction (Holm p = 0.2428). This is the honest null that separates a careful paper from a hype paper. The author reports it plainly and doesn't bury it. Meanwhile, post-hoc matching on co-candidate mentions and relation templates preserves the content effects' direction, but matching on fluency weakens one effect. So the content contrasts "bound an effect rather than establish one" — the author's own words, appropriately cautious. A forced single-token probability read disagrees in direction with the free-text choice on the same contrast. Three candidate explanations for this disagreement (context shift, probability mass redistribution, tokenization artifacts) find no support. This is an unresolved puzzle the paper flags rather than resolves, and it matters: if your probing methodology and your behavioral methodology disagree, you don't have a clean causal story yet. The validation discipline here is unusually strong for an XAI workshop paper. Every measurement is a string-matching rule, and each was validated against the records it reads. This validation process caught eight defects, the largest being a choice-parsing rule that misidentified the model's pick in 17.1% of adjudicable responses — returning the option the model had just rejected. Had this gone unfixed, the paper would have reported six surviving contrasts instead of four. That's a 50% inflation in headline results from a single parsing bug, and the author's willingness to document this is itself a methodological contribution. The paper lands in a live and important fight: when LLMs explain themselves, are those explanations faithful to the computation, or are they post-hoc rationalizations that share the surface structure of reasoning but not its causal backbone? This work says: partly faithful, mostly placement-driven, and the tools for distinguishing the two are not yet sharp enough. It's a well-scoped, honest brick in a wall that the field badly needs built.
