Imagine you're a detective investigating which textbook taught a student a specific fact. You dust the entire book for fingerprints, find smudges on every page, and conclude the evidence is inconclusive. But the answer was on page 47 — the student only touched three sentences. The problem was never your fingerprinting technique; it was averaging faint prints across 300 pages instead of looking for the brightest ones. That's what this paper shows is happening with LLM data attribution. The committed claim: existing data attribution methods for language models — the algorithms that try to answer "which training document caused this output?" — fail not because their gradient estimators are broken, but because two overlooked design choices (the loss function used to compute gradients and how per-token scores get aggregated into document scores) drown the real signal in noise. The fix is surgical: replace standard cross-entropy loss gradients with negative log-odds gradients, then aggregate only the top-scoring tokens rather than averaging across the whole document. This recovers 35 to 72 percentage points of ground-truth recall across multiple baselines. To isolate exactly what goes wrong, the authors build CATT — the Controlled Attribution Benchmark — by fine-tuning LLMs on entirely fictitious knowledge. Because the facts are synthetic and unambiguous, there's no question which training document is the ground truth. This lets them turn individual difficulty knobs one at a time: document redundancy, lexical distractors, paraphrase complexity. The result is a diagnostic testbed where you can watch an attribution algorithm degrade as you add one confound at a time, rather than guessing which of five simultaneous confounds killed it. The architectural insight is that causal evidence within a training document is spatially concentrated — a handful of tokens carry the attribution signal, while the rest are noise. Standard methods compute a gradient-based score for every token and then average them, which is equivalent to diluting a strong signal with hundreds of zero-signal tokens. The negative log-odds trick works because it suppresses uninformative tokens at the gradient level, making the top-k aggregation dramatically more effective. This is not a new estimator; it's a correction applied to existing estimators (TRAK, gradient dot-product, cosine similarity). The results on the synthetic benchmark are striking: 35 to 72 percentage-point improvements in recall. But synthetic benchmarks are easy to overfit to. The critical validation is FTRACE-TREX, a counterfactual-retraining benchmark built on real knowledge. Here, the negative log-odds correction improves every tested method, with the best achieving a 4.9× improvement. The counterfactual design — actually retraining models with documents removed and checking whether predictions change — is the strongest form of attribution validation available without ground-truth provenance labels. What makes this paper worth your time is not the specific numbers but the diagnostic methodology. The field of data attribution has been comparing method A vs method B on full-pipeline benchmarks where five things vary at once. CATT provides the controlled experiment: one knob at a time. The finding that the bottleneck is aggregation and loss choice, not the estimator family, reframes the entire research agenda. If your attribution method is failing, don't redesign the estimator — check whether you're averaging away the answer. The honest limitation: all results use fine-tuned models where the synthetic knowledge was injected in a controlled setting. Pre-training scale attribution — tracing a GPT-4-class output back to one of billions of training documents — remains untouched. The gap between fine-tuning-scale and pre-training-scale attribution is enormous, and it's not obvious that top-k aggregation scales when document lengths and training set sizes grow by orders of magnitude.
