Imagine you're judging a cooking competition, but every time you swap out a judge, change the scoring rubric, or rephrase the question — "Is this dish salty?" vs. "Rate the saltiness on a scale" — the winner changes. You'd stop trusting the contest before you stopped trusting the chefs. That's the core mechanism here: the paper audits the evaluation interface used to test whether de-identified text actually protects privacy, and finds the interface itself is a dominant source of noise. The committed claim: privacy verdicts on de-identified clinical and social-media text are not stable properties of the text — they are properties of the evaluation interface. The authors define an "evaluation interface" as the tuple of (queried person, answer options, prior distribution, adversary LLM, and scoring rule), then systematically vary each component across three corpora. The result is that changing who the adversary model is — GPT-4o vs. Llama-3.1 vs. others — reverses editor rankings in 5 of 124 pairwise comparisons, and the query itself can target the wrong person entirely. Most other interface changes produce gains or losses of separability rather than outright reversals, but 5/124 adversary-driven reversals on fixed text is enough to question any single-model evaluation. The stress test is the sharpest exhibit. The authors build a deliberately adversarial editor that, instead of redacting sensitive spans, rewrites them to the opposite-sex form at the same locations a redaction editor would mask. This "opposite-sex" editor actively misleads the adversary about most people — and yet ranks first among editors on MedQA entries under attack accuracy and proper scoring rules read as privacy. A relabelling-invariant reading of AUC drops it to sixth of eight. The gap between first and sixth on the same text, same task, same corpus, purely from scoring-rule choice, is the paper's central exhibit. They introduce a prior-normalised clipped leakage score that measures the adversary's confidence gain on the true attribute value, designed to be less gameable than raw accuracy. But even this score is interface-dependent: a rule-aware adversary recovers the protected attribute from text that fooled a naive adversary. The takeaway isn't that their score is bad — it's that no single score is trustworthy without interface-level reporting. SACC, the traceable editor they build as an instrument, lets them ablate two editing principles and identify where its advantage breaks down. This is the constructive half of the paper: rather than just tearing down evaluation, they demonstrate what a transparent, auditable editing pipeline looks like when you treat the evaluation interface as a first-class object. The ablation shows which editing choices matter and which are decorative. The paper runs 49 pages including appendices, and the density is justified — each interface dimension gets its own ablation, each corpus its own table. The integrity posture is strong for an audit paper: they vary six interface choices across all pairs, report all 124 adversary comparisons rather than cherry-picking the dramatic ones, and explicitly show where their own editor fails. The weakness is that all validation is internal — no independent team has replicated the interface-sensitivity findings on a fourth corpus. The practical upshot: if you are publishing or consuming privacy evaluations of de-identified text, you need to report the full evaluation interface or the result is not interpretable. A single adversary, a single scoring rule, a single set of answer options — any of these can flip the verdict. The paper doesn't solve privacy evaluation; it proves the current apparatus is under-specified.
