Imagine you're playing Clue. No single card in your hand names the murderer. But three cards together — the candlestick was in the library, Colonel Mustard was absent from dinner, the library door was locked — narrow it to one person. That's quasi-identifier inference: individually harmless facts that jointly unmask someone. This paper builds the first benchmark specifically designed to test whether AI privacy detectors can spot those compound leaks, conditioned on what a sender's disclosure policy actually permits. The committed claim: existing privacy benchmarks focus on explicit identifiers (names, SSNs) or single-attribute inference, but miss the combinatorial case where a set of weak quasi-identifiers jointly reveals a private attribute. QuasiLeak-Bench fills that gap with a reverse-construction pipeline that first selects weak quasi-identifiers, composes them into verified groups, generates realistic documents around them, and applies automated quality control. The result is 37 pages of benchmark infrastructure — 15 figures, 25 tables — built to stress-test detection at a granularity nobody was measuring. Where does this sit on the ladder? The paper explicitly shows that existing privacy detectors — the kind deployed in production systems — transfer poorly to this task. Locally deployable LLM-based detectors fall far behind proprietary models (think GPT-4-class), primarily because they miss most quasi-evidence groups entirely. The gap isn't about nuance; it's about detection at all. The proprietary models do better but still aren't solving the problem. The interesting signal: a fine-tuned 4B-parameter model trained on the benchmark's training split surpasses the strongest prompted model on held-out documents, demonstrating that the task is learnable — you just need the right training data. Architecturally, this is a benchmark paper, not a model paper. The core contribution is the reverse-construction pipeline: select quasi-identifiers → compose into groups → verify joint revelation → generate documents → QC. The evaluation targets are prompted LLMs (both proprietary and open-weight) plus a fine-tuned 4B model. The method leans on the compositional reasoning capabilities of large language models, testing whether they can perform multi-hop inference over scattered textual cues. The policy-conditioning layer adds a constraint-satisfaction dimension: it's not enough to find the group; you must also check it against sender-specified disclosure rules. Integrity is mixed but honest. The benchmark is self-constructed — the authors built the ground truth, which introduces circularity risk. However, the reverse-construction pipeline with automated QC is a reasonable mitigation, and the paper is transparent about the construction process. The evaluation covers multiple model families (proprietary and open-weight) across held-out test documents, and the fine-tuning experiment provides a calibration point. No pre-registration, no independent replication yet, and the benchmark's ecological validity — how well synthetic quasi-evidence groups mirror real-world privacy leaks — remains an open question. The milestone that matters: can a locally deployable model (sub-10B parameters) match proprietary-model performance on quasi-evidence group recovery? The 4B fine-tuned model already surpasses the best prompted model, but the absolute recovery rates likely remain low. The next concrete number to watch is whether a fine-tuned 7-8B model can hit >80% group-level recall on held-out documents — that's the threshold where this becomes deployable as a real privacy filter rather than a research artifact. The experiment they didn't run: adversarial document generation, where an attacker deliberately distributes quasi-identifiers across multiple messages or conversation turns to evade detection. This is the real-world threat model — privacy leaks that span context windows. The honest read is (a) compute and scope constraints for a benchmark paper, and (c) it's the obvious follow-up paper. Red-teaming the benchmark against adversarial evasion would be the natural next step.
