Imagine you hire a locksmith who guarantees your door can withstand 500 pounds of force. You don't have a battering ram — but you can lean on the door with progressively heavier weights and measure when the latch starts to give. If it budges at 300 pounds, the guarantee is busted, even if you never kicked the door down. That's what this paper does to differential privacy claims on graph data. The committed claim: the authors present the first systematic empirical auditing methodology for differentially private graph release mechanisms, identify two graph-specific design problems (canary construction and witness statistic selection), and — critically — use the methodology to falsify the privacy guarantee of a published state-of-the-art Node-DP mechanism. The lower bounds on epsilon they obtain exceed the value the mechanism claims, producing certified counterexamples. This isn't a theoretical objection; it's an empirical falsification with statistical certificates. Why graphs are harder than tables: DP auditing for tabular data and ML pipelines has a decent toolkit ecosystem. You pick a neighboring dataset pair, run the mechanism many times, and check whether the output distributions are as close as the epsilon promise says. For graphs, two problems bite immediately. First, canary construction — what counts as a 'neighboring' graph? Adding a node changes edge structure globally; the choice of canary pair can make or break your audit. Second, witness statistic selection — which property of the released graph do you measure to detect a privacy leak? Degree distribution? Triangle count? Clustering coefficient? The paper shows that your choice of witness statistic materially affects whether you catch a violation, meaning single-statistic audits are dangerously incomplete. The architecture is straightforward: generate canary pairs (neighboring graphs differing by one node), run the target mechanism thousands of times on each, compute a battery of witness statistics on the outputs, and apply a statistically certified hypothesis test (Clopper-Pearson confidence intervals on the likelihood ratio) to obtain a certified lower bound on epsilon. The toolkit is implemented in Python and released as open source. The target mechanism audited is a published Node-DP graph release algorithm — a state-of-the-art method designed to satisfy node-level differential privacy for synthetic graph generation. The results are damning for that specific mechanism. Across synthetic and real-world canary pairs, the certified epsilon lower bounds exceed the claimed epsilon. The violations persist even when the input domain is progressively restricted to non-empty and connected graphs — ruling out the defense that the counterexamples are degenerate edge cases. The paper also demonstrates that different witness statistics yield different detection power: some statistics catch the violation clearly, others miss it entirely. This means any audit using a single witness statistic is structurally incomplete and could give a false all-clear. The broader implication is institutional. Differential privacy on graphs is increasingly deployed — social networks, contact tracing, financial transaction graphs. If published mechanisms can pass peer review while carrying falsifiable epsilon claims, the field has a validation gap. This toolkit is the beginning of empirical accountability infrastructure. The fact that the violation was caught by an external team using a general-purpose methodology — not by the original authors — underscores why independent auditing capacity matters. What's missing is scale and breadth. The paper audits one mechanism. The canary pairs, while varied, are still constructed by the auditors — adversarial but not exhaustive. The next obvious step is applying this toolkit to the broader zoo of DP graph mechanisms and seeing how many survive. The authors don't report that experiment, and the honest read is they're scoping this paper to methodology-plus-one-case-study, saving the survey for follow-on work.
