Imagine you're a restaurant health inspector, but instead of checking kitchens yourself, you've hired seven substitute inspectors. Each one has a clipboard and a rulebook. The question isn't whether the restaurants are clean — it's whether your inspectors agree with each other, and whether any of them can be fooled by a kitchen that hides the dirty pans behind a curtain. That's what C-SafeQA does: it builds 37,660 labeled meals (query-response pairs) and then watches seven judges try to grade them. The committed claim: C-SafeQA is the first policy-grounded, response-level safety benchmark for Chinese LLMs that evaluates both the models being tested AND the automated judges used to test them, using shared reference labels derived from multi-model adjudication plus blind human expert audits. This is a benchmark-of-benchmarks paper — meta-evaluation infrastructure. The numbers tell a clear story about adversarial fragility. On 538 base queries, unsafe-response rates across four LLM deployments range from a tight 0.93% to 3.35%. Throw adversarial transformations at those same models — 8,877 adversarial queries — and the rates explode to 11.68%–30.05%. That's a 3.5× to 32× amplification factor depending on the model, which tells you the safety alignment layer is shallow. It catches the obvious stuff and misses the reworded version. The architecture here is evaluation infrastructure, not a model or training method. The pipeline works in stages: construct base queries grounded in specific content policies, generate adversarial variants (including acrostic transformations that encode harmful intent across first characters of lines), collect responses from four production LLM deployments, then label each query-response pair as safe, unsafe, or disputed through agreement-aware multi-model adjudication validated by three human safety experts doing blind stratified audits. Seven automated safety judges are then benchmarked against these reference labels. The integrity picture is mixed in a characteristic way. On the strong side: the reference labels use a multi-model consensus mechanism cross-checked by independent human experts, the dataset records and verification code are publicly released, and the evaluation covers seven judges rather than cherry-picking one. On the weaker side: benchmark construction methodology and adjudication details remain private, the four target LLMs aren't named (described as "full-model deployments"), and there's no pre-registration. The "disputed" label category is honest — it acknowledges genuine ambiguity rather than forcing binary labels — but we don't know the disputed rate from the abstract alone. The most actionable finding is the acrostic vulnerability. Both acrostic transformation types reduce unsafe recall for all seven judges. This isn't a marginal effect on one or two weak judges — it's a universal failure mode where encoding harmful intent as an acrostic pattern defeats every evaluated safety classifier. If you're deploying safety judges in production Chinese-language systems, this is the result that should keep you up at night. The obvious next experiment the authors didn't run: testing whether fine-tuning judges on acrostic-augmented training data closes the gap — and if so, whether a new adversarial transformation immediately reopens it. My read is (c) — they're saving the arms-race dynamics for the follow-up paper. The current paper establishes the diagnostic; the intervention paper writes itself. The deeper question this paper participates in — whether automated safety evaluation can ever be robust enough to replace human review at scale — remains open, and C-SafeQA's honest answer is "not yet, and here's exactly why."
