Imagine you have a bathroom scale that reads 10 pounds lighter whenever you step on it wearing socks instead of barefoot. You might celebrate the weight loss — until you realize the scale is just responding to the socks, not your body. That is the state of multilingual bias benchmarking: researchers translate English stereotype tests into Hindi or Swahili, observe lower scores, and conclude the model is less biased in those languages. This paper demonstrates that the lower reading is mostly the socks. The committed claim: across all 70 model-language pairs tested, stereotype scores drop below English levels when items are translated — but the drop is not evidence of reduced bias. It is evidence that translation reshuffles item-level scores into a different pattern that happens to average lower. The correlation between an item's English score and its translated score is just 0.23 (0.27 for verified-faithful translations), versus 0.48 for mere English rephrasings. The average moves, but the signal underneath is scrambled. The ladder matters here. The standard practice in the field — comparing aggregate stereotype scores across languages using benchmarks like StereoSet and CrowS-Pairs — is the baseline this paper attacks. There is no named prior work that systematically ran the three-way comparison: original English, rephrased English, and translated. That makes this a methodological first for the specific question of how much translation noise contaminates bias audits, though the broader concern about cross-lingual benchmark validity has been raised before. Architecturally, this is a measurement-methodology paper, not a model paper. The authors audit ten open models (the specific names aren't given in the abstract but the code is released) using StereoSet and CrowS-Pairs items translated into eight languages: Hindi, Korean, Swahili, Spanish, and four others. The key innovation is the three-condition design: English original, English rephrasing (to isolate rewording noise), and translation. They also inject known bias shifts to verify their pipeline can detect real signal — a sensitivity check that most bias audit papers skip entirely. The integrity story is unusually strong for a fairness paper. The authors run sensitivity analyses (injecting bias to confirm detection), check translation quality item-by-item and separate faithful from imperfect translations, and document three common pipeline errors that silently erase bias signal — errors other researchers are likely making right now. They release code. The main limitation is that this is still the authors grading their own methodology; no independent replication exists yet. But the design is more rigorous than the standard it critiques. The numbers that stick: within-language, a translated item's score correlates with its own rephrasing at 0.82–0.84 (Hindi, Spanish), meaning the model's behavior on that concept is stable within a language. But cross-lingually, the item-level prediction collapses to 0.23. The average bias score is a lossy summary that cannot distinguish 'less biased' from 'differently scrambled.' For Hindi, Korean, and Swahili, the translation-induced drop clearly exceeds what rewording English alone produces, pointing to language-specific measurement artifacts, not fairness differences. The practical upshot is uncomfortable: any paper that reports 'Model X is less biased in Language Y than in English' using translated benchmarks is likely reporting noise. The field needs item-level analysis, not aggregate comparisons. The three pipeline errors the authors document — which silently erase bias signal — mean some published results may be even more wrong than the averages suggest. This paper does not solve the measurement problem, but it makes it impossible to ignore.
