Bengali Bias Benchmark Exposes a Blind Spot: LLMs Handle Stereotypes Differently Across Languages, and Accuracy Doesn't Fix It

61,312 QA instances across 12 social categories reveal that LLMs aren't just worse in Bengali — they're biased differently. Higher accuracy and lower stereotype alignment are independent axes.

Imagine you're a restaurant health inspector who grades kitchens in two cities. In City A (English), your inspectors are pretty good — they catch most violations. In City B (Bengali), they miss more, but here's the twist: inspectors who catch the most violations in City B aren't necessarily the ones least likely to give a pass to restaurants owned by their friends. Accuracy and favoritism are separate problems. That's the core mechanism of this paper: cross-lingual performance and stereotype-aligned bias in LLMs are orthogonal dimensions, not two sides of one coin. The committed claim: existing Bengali bias benchmarks — including the authors' own prior BanglaBBQ — are too small and structurally incomplete to expose how LLM bias shifts across languages. BDBBQ extends the dataset to 61,312 instances across 12 social categories (religion, gender, disability, politics, physical appearance, etc.) in both English and Bengali, with balanced answer permutations and contrasting question framings that make it much harder for a model to game the evaluation. The ladder here matters. BanglaBBQ was the prior Bengali bias benchmark; BDBBQ is roughly a 10× expansion with tighter structural controls. Against BBQ (the original English-only benchmark by Parrish et al., 2022), this extends the methodology into a bilingual, culturally grounded setting that BBQ never attempted. The paper doesn't claim to beat a SOTA bias-mitigation method — it's a measurement instrument, not a debiasing technique. The comparison that counts is instrument quality: 220 statistical tests, four expert annotators, Fleiss' κ = 0.8349 inter-annotator agreement. That's a strong validation for a benchmark paper. Eight models were evaluated across 490,496 total instances: GPT-4o-mini, Llama 3.1 8B, Gemma 2 9B, Qwen 2.5 7B, Mistral-Nemo 12B, and others. The five key findings map a landscape that's more complex than 'models are worse in Bengali.' Disambiguated contexts boost Bengali accuracy by 36.59 percentage points versus only 17.71 in English — meaning Bengali performance is dramatically more context-dependent. Mistral-Nemo shows the smallest cross-lingual accuracy gap at 5.12 pp. But stereotype-aligned bias doesn't track accuracy: Gemma 2 9B gets more biased in Bengali while Mistral-Nemo gets less biased, despite both improving in accuracy with disambiguation. Qwen 2.5 7B shows particularly strong stereotype alignment in Bengali for Politics and Physical Appearance categories. Architecturally, this is a benchmark-design paper, not a modeling paper. It belongs to the BBQ family (bias benchmark via QA) extended with bilingual permutation controls and culturally grounded social categories specific to Bangladesh. The compute property it leans on is brute-force combinatorial coverage: every question is tested with all answer orderings and both ambiguous/disambiguated framings, generating the 490,496-instance evaluation matrix. The method is language-agnostic in structure but culture-specific in content. Integrity is above-average for a benchmark paper. The 220 statistical tests and expert annotation with reported inter-annotator agreement (κ = 0.8349) are real quality signals. The structural checks — verifying answer-position balance, framing symmetry — address common benchmark gaming vectors. The main integrity gap: the eight models were selected by the authors, and we don't know if other models were tried and dropped. The benchmark itself is the contribution, so the 'who grades the homework' question is about the benchmark's internal consistency, which they address thoroughly. The obvious next experiment the authors didn't run: applying debiasing techniques to these models and measuring whether BDBBQ is sensitive enough to detect improvements. They built the thermometer but didn't test whether it can measure treatment effects. Honest read: they're saving it for the next paper. A benchmark launch paper needs to establish the measurement instrument first; the intervention study is the obvious follow-up and a clean second publication.