LLMs Translating AI Decisions Into Plain English Are Quietly Discriminating — and Nobody Was Checking

When LLMs narrate why a loan was denied, they give less explanation and harsher framing to historically disadvantaged groups. First systematic audit of fairness in AI-generated explanations.

Imagine you're a translator at an embassy, converting a judge's written verdict into the defendant's native language. Your job is faithfulness — same meaning, same completeness, same tone. Now imagine the translator unconsciously softens the verdict for some defendants and sharpens it for others, drops key reasoning for certain nationalities, or invents facts that sound plausible but aren't in the original. The defendant has no way to check — they only see the translation. That's what's happening when LLMs turn SHAP values into plain-English explanations of adverse decisions. The committed claim: LLM-generated explanatory narratives (XAINs) of XAI artifacts systematically discriminate across social groups in three measurable ways — faithfulness to the underlying SHAP values, feature coverage (how many features get mentioned), and sentiment. This is the first paper to define fairness in XAINs as a research problem and to build a causal decomposition framework that separates WHY the disparities arise. The framework is methodologically clean. For each adversely classified individual, narratives are generated under three conditions: protected attribute disclosed, withheld, and counterfactually swapped (e.g., male → female). Five pairwise comparisons then isolate whether a disparity comes from the feature values themselves (unequally distributed across groups) or from the LLM reacting to the protected attribute directly. They test across four datasets — credit scoring, education, employment — with five LLMs, evaluating faithfulness, feature coverage, and sentiment. The finding that holds across all datasets and models: once the LLM knows the applicant's group membership, it gives less explanation to and frames more negatively the narratives of historically disadvantaged groups. The causal decomposition is what elevates this above a bias audit. Faithfulness disparities arise from feature values: LLMs hallucinate plausible-sounding but false statements triggered by values that happen to be unequally distributed (e.g., certain income ranges or employment types). Coverage and sentiment disparities arise from the protected attribute itself — the LLM's prior associations do the damage. This distinction matters practically because the fixes are different: a more faithful model removes the first kind, withholding the protected attribute mostly removes the second, but neither fix alone is sufficient. The ladder position is strong for a first-of-kind framing paper. There's no direct prior work on XAIN fairness to compare against — the authors are defining the benchmark, not competing on one. They do compare five LLMs head-to-head and show that model choice matters: more capable models reduce faithfulness disparities but don't eliminate coverage or sentiment disparities. The architectural lens here is prompt-based generation over structured SHAP artifacts — no fine-tuning, no RLHF intervention, just vanilla prompting across GPT-4o, Claude, Llama, Gemini, and Mistral. Integrity is above average for a fairness paper. Four public datasets, five models, three generation conditions, five comparison types, and three metric families create a large combinatorial space that's harder to cherry-pick. Code, prompts, generated narratives, and extracted evaluations are all released. The main integrity limitation is that faithfulness and sentiment are themselves evaluated by LLMs (GPT-4o as judge), introducing potential circular bias — though the authors acknowledge this and the alternative (human annotation at this scale) is impractical. The milestone question is where this gets urgent. Regulatory frameworks like the EU AI Act already require explanations for adverse automated decisions. If the explanation layer itself discriminates, the entire right-to-explanation apparatus is compromised. The concrete next number: what percentage of real-world deployed XAIN systems pass a fairness audit? Nobody has measured this yet. The obvious successor experiment the authors didn't run — and my honest read is resource constraints plus wanting to establish the framework cleanly first — is testing whether RLHF-aligned or fine-tuned explanation models show the same disparities, or whether the bias is a property of base-model priors that alignment partially addresses.