Imagine you're grading students' essays, but your answer key has typos in it. Every time a student spells a word correctly and your answer key has it wrong, you mark the student down. Now imagine nobody ever checks the answer key against the original text — they just assume it's right. That's what happens when we use automatic speech recognition to evaluate text-to-speech systems. The committed claim here is diagnostic, not constructive: a large fraction of what TTS benchmarks report as synthesis errors are actually recognition errors. The authors measure this by running the same ASR systems used for TTS evaluation on human recordings of the same test sentences. On LibriSpeech and FLEURS, the evaluation recognizer misses 33-67% of very rare words even in clean human speech. When all three recognizer families fail, the miss rate is still 22-42%. This isn't a TTS problem — it's a measurement problem baked into the standard evaluation pipeline. The fix they test is a multi-recognizer adjudication protocol: check every word with two additional ASR families beyond the primary evaluator. This drops apparent TTS error on words that at least one recognizer hears correctly by 2-3× (from 10-13% down to 3-5% on very rare LibriSpeech words). The improvement is real but the residual is stubborn — very rare words remain the hardest category even after adjudication. The integrity move that elevates this paper is the authors' honesty about the limits of their own fix. They test and explicitly reject a simpler approach: just ignoring words the recognizer misses in human speech. This mask is biased — the other recognizers find TTS errors 4-6% of the time on masked words versus only 0.2-0.3% on unmasked words, and overturn about half the errors the mask would keep. The naive fix is worse than the disease. They also acknowledge that roughly half of remaining adjudicated errors may be spelling variants rather than genuine mispronunciations, meaning even their improved protocol overstates true TTS error. Architecturally, this is a metrology paper, not a model paper. There's no new TTS or ASR system. The method is ensemble disagreement across ASR families — conceptually simple, computationally cheap, and immediately deployable by anyone running TTS benchmarks. The three recognizer families aren't named in the abstract, but the design principle is clear: diversity of decoder architecture matters more than quantity of recognizers. The broader field fight this enters is whether word error rate (WER) via ASR is a valid proxy for TTS quality at all. The TTS community has long known ASR-based evaluation is noisy, but most papers treat it as acceptable noise and move on. This paper puts a number on the bias — not just noise, but systematic bias against rare words — and shows it's large enough to change conclusions about which systems are better. The implication is that published TTS comparisons on rare-word performance may need to be revisited. The paper is submitted to ICASSP 2027, which means it's a 4-page conference submission — tight, focused, and not trying to do more than it should. The honest residual is that a listening test is still needed to ground-truth even the adjudicated results. The authors say this explicitly. What they've built is not the final evaluation protocol but a demonstration that the current one is broken in measurable, quantifiable ways.
