A Cheap LLM Judge for Translation Quality: Pterodactyl Shows You Don't Need Heavy Machinery for Binary Triage

WMT26 submission uses prompt-engineered LLM-as-a-judge with ROC-guided model selection for ultra-low-cost binary translation quality classification. Engineering paper, not a science paper.

Imagine you're sorting mail at a post office. You don't need to read every letter — you just need to spot the ones with wrong addresses and toss them into a second-look pile. That's Pterodactyl. It's not trying to understand translation quality deeply; it's trying to sort segments into "good enough" and "needs review" as cheaply and quickly as possible. The mechanism is triage, not diagnosis. The committed claim: an LLM-as-a-judge approach, guided by ROC analysis for prompt and model selection, can perform competitive binary classification of translation quality at ultra-low cost. This is a systems-engineering submission to WMT26's shared task on automated quality evaluation, specifically subtasks 2 (segment-level score prediction) and 3 (error-free segment detection). The authors — Evelyn Y. Garland and Carola F. Berger from 1020 Group — are explicit that they optimized for cost and customizability, not for pushing state-of-the-art accuracy. The architecture is straightforward: take an existing LLM, craft prompts that frame translation evaluation as a binary classification problem, and use ROC curves to select which model and which prompt configuration maximizes the area under the curve for the specific subtask. Several variants of Pterodactyl were developed for subtask 3. This sits squarely in the "LLM-as-a-judge" family — no fine-tuning, no custom model training, just prompt engineering on top of commercial or open-weight models. The compute property being leaned on is the zero-shot or few-shot capability of pretrained LLMs, which means the cost floor is essentially the inference API price. The integrity picture is mixed but honest about its own scope. This is a shared task submission, which means evaluation will be done by WMT26 organizers against a common test set — that's the strongest form of validation available in MT evaluation. The paper itself is short (6 pages, 4 figures), which means we're looking at a system description, not a full empirical study with ablations. The ROC-guided model selection is sensible methodology, but we don't yet see how Pterodactyl stacks up against other WMT26 submissions or against dedicated fine-tuned metrics like COMET or MetricX. The baselines are implicit — whatever other systems submit to the same shared task. The real contribution here is economic, not algorithmic. If Pterodactyl achieves even 80-90% of the accuracy of fine-tuned neural metrics at a fraction of the cost, that's useful for translation companies doing data curation at scale. The "highly customizable" angle matters: prompt engineering is cheaper to iterate than model retraining. But the paper doesn't give us cost numbers or accuracy numbers to verify that claim yet — we're trusting the design philosophy before seeing results. The obvious next experiment is running Pterodactyl against COMET-22, MetricX-23, or XCOMET on a public benchmark and publishing the accuracy-vs-cost Pareto curve. The authors almost certainly haven't done this yet because WMT26 results haven't been released — the shared task is the experiment. If the results show Pterodactyl within striking distance of heavier systems on binary classification, this becomes a useful reference for the "good enough at 1/100th the cost" argument that matters in production MT pipelines. If it doesn't, it's a well-documented engineering exercise.