Imagine you're a teacher grading a stack of 1,000 essays. You could read every one carefully — accurate but slow. Or you could hire a fast but slightly sloppy teaching assistant who flags the ones they're unsure about for your review. The trick isn't finding a perfect assistant; it's finding one who knows when they don't know. That's the core mechanism here: Jev, a lightweight decision model, is worse than Claude Opus overall but much better at saying "I'm not sure" when it's wrong — and that self-awareness makes it the ideal first pass in a two-stage system. The actual problem is identity resolution grading: given a cluster of records that a system says belong to one person, decide whether those records really are one person or an over-merge of two or more. Hightouch built a benchmark of 1,000 synthetic identities using pseudopeople (from the University of Washington's IHME), half clean and half deliberately over-merged, with realistic noise — typos, nickname substitutions, swapped date fields, household data leaking between relatives. They tested five models: Claude Opus 5.5, Gemini 3.1 Pro, GPT-6 Sol, GPT-6 Luna (Decisions API), and Jev. Out of the box on the easier benchmark, the hierarchy looks predictable: Opus at 99.0%, Gemini and Sol at 98.3%, Luna at 97.3%, Jev at 95.6%. The cost gap is enormous — Jev runs at $0.05 per 1,000 identities and 0.15 seconds each; Opus costs $3.86 per 1,000 and takes 2.2 seconds. That's roughly 77x cheaper and 15x faster. But the accuracy delta matters when you're estimating a rare rate like over-merging, where you need thousands of samples to get tight confidence intervals. The key finding emerges when they crank the difficulty. On a harder benchmark built exclusively from tricky cases — twins, juniors and seniors, typos on critical fields like birth dates — Luna drops to parity with Jev while frontier models hold steady. More important than the accuracy numbers is the confidence calibration. Jev's mistakes cluster near 50% confidence, exactly where a well-calibrated model's errors should live. Luna's mistakes come with 90-100% confidence — it's wrong and sure about it. This is the worst possible failure mode for a triage system. The practical payoff: a hybrid architecture where Jev handles the bulk at $0.05/1K and routes low-confidence cases to Opus achieves frontier-grade accuracy at more than 4x lower cost than running Opus on everything. The confidence threshold tuning matters — Jev at 60% and Luna at 70% maximize accuracy, but Jev's threshold has much more impact on the error distribution than Luna's. This isn't just a cost optimization; it's a statement about what you actually need from your cheap model. You don't need it to be right — you need it to know when it's wrong. The prompt engineering is worth noting: the biggest accuracy gain across all models came from listing specific error types (typos, nickname substitutions, date-format swaps) rather than providing contextual information about data sources. Decision models gained almost nothing from source context, which the authors interpret as evidence they don't reason about provenance — they pattern-match against the error catalog. This is a useful design principle for anyone building similar pipelines. The benchmark itself has limits. Pseudopeople generates synthetic data from a single simulated US town. Real-world IDR deals with international name systems, non-Latin scripts, adversarial duplicates from fraud, and data distributions that shift over time. The 50/50 split between clean and over-merged identities is also unrealistic — in healthy production systems, over-merges are rare, which means the base rate in production would be very different from the benchmark. The authors acknowledge this implicitly when they note that estimating rare rates precisely requires large samples, but they don't test how well the hybrid performs when only 2-3% of identities are over-merged.
