Imagine you're grading a geography test where the only answers marked correct are "Paris" and "London," and every other city on Earth — including Lyon, Munich, and Berlin — counts as equally wrong as "banana." That's how skill extraction benchmarks have worked: a handful of human-annotated positives, and the entire 13,939-skill ESCO taxonomy treated as uniformly incorrect otherwise. The WorkRB Challenge at RecSys-HR 2026 argues this is broken and builds something better. The core claim is methodological, not algorithmic: replacing binary relevance (right/wrong) with graded relevance (0-4) on every candidate skill in the ranking window fundamentally changes what benchmarks measure and which systems win. The authors found that ternary labels (irrelevant/partial/relevant) were too coarse for annotators to agree on, so they decomposed relevance into a cascade of four binary questions — same domain? activity mentioned? right granularity? could replace a gold positive? — that a smaller LLM judge can answer reliably. Human and frontier-model blind re-annotation validates the cascade. Five benchmark datasets were constructed: four use vacancy sentences as queries, one uses lexically distant skill surface forms to stress semantic understanding. The evaluation metric is nDCG@100 against these graded labels. This is where it gets interesting: at the top of the leaderboard, the graded ordering of submissions is uncorrelated with what binary evaluation would have said. Binary metrics rewarded systems for ranking annotated positives high — possibly for spurious reasons — while graded metrics reveal whether a system actually understands the semantic landscape around a query. Every competitive submission converged on multi-stage retrieve-and-rerank pipelines. The insight the authors extract from this convergence is telling: the gains come from clearing semantically nonsensical predictions out of the ranking window, not from surfacing more true positives. In other words, the winning move isn't "find the needle" — it's "stop returning haystacks labeled as needles." Graded labels expose much larger gaps between top submissions and baselines than binary evaluation does, confirming that binary metrics were compressing real performance differences into noise. The efficiency problem is stark. Every submitted system the authors could size uses at least 42 times — and typically several hundred times — more parameters than the baselines it beats. The field has solved the quality problem at brutally uneconomical scale. The paper names this explicitly as the clear next direction: bringing efficiency back without losing the semantic understanding that large retrieve-and-rerank pipelines provide. The integrity picture is solid for a benchmark-construction paper. The LLM cascade is validated against blind human re-annotation, and the graded labels are released as a shared evaluation resource. The weakness is what you'd expect: the cascade itself is an LLM judge, which means the benchmark is ultimately graded by a language model, not by domain experts with hiring experience. The authors acknowledge this but argue the binary decomposition makes LLM judgments more reliable than holistic relevance scoring. What this paper actually changes is the conversation. If you're building or evaluating skill extraction systems — for HR tech, job matching, talent analytics — the message is: your binary benchmarks are lying to you. They're rewarding systems that get lucky on a few annotated positives while ignoring whether the rest of the ranking is semantic garbage. Graded relevance is harder to construct but tells you what you actually need to know.
