When You Can Only Spot Mistakes, Stop Rewarding Guesses: A New RL Strategy for Imperfect Verifiers

NOPO only punishes verified failures, treating everything else as unknown. Gains hold across code, math, and logic when full verification is unavailable.

Imagine you're grading a stack of multiple-choice exams, but your answer key only tells you which answers are definitely wrong — it can't confirm which are right. The standard approach in education would be absurd: give full marks to every answer you can't prove wrong, then wonder why students learn to game the test. That's exactly the failure mode of standard reinforcement learning with verifiable rewards (RLVR) when the verifier can only falsify, not confirm. This paper formalizes that asymmetry and builds a training algorithm that refuses to play the game. The core contribution is a formalization the authors call one-sided verifiability (OSV): your verifier can reliably flag wrong answers (verified negatives), but anything it doesn't flag is merely non-falsified — not confirmed correct. Standard RLVR treats non-falsified outputs as positives and reinforces them, which directly rewards shortcuts that pass partial checks. Think of a coding agent that produces syntactically valid but logically wrong code that passes unit tests by accident. NOPO's fix is conceptually simple: never reinforce positives, only suppress negatives. Treat every non-falsified sample as unlabeled. Mechanically, NOPO combines two pieces. First, a reference-relative negative suppression loss that pushes the policy away from verified failures relative to a reference policy — not toward any particular positive. Second, a group-level adapter that scales each gradient update by the proportion of non-falsified responses in each batch. When a problem is easy and most responses pass the partial check, updates are small; when most responses fail, the signal is strong. This is an elegant way to handle the information asymmetry without needing a confidence threshold or a secondary reward model. The evaluation covers three verifier types — unit tests for code, constraint checks for logic puzzles, and LLM self-verification for math — across standard benchmarks. The headline result: NOPO improves pass@k over baselines (including standard RLVR and rejection sampling) specifically in the regime where oracle rewards are unavailable. Crucially, gains persist under inference-time scaling (more samples at test time) and cross-benchmark transfer, suggesting the policy learns generalizable reasoning rather than verifier-specific shortcuts. The baselines are reasonable but not exhaustive. The paper compares against standard RLVR, rejection sampling, and ablations of its own components. What's missing is a head-to-head against recent work on reward model uncertainty (e.g., ensemble-based reward models) or conservative reward estimation approaches. The NeurIPS Spotlight acceptance lends credibility, but the benchmarks are researcher-chosen, not pre-registered, and no code availability is mentioned in the abstract. The deeper significance is methodological. Most real-world verification is one-sided: you can run a code linter but not prove correctness, you can check a legal document for known errors but not verify it's optimal, you can flag a medical diagnosis as inconsistent but not confirm it's right. NOPO suggests that the right response to partial verification is humility — update only on what you actually know. This is a framework paper dressed as a methods paper, and the framework is the more durable contribution. The obvious next step the authors didn't run: scaling NOPO to frontier-scale models (70B+) where the cost of false-positive reinforcement is highest. The honest read is compute budget — this is a multi-institution academic team, not a lab with thousands of GPUs. Whether NOPO's gains hold when the base policy is already strong enough to produce mostly-correct outputs is the question that determines whether this is a training trick or a training paradigm.