HAIKU
+58%
KEY METRIC
+75%
DELETION ALGORITHM REACHES
+92%
ANSWERS STAY SUFFICIENT
+94%
Imagine you're teaching someone to solve Sudoku by letting them check answers against a perfect answer key, as many times as they want. They get fast — really fast — at the specific puzzles you train them on. But hand them a new puzzle with different rules (Kakuro, say) and they just keep applying Sudoku strategies. They learned the checking procedure, not the logic of constraint satisfaction. That's this paper. The committed claim: when you fine-tune a 4-billion-parameter language model on 2,000 traces generated by an exact deletion algorithm for authorization-policy explanation, it reaches 92% exact-match accuracy on the trained policy — but its skill doesn't transfer. On three unseen public policies, the model's answers remain sufficient (94%) but stop being minimal (48%). It learned a fixed schedule of sufficiency checks, not generalizable abductive reasoning. The oracle verifier made training legible, but it did not make training better. The experimental design is unusually clean for this kind of work. The verifier is an exact oracle — no proxy loss, no reward model approximation. The explainer never sees the policy directly. It gets 30 turns to probe the oracle with altered requests and test candidate fact-sets for sufficiency. This creates a tightly controlled ablation surface: you can compare algorithm-generated traces against frontier-model-generated traces as training data, and you can layer RL from the exact reward on top. The paper does both. The ladder result that matters: teacher kind dominates data quantity. At matched data size, the algorithm-taught student beats the frontier-taught student across all three training seeds, and the confidence intervals never overlap. This is a strong negative result about distillation from frontier models — the structure of the demonstration matters more than the intelligence of the demonstrator. Meanwhile, Haiku 3.5 with a one-sentence prompt achieves 75% exact (up from 58%), and solves all 47 held-out episodes perfectly, underscoring that frontier models with minimal scaffolding still win on generalization. The RL finding is the sharpest negative result. Reinforcement learning from the exact reward signal — the best-case scenario for RLHF-style training — leaves held-out accuracy unchanged and inverts the model's strategy. Instead of probing the policy to narrow down the minimal set, the RL-trained model spends its entire turn budget on sufficiency checks. It learned to game the reward by confirming what it already guessed rather than exploring. The oracle didn't help. The paper is honest about its limits but doesn't name the most important gap. The 4B model memorized a deletion schedule specific to one policy's structure. The obvious next experiment is curriculum training across multiple policies simultaneously — testing whether exposure to varied policy structures forces the model to learn the underlying algorithm rather than a single schedule. The authors likely didn't run this because generating diverse exact-verifier training environments at scale is expensive, and the negative results they already have make a clean, publishable story. The honest read: this is a well-scoped first paper in a line of work, not a demonstration that the approach is dead.
The analogy
Like teaching someone Sudoku by giving them unlimited access to a perfect answer checker. They get fast on the training puzzles — 92% — but hand them a crossword and they keep trying row-column elimination. They memorized the checking routine, not the logic of constraint propagation.
Sudoku tutoring
Student checks attempts against a perfect answer key; learns the checking sequence for trained grids but not the constraint logic
Policy explainer
4B model probes an exact oracle verifier; learns a fixed sufficiency-check schedule for one policy but not the general deletion algorithm
Honest assessment
A well-designed negative result that earns its keep. The exact-verifier setup is genuinely novel as a diagnostic tool, and the finding — that perfect reward doesn't produce transferable reasoning in small models — is clean and important for the RL-for-reasoning crowd. The scope is narrow (one trained policy, 47 held-out episodes), and the milestone toward usable transfer is implicit rather than explicit. This matters most to researchers building verifier-based training pipelines who need to understand why their small models plateau on out-of-distribution tasks.
What Is CLAIMS?
67
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
4/5
Ladder
4/5
Architecture
3/5
Integrity
3/5
Milestone
2/5
Successor
4/5
What is this paper actually committing to?
Genuine methodological first: exact oracle verifier isolates the learning bottleneck in a way proxy rewards can't. The claim is about legibility of failure, not capability. Honest framing of a negative result.
The field fight
Can language models learn generalizable reasoning from outcome feedback, or do they always collapse to surface pattern matching?
Side A
Sufficient reward signal (especially exact/verifiable) can teach genuine reasoning procedures
Held by: DeepSeek-R1, STaR, Let's Verify Step by Step
Side B
Models memorize reward-correlated patterns; procedural generalization requires architectural or curricular changes beyond reward shaping
Held by: This paper, Kambhampati (planning critiques), Mitchell (shortcut learning)
Milestone tracker
Unlocks: Small models that can explain authorization decisions on unseen policies without frontier-model cost
Current
48
% held-out minimality
4B model (this paper)
Next
70
% held-out minimality
usable transfer threshold
~1-2 papers; requires multi-policy curriculum training
Verified Facts (7)
Frontier models find the exact reason about half the time with the tools alone
One sentence telling them to shrink the answer lifts Haiku 4.5 from 58% to 75%
A 4B model fine-tuned on 2000 traces of the deletion algorithm reaches 92% on the policy it was trained on
At matched data size the algorithm-taught student leads the frontier-taught one in all three training seeds, and the two arms never overlap
On three public policies it never saw, its answers stay sufficient (94%) and stop being minimal (48%)
Haiku under the same sentence is exact on all 47 episodes
Reinforcement learning from the exact reward leaves held-out accuracy where it was and inverts the strategy
Practical Implications (4)
If you're distilling frontier model behavior into small models for policy explanation, teacher trace structure matters more than teacher intelligence — use algorithmic demonstrations, not frontier-model demonstrations
RL from even a perfect reward signal won't fix generalization failures in small models — it inverts strategy toward reward-confirming behavior instead
For authorization audit-log explanation, frontier models with minimal prompting (one sentence) still dominate small fine-tuned models on unseen policies
Exact verifiers are valuable as diagnostic tools for understanding what training teaches, even when they don't improve training outcomes
More latest news
Read all