Imagine you're a teacher with 30 students and only enough time for 100 practice problems. Some students already ace multiplication but can't touch long division. A uniform approach — 50 multiplication, 50 division — wastes half your budget on kids who don't need it. The smart move is to diagnose who's struggling where and send the hardest problems to the students who need them most. That's PASS. The paper's committed claim: pretrained LLMs carry a measurable "prior barrier" for each concept — how strongly the model's existing knowledge supports wrong answers over the right one — and this barrier follows a long-tail distribution. Head concepts (frequent during pretraining) have low barriers; tail concepts (rare) have high barriers. SFT instruction selection should account for this asymmetry rather than treating all concepts equally. The theoretical contribution is a predictive risk bound that decomposes SFT performance into two terms: the prior barrier magnitude and the accumulated fine-tuning evidence. This isn't just a heuristic observation — they derive conditions under which more evidence is needed for high-barrier concepts, giving the adaptive allocation a formal foundation. The bound explicitly characterizes how many instructions you need per concept as a function of its prior barrier height. PASS works by constructing "reference-derived concepts" from the instruction pool — essentially clustering the training data by what distinguishing evidence each example provides — then estimating how much evidence the current selection already supplies for each concept. The selection budget is then shifted toward concepts that remain insufficiently covered. It's greedy but principled: at each step, the method picks the instruction that most reduces the worst coverage gap. The experimental setup is reasonably thorough: seven baseline instruction selection methods (including Alpagasus, IFD, LESS, and others from the 2023-2024 wave), tested across four backbone-budget configurations. PASS consistently outperforms all seven. The ablation study isolates the adaptive allocation mechanism by comparing against uniform allocation with the same reference-derived concepts, confirming that the allocation strategy — not just the concept representation — drives the gains. The architecture sits in the broader family of data-centric AI methods, specifically curriculum learning and coreset selection adapted for LLM fine-tuning. The key computational dependency is estimating prior barriers, which requires forward passes through the pretrained model to score competing concept likelihoods. This is a one-time diagnostic cost, not a training overhead, making it practical at moderate scale. The honest limitation: we're seeing this on instruction-tuning benchmarks, not on deployment-scale fine-tuning with tens of thousands of proprietary concepts. The gap between "beats LESS on MT-Bench" and "changes how Anthropic or OpenAI selects fine-tuning data" remains wide. The prior barrier framework is elegant, but its practical impact depends on whether the long-tail structure persists when you move from academic benchmarks to real enterprise fine-tuning distributions.
