LLM Scientists Get Better When You Force Them to Consider Alternatives, Not Just Reject Hypotheses

Contrastive Discovery replaces one-at-a-time hypothesis testing with competing explanation sets, resolving 67% more scientific questions from the same evidence budget.

You know how a good doctor doesn't just run a test to rule out one disease — they hold three or four diagnoses in mind simultaneously, and each test result shifts weight across the whole list? That's the core mechanism here. Most LLM-based discovery agents test one hypothesis at a time: propose h, test h, reject or accept h. When h is rejected, they know what the data ISN'T, but not what it IS. The information that could have pointed them toward an alternative is wasted. Contrastive Discovery fixes this by replacing the single hypothesis with a 'contrast set' — a scientific question paired with mutually exclusive, collectively exhaustive candidate explanations. Every experiment is scored by how much it redistributes belief across the entire set, not just how surprising its result is for one candidate. Resolution happens when belief concentrates on a single explanation. The numbers are clean and the experimental coverage is broad: six domains, 38 human expert evaluators in a blinded study. The headline result is that 89% of the largest belief shifts under pointwise testing are rejections — the agent learns what's wrong but not what's right. Contrast sets recover a specific winning explanation for 73% of those rejected candidates, using evidence the agent already had. When contrast sets actively guide which experiments to run next, the agent resolves 67% more questions than pointwise surprisal under the same experiment budget. Human experts agreed with the system's resolved explanation 78% of the time overall, and 82% on the exact cases where the pointwise agent rejected and stopped. The architecture is lightweight and modular by design. Contrast sets change only what the agent proposes (a question with competing answers instead of a single hypothesis) and how it scores evidence (belief redistribution across a set instead of pointwise surprisal). This means you can bolt contrast sets onto any existing hypothesis-proposing agent without changing the agent's core loop or search strategy. The authors demonstrate this by retrofitting an existing agent and re-scoring evidence it had already collected — no new experiments needed for the 73% recovery result. The information-theoretic framing is the load-bearing intellectual contribution. Scoring an experiment by how much it shifts the full belief distribution across alternatives is a direct application of Bayesian experimental design. What's new is packaging this in a form that LLM agents can actually use: natural-language contrast sets that decompose a question into mutually exclusive branches, scored with a belief-redistribution metric the LLM can compute. The bridge from theory to LLM-native implementation is the genuine contribution. Integrity is solid for a methods paper. The blinded expert study with 38 evaluators across six domains is well above the bar for this kind of work. The 78% agreement rate is reported honestly — it's not 95%, and the paper doesn't pretend otherwise. The comparison baseline is pointwise surprisal, which is the standard operating procedure for most existing discovery agents. Code and a project page are provided. The main integrity concern is that the authors designed both the contrast set framework AND the evaluation — no independent team has stress-tested this yet. The obvious gap is scale and autonomy. Six domains is respectable breadth, but the contrast sets in this paper appear to be human-structured or at least human-seeded. The paper doesn't demonstrate fully autonomous contrast set generation from raw data — the agent isn't discovering the right questions to ask, only resolving pre-formed questions more efficiently. The successor experiment is clear: can an LLM generate high-quality contrast sets from scratch, without a human framing the competing explanations? My read is this is being saved for the next paper — the modular architecture practically begs for a 'contrast set generator' module, and the authors likely have early results.