LLMs Can Judge Good Research Ideas but Can't Generate Them — and a Twenty Questions Game Proves It

A clever diagnostic game reveals that LLM ideation plateaus after ~30 questions, and the bottleneck is generation, not judgment. The fix? Feed them external candidates to evaluate.

Imagine you're playing Twenty Questions, but instead of guessing "elephant," you're trying to rediscover the core idea behind a published machine learning paper. You only know the research problem — the setup, the gap, the motivation — and you have to ask yes-or-no questions to an oracle who has read the answer paper. That's Re-Search, and it turns out to be a brutally clean diagnostic for how LLMs actually think when they ideate. The committed claim: LLMs plateau early and hard during iterative research ideation, and the bottleneck is generation — not judgment. Across six frontier models tested on 200 ICML 2026 papers with up to 100 yes-or-no questions each, nearly all meaningful progress happens in the first 30 questions. After that, models flatline. They stop discovering new ground not because they can't recognize a good idea when they see one, but because they can't produce the right question or hypothesis to move forward. The game design is the real contribution. By reducing ideation to a binary-feedback search over idea space, Re-Search isolates the generation-vs-judgment question with surgical precision. The authors run controlled interventions: when you hand a model a set of candidate questions that includes a good one, it picks correctly. When you expand its hypothesis space with externally sourced ideas, it improves. The model's evaluator is fine; its generator is the weak link. This is a specific, testable, and architecturally actionable finding. The failure modes are instructive. Models either over-exploit — drilling into one narrow direction until they exhaust it — or over-explore, scattering attention across too many threads without committing. Both patterns waste questions and lead to the same plateau. The trace analysis across 200 papers gives this finding statistical weight rather than anecdotal flavor. On the ladder: there isn't a clean prior benchmark to compare against because Re-Search is a new evaluation format. The closest relatives are IdeaBench and ResearchArena, but neither isolates the iterative search process the way this game does. The paper's contribution is primarily diagnostic infrastructure, not a performance claim against SOTA. That's honest, but it also means the "result" is a behavioral observation about models, not a metric you can track improving over time. The integrity picture is mixed. The 200-paper dataset from ICML 2026 is well-scoped and recent enough to avoid training-set contamination concerns. Six models tested gives breadth. But the oracle is a prompted LLM (not a human expert), which introduces circularity — an LLM judges whether another LLM's question is on the right track. The authors acknowledge this, but it's a real limitation. Code and data availability aren't confirmed in the abstract. The actionable takeaway for anyone building autoresearch systems: stop trying to make a single model generate better ideas in isolation. Instead, build retrieval-augmented ideation pipelines that surface external candidate ideas and questions for the model to judge. The generation bottleneck is real and may be structural — it's cheaper to improve the search corpus than to close the generation gap through scaling alone.