Imagine you're a scout for a baseball team. You can identify which positions need filling and which minor league systems are producing talent — you see the landscape clearly. But when it comes to picking the specific player who'll break out next season, you're wrong four out of five times. That's what IdeaTaste reveals about LLMs as research strategists: they read the map but can't navigate the last mile. The core claim is a diagnostic, not a capability. Zhong et al. freeze a literature snapshot at early 2025, hand it to six contemporary LLMs, ask each to generate research proposals, then score those proposals against what human communities actually published in 2026. The benchmark covers four research areas from established to emerging, generating over 90,000 proposals total. The gap is stark: LLMs achieve roughly 26% of human-level performance on citation matching and only 18% on idea matching. They can identify the right tasks, methods, and even cite as diversely as a human community — but they rarely land on the concrete approaches and research plans that humans actually pursue. The architecture here is straightforward agentic prompting: LLMs consume a structured literature context, generate proposals, and those proposals are scored against a ground-truth publication set using both citation overlap and semantic idea matching. There's no fine-tuning, no RL, no special training signal — this is purely a probe of what's already inside the model's latent space. The interesting structural choice is treating the benchmark as a population-level measurement: you don't ask one model for one proposal, you ask for thousands and compare the distribution against the human community's distribution. The ladder here is unusual because there's no prior benchmark doing exactly this. The closest relatives are SciMON, ResearchAgent, and AI-Scientist-style systems that generate or evaluate research ideas, but none use future publications as ground truth in a time-split design. That temporal separation is the load-bearing innovation — it turns 'research taste' from a subjective judgment into a measurable quantity. The 26% and 18% numbers have no direct predecessor to beat, which makes them a baseline rather than a delta. Integrity is solid for a benchmark paper. The time-split design (2025 freeze, 2026 ground truth) is a natural pre-registration of the test set — the answers literally didn't exist when the models were trained. Blinded expert review adds a second validation layer, and the finding that AI proposals trail human papers most on 'expected success' and least on 'presentation' is a useful calibration. The weakness is that 'idea matching' is itself a metric that requires validation — how well does semantic similarity capture genuine intellectual overlap versus surface-level keyword alignment? The most revealing finding isn't the headline numbers but the structure of the failure. Richer field context and larger population sizes don't substantially close the gap. This suggests the bottleneck isn't information access — the models have the context — but something about converting landscape awareness into specific, viable research plans. In blinded review, proposals whose ideas matched no human paper scored worst overall, suggesting these weren't hidden gems the community missed but genuinely weaker directions. That's a sobering finding for anyone hoping LLMs will discover overlooked research opportunities. The benchmark is released as a continuously updated leaderboard, which is the right move — as models improve, the 26% and 18% numbers become trackable over time. The real question IdeaTaste poses isn't whether LLMs can replace research taste today (clearly not) but whether the gap is compressible. If the bottleneck is in concrete planning rather than direction-finding, that's potentially addressable through better chain-of-thought, tool use, or iterative refinement. If it's something more fundamental about how novelty is generated versus predicted, the gap may prove stubborn.
