Imagine you're a restaurant owner who hired six sous chefs and told each: "Read this recipe book, then propose the single dish we should add to the menu." Every chef proposes several dishes, and some are genuinely good — but when you ask them to RANK which dish will sell best, their top pick isn't reliably better than just cooking whatever they wrote down first. They have creativity. They lack taste. That's the core finding of rTaste. Art Kanke hands seven recent NLP papers to six frontier language models (working solo and in teams) and asks each to propose minimum viable experiments (MVEs) — small, testable extensions of the seed paper. A separate executor model attempts every single MVE, including the ones ranked last, and three model auditors grade the results. Of 415 total MVEs, 194 are judged sound, valuable, and new. But only 73 of those — fewer than half — are judged both beyond the seed paper AND non-obvious. The agents can generate; they struggle to discriminate. The ranking results are the paper's sharpest contribution. Agents' first-choice experiments match the auditors' most-valued experiment more often than a random pick — but not more often than simply selecting the experiment the agent happened to write down first. That's a devastating control. It suggests the models' ranking step adds no measurable signal beyond whatever implicit ordering is baked into generation order. Their explicit forecasts (will this MVE pass a threshold?) do beat a base rate, but aren't more accurate than a fresh model instance given only the paper and registrations — so the forecasting isn't an emergent property of the agent pipeline either. Teams of agents cost more tokens without clear improvement in proposal quality over solo agents. This is a useful negative result for the agentic-AI-for-science community, which has been betting heavily on multi-agent collaboration as a scaling axis. If team deliberation doesn't sharpen taste at this scale, the bottleneck isn't coordination — it's something about how these models represent epistemic uncertainty over experiment outcomes. The evaluation harness itself is well-designed: every MVE is executed (no cherry-picking of only top-ranked proposals), three auditors judge outcomes, and the full dataset of 415 MVEs with forecasts and audited outcomes is released. The main integrity concern is circularity — models propose, models execute, models audit. There's no human-in-the-loop validation of the auditors' judgments, so the entire pipeline grades itself. The paper acknowledges this but doesn't resolve it. The practical upshot for anyone building research agents: generation is cheap and these models are decent at it. But if you're building a system that needs to triage which experiments to run under a compute budget, the current ranking signal from frontier models is weak enough that you might do just as well with a FIFO queue. The value of agentic research pipelines may live in volume and coverage, not prioritization. The dataset of 415 MVEs is arguably the paper's most durable artifact — a benchmark for future work on experiment selection, taste evaluation, and research-agent calibration. Whether anyone builds on it depends on whether the field decides taste is a tractable research target or an irreducible property of domain expertise that models approximate poorly.
