
Imagine you're a chess coach evaluating students. You give each student the same set of practice puzzles and the same reference books — not their own library, yours. If one student scores higher, you know it's their reasoning, not their bookshelf. Now imagine you discover that swapping the reference books changes scores more than swapping students. That's this paper's punchline: the "books" (web search results) matter as much as the "student" (the LLM). ForecastGym's committed claim is that fair evaluation of LLM-based forecasting agents requires frozen, time-stamped evidence — and that when you do freeze evidence, search source variation dominates model variation. The authors collected what four commercial search APIs returned every two days for 150 open prediction-market questions, accumulating 614 evidence states and 131,000 search results. Once outcomes resolved, they replayed this frozen evidence to multiple LLMs under identical conditions. The ladder result is sobering: every tested LLM trails market-consensus accuracy on Brier score. No model beats the prediction market. But the diagnostic is sharper than a simple scoreboard. Switching the search API changes a model's Brier score by up to 0.043 — a spread larger than the differences between models themselves. That's the equivalent of discovering your telescope matters more than your astronomer. The architecture is straightforward: a retrieval-then-reason pipeline where the retrieval step is frozen into a replayable corpus. The key methodological contribution is a protocol, not a model. The system also tests a "persistent notes" memory strategy against full-context rereading, finding comparable accuracy at half the input tokens — a practical efficiency win for deployed agents. The integrity story has a genuinely useful contribution: the authors demonstrate that rerunning the same search queries after a question resolves leaks the answer up to 92% of the time, despite date filters. This is the paper's strongest integrity move — it names the contamination problem everyone suspected but few had quantified. The diagnostic analysis attributes more of the accuracy gap to missed public information (evidence the agent never saw) than to misreading (evidence the agent saw but interpreted incorrectly). The milestone question is about scale and coverage. 150 questions and 614 evidence states is a proof of concept, not a community benchmark. The next real number is on the order of 1,000+ questions across diverse domains with continuous collection over a full year. That would let the community distinguish domain-specific retrieval effects from general retrieval noise. The obvious next experiment not run: testing open-source or fine-tuned models with retrieval-augmented generation loops that adaptively re-query based on what the first search returned. The authors held the retrieval policy fixed by design — that's the whole point of the benchmark — but the natural follow-up is to let agents choose their own queries within the frozen corpus. My read: they're saving this for the next paper, since the benchmark infrastructure makes it trivial to add.
The analogy
You're a chess coach evaluating students. You give each one the same reference books and the same puzzles — not their own library, yours. If swapping the books changes scores more than swapping students, the books are the bottleneck, not the talent. ForecastGym freezes the 'books' (web search results) so you can finally measure whether it's the LLM or the evidence doing the work.
Chess coaching
Give every student the same reference books and puzzles to isolate reasoning skill from library quality
ForecastGym
Give every LLM the same frozen search results and questions to isolate reasoning from retrieval quality
Honest assessment
ForecastGym is a methodological contribution, not a modeling breakthrough — and that's fine. The frozen-evidence protocol and the 92% leakage quantification are genuinely useful infrastructure for anyone building or evaluating forecasting agents. The headline finding that search API choice matters more than model choice is a cold shower for the 'just use a better LLM' crowd. The main limitation is scale: 150 questions is a proof of concept, and community adoption will determine whether this becomes a standard or a one-off.
What Is CLAIMS?
67
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
3/5
Ladder
4/5
Architecture
3/5
Integrity
3/5
Milestone
3/5
Successor
4/5
What is this paper actually committing to?
Real contribution but more benchmark than breakthrough. Demonstrates retrieval dominance over model choice — a useful finding, not a paradigm shift. The frozen-evidence protocol is the genuine novelty.
The field fight
Is the bottleneck in LLM forecasting the model's reasoning or the evidence it retrieves?
Side A
Better models with stronger reasoning will close the gap to human/market accuracy
Held by: OpenAI forecasting demos, Anthropic Claude forecasting benchmarks
Side B
Retrieval quality and evidence coverage are the binding constraint; model improvements are secondary
Held by: ForecastGym, retrieval-augmented generation skeptics
Milestone tracker
Unlocks: Statistically robust evidence that retrieval dominance holds across domains, enabling principled search-API selection for deployed forecasting agents
Current
150
questions
ForecastGym v1
Next
1000+
questions
Community benchmark
~1-2 years if adopted by forecasting community
Verified Facts (6)
First release covers 150 questions, 614 evidence states, one per question and collection time, and 131k search results
Rerunning the same queries after settlement returns results that all but state the outcome for up to 92% of the questions, despite the tested date filters
On the frozen evidence, all tested LLMs trail the market in forecasting accuracy
Switching the search API changes a model's Brier score by up to 0.043, more than the spread across models
Persistent notes score similarly to rereading all evidence at half the input tokens
Evidence collected from four commercial web search APIs every two days
Practical Implications (4)
Teams deploying LLM forecasting agents should invest as much in search API selection and evidence pipeline as in model choice — the evidence bottleneck is real and measurable.
Persistent-notes memory strategies can halve input tokens with comparable accuracy to full-context rereading — a direct cost savings for deployed systems.
Retrospective search evaluation is contaminated up to 92% of the time despite date filters — any benchmark using post-hoc search is suspect.
Prediction-market consensus remains a hard baseline to beat; teams should calibrate expectations accordingly.
More latest news
Read all