You know how a poker player might claim they haven't seen a card, but their betting pattern says otherwise? That's exactly what frontier LLMs are doing with financial data. They insist they don't know last quarter's Fed rate decision — but when you ask them to pick the right number, they nail it. This paper builds the instrument that catches the bluff. The core claim is uncomfortable for anyone running LLM-based backtests: models systematically understate their own knowledge of historical financial facts. The authors introduce FinDendro, a benchmark of 2,226 dated U.S. financial facts spanning 1999–2026, covering macroeconomic series, month-end stock prices, and Federal Reserve decisions. Each fact gets four prompt variants — a raw value question, Recall and Predict framings without a declared date, and a Predict framing with a historical declared date. Controls for persistence and dominant outcomes pin chance accuracy near 27%. The result is a clean behavioral measurement of what the model actually knows versus what it says it knows. The numbers are stark. GPT-5 answers 76% of controlled facts correctly but claims to know only 38%. Worse, it answers 61% of the facts it explicitly says it does not know. This isn't a quirk of one model — the pattern holds across frontier systems. Knowledge claims can persist up to seven quarters beyond the measured knowledge boundary, meaning a model might claim familiarity with data it demonstrably cannot retrieve. The three safeguards practitioners rely on — vendor-stated cutoff dates, model self-reports, and date-instruction prompts — all fail in different ways. The declared-date trick is particularly revealing. When you tell a model to "answer as of March 2022," it dutifully stops claiming prior knowledge of post-cutoff events — but its actual accuracy doesn't change. The instruction suppresses the self-report without suppressing the knowledge. This is a dissociation between the model's metacognitive outputs and its factual retrieval, and it means you cannot trust the model to police itself. Architecturally, this is a prompt-engineering and evaluation paper, not a new model architecture. The contribution is the benchmark design and the controlled experimental methodology. The four-prompt design per fact — value, Recall, Predict-no-date, Predict-with-date — is the key structural choice, enabling direct comparison between behavioral knowledge and declared knowledge within the same model on the same fact. The 27% chance baseline, enforced by controlling for temporal persistence and dominant outcomes, is what makes the accuracy numbers interpretable rather than inflated by lucky guessing on stable time series. The integrity regime is solid for an evaluation paper. The benchmark covers 27 years of public financial data, the prompts are systematically constructed, and the controls address the most obvious confounds (a model that always guesses "rates stayed the same" would score well without knowledge — the persistence controls prevent this). The main limitation is that this is a same-team evaluation with no independent replication yet, and the benchmark facts are drawn from public data that could, in principle, be gamed by future model providers. No pre-registration. The practical implication is direct: if you're running financial backtests using LLMs, you cannot certify them as leak-free by asking the model, reading vendor documentation, or using date-instruction prompts. You need behavioral measurement — probe the model with known facts and find the boundary where accuracy drops to chance. FinDendro gives you the instrument. The successor question is obvious: test this on non-U.S. markets and non-financial domains where training data contamination is equally dangerous but less studied. The authors likely chose U.S. financial data because it's clean, timestamped, and unambiguous — extending to noisier domains is harder and probably paper two.
