You know that friend who plays chess at exactly one level? They learned the Sicilian Defense, they play the Sicilian Defense, and no matter what you do on the board — sacrifice a bishop, play an irregular opening, stall for time — they keep executing the Sicilian. They're not reading you. They're running a cached program. That's what LLMs do in behavioural game theory, and this paper nails the diagnosis with unusual precision. The committed claim: LLMs are static level-k players, where k — the depth of strategic reasoning — is set by model scale and then frozen. A 0.5B model plays like a level-0 randomiser. A 32B model plays like a level-2 or level-3 thinker. But none of them update beliefs within the game. The level-k framework from cognitive economics says a level-0 player picks randomly, a level-1 player best-responds to level-0, a level-2 best-responds to level-1, and so on. Humans distribute across these levels and — crucially — shift between them as a game unfolds. LLMs pick a level at deployment and stay put. The experimental design is the strongest part of the paper. Teo runs a 360-cell factorial across five deployment axes: temperature, model scale (0.5B to 32B), quantisation, instruct-vs-base, and prompt framing — all within a single model family. Two classic games serve as testbeds: the p-beauty contest (where you try to guess 2/3 of the group average) and the public goods game (where you decide how much to contribute to a shared pot). Each cell's output distribution gets compared against published whole-distribution human data, not just means. This is the right comparison — matching distributional shape, not just central tendency. The results are cleanly damning. Deployment settings like temperature and framing can recover the statistical dispersion of human choice distributions — you can make an LLM's outputs look spread-out like human data. But the strategic process underneath is wrong. In the public goods game, human contributions decay over rounds as players learn that free-riding pays. LLM contributions stayed flat or actually increased at every scale tested. When given a finite vs. indefinite horizon, LLMs were more cooperative under indefinite horizons (matching the textbook category) but completely ignored their position within the game — no last-round defection, no endgame calculation. They retrieved the right category label ('indefinite horizon = cooperate more') without doing the actual backward induction that would produce round-sensitive play. The ladder here is interesting because the baseline isn't another algorithm — it's human subjects. Prior work by Aher et al. (2023), Horton (2023), and Argyle et al. (2023) established the 'LLM-as-homo-silicus' paradigm: use LLMs as cheap stand-ins for human experimental subjects. This paper doesn't beat that paradigm — it stress-tests and partially breaks it. The match is surface-level (distributional shape can be tuned) but mechanistically hollow (no belief updating, no backward induction, no round-sensitivity). That's a meaningful negative result against a paradigm that's gaining real traction in computational social science. The architecture story matters for anyone building LLM agents. These are autoregressive transformer models (one family, 0.5B–32B parameters) running standard next-token prediction. The paper shows that scale shifts the cached strategy level monotonically — bigger models retrieve 'deeper' game-theoretic reasoning patterns from training data. But retrieval is the operative word. The models aren't computing game-theoretic equilibria; they're pattern-matching to the closest training-data category. Quantisation, notably, had no significant effect on strategic behaviour, which is useful to know if you're deploying smaller quantised models for agent applications. The integrity profile is solid for what it is — a behavioural comparison study, not a benchmark-beating race. The 360-cell factorial with controlled axes is well-designed. The comparison against published human data (not the authors' own human subjects) adds independence. The main vulnerability is that we're seeing one model family. If the level-k retrieval pattern holds across architectures (Llama, Mistral, GPT), it's a robust finding about transformers. If it's family-specific, it's an artifact. That cross-family test is conspicuously absent.
The analogy
You know the friend who plays chess at exactly one level — they learned the Sicilian Defense and they play the Sicilian Defense regardless of what you do on the board? They're not reading your moves; they're running a cached program. That's LLMs in game theory: they retrieve a strategy depth from training data based on their size, then execute it without updating. A 7B model plays like someone who read the Wikipedia intro to game theory. A 32B model plays like someone who read the textbook chapter. Neither one is actually thinking about what YOU just did.
Chess memoriser
Learns one opening from a book, executes it regardless of opponent moves, never adapts mid-game
LLM level-k player
Retrieves a strategic depth from training data based on scale, executes it regardless of opponent play or round position, never updates beliefs
Honest assessment
This is a clean, well-designed negative result that matters to anyone building LLM agents for strategic settings or using LLMs as experimental subjects. The core finding — scale sets the strategy tier, nothing updates it mid-game — is specific, testable, and practically important. The single-family limitation is real but addressable, and the 360-cell factorial shows rigour that's unusual for a single-author paper. If you're deploying LLM agents in auction, negotiation, or mechanism-design settings, this paper should change how you think about what your agent is actually doing.
What Is CLAIMS?
70
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
4/5
Ladder
4/5
Architecture
3/5
Integrity
3/5
Milestone
3/5
Successor
4/5
What is this paper actually committing to?
Real characterisation result, not just a benchmark. Names the failure mode precisely: static category retrieval, not iterative strategic reasoning. Genuine contribution to the homo-silicus debate, but builds on an established framework (level-k).
The field fight
Can LLMs serve as valid stand-ins for human subjects in behavioural experiments (homo silicus)?
Side A
Yes — LLM choice distributions meaningfully approximate human play, enabling cheap, fast, scalable behavioural experiments
Held by: Horton (2023), Aher et al. (2023), Argyle et al. (2023)
Side B
No — distributional match is surface-level; the strategic reasoning process diverges fundamentally, making LLMs unreliable proxies for mechanism-dependent questions
Held by: This paper (Teo)
Milestone tracker
Unlocks: Confirmation that static level-k retrieval is a transformer universal, not a family artifact — would reshape how computational social science uses LLMs
Current
1
model family
this paper
Next
3+
model families
cross-architecture replication
~1-2 years, compute is modest per family
Verified Facts (7)
The study investigates five local-model settings within the same model family varied in a 360-cell factorial balancing temperature, scale (0.5-32B), quantisation, instruct vs base, and framing
Two games tested: p-beauty contest and public goods game
LLMs act as static, category-retrieved level-k players, where k is set by the model scale
While human contributions decayed in the public goods game, LLMs stayed flat or rose at every scale
LLMs were more cooperative under an indefinite horizon compared to a finite one
LLMs ignore their relative round position, so no last-round defection was displayed
Quantisation was the only deployment setting that did not govern an aspect of fidelity
Practical Implications (4)
LLM agents in auction, negotiation, or mechanism-design systems should not be assumed to perform backward induction or adapt strategy to opponent behaviour — they will execute a fixed strategy tier regardless of game dynamics
If you're using LLMs as synthetic subjects in behavioural experiments, distributional match on aggregate statistics does NOT validate process-level fidelity — your LLM subjects may produce the right histogram for the wrong reasons
Model scale is the primary lever for strategic depth in LLM agents — temperature and framing adjust dispersion but not the underlying reasoning tier
Quantisation does not significantly affect strategic behaviour, which is good news for deploying smaller quantised models in agent applications where cost matters
More latest news
Read all