
You know those movie props that look real from the front but are hollow plywood from behind? A film set only needs to fool the camera angle. Traditional honeypots work the same way — they emulate a few services just well enough to log an attacker's first three commands, then the illusion collapses the moment someone runs `ls -la /proc` or chains a pipe. ShellGames asks: what if the prop could improvise? The committed claim: an LLM-driven SSH simulator that maintains believable interactive shell sessions across extended attacker engagement, solving the five problems that make naive LLM honeypots collapse — statelessness, hallucination, latency, inconsistency, and susceptibility to probing attacks designed to unmask the deception. This is not the first paper to put an LLM behind a fake shell. It is the first to systematically engineer around each failure mode and benchmark the result with a standardized protocol. The architecture is a hybrid: an LLM (the paper doesn't name which, but the techniques are model-agnostic) augmented by five layered systems. Automatic Chain-of-Thought plus few-shot prompting handles correctness. A memory management module maintains filesystem and process state across commands — this is the key structural choice, because without it every `mkdir` is forgotten by the next `ls`. Speculative command execution pre-computes likely follow-up responses to mask LLM inference latency. Complex interactive commands (think `vim`, `top`, `ssh` tunnels) get routed to a real sandboxed environment rather than hallucinated. And a subversion detector watches for adversarial inputs designed to break character — prompts like "ignore all previous instructions" or commands probing for LLM-specific tells. The numbers tell a clear story against baselines. On command-level correctness, ShellGames scores 0.898 accuracy — a 5.3 percentage-point improvement over the baseline. The real gap opens on harder tasks: 0.918 on sequence-level consistency (+36pp), 0.98 on state tracking (+18.3pp), and 0.95 on robustness to adversarial subversion (+37pp). Those last three numbers are where naive LLM approaches fall apart. Consistency means the system remembers what it told you three commands ago. State tracking means `touch file.txt && ls` actually shows `file.txt`. Robustness means an attacker asking "are you an AI?" doesn't get a confession. The user study (n=20) adds a crucial reality check beyond automated benchmarks. Participants interacted with both ShellGames and a real shell under free exploration — and rated ShellGames' realism as comparable to the real thing. It also outperformed traditional honeypots on perceived command coverage. Twenty participants is small but appropriate for this kind of qualitative evaluation; the automated benchmarks carry the quantitative weight. The integrity picture is mixed. The authors introduce their own benchmarking protocol and dataset — necessary because no standardized benchmark existed for this task, but also means they're grading their own homework. The four evaluation dimensions (correctness, consistency, state tracking, robustness) are well-chosen and independently meaningful. The baselines appear to be vanilla LLM approaches without ShellGames' five augmentations, which is the right comparison for an ablation-style argument but leaves open whether other engineered LLM honeypot systems exist that weren't compared. The field fight here is whether deception systems should be static rule-based emulations or dynamic LLM-driven simulations. ShellGames lands firmly on the LLM side but hedges smartly by routing genuinely complex commands to real sandboxed environments — acknowledging that pure generation isn't enough. The obvious next experiment they didn't run: deploying ShellGames against real attackers in the wild rather than study participants. My read is (c) — they're saving it for the next paper, because a production deployment would be a separate publication with IRB considerations and operational security constraints that take time to navigate.
The analogy
You know movie sets — convincing from the camera angle, hollow plywood behind? Traditional honeypots are the same: they fool the first three commands, then collapse. ShellGames builds a set that can improvise when the actor goes off-script, with a real room behind the door for scenes that need it.
Movie set
Convincing facade from one angle, with real rooms behind select doors for scenes that demand depth
ShellGames
LLM-generated shell responses for most commands, with sandboxed real execution behind complex interactive ones
Honest assessment
ShellGames is a well-engineered systems paper that solves five real problems with LLM-based honeypots and benchmarks the solutions rigorously against ablated baselines. The consistency and robustness gains (+36pp and +37pp) are the load-bearing numbers — those are where naive LLM approaches actually fail in practice. The main weakness is that all evaluation is lab-based with author-defined benchmarks and a small user study. The paper that matters is the next one: real attackers, real networks, real session durations. If that replicates, this becomes the foundational architecture for adaptive cyber deception.
What Is CLAIMS?
67
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
3/5
Ladder
3/5
Architecture
4/5
Integrity
3/5
Milestone
3/5
Successor
4/5
What is this paper actually committing to?
Solid engineering claim, not a first-of-kind discovery. The novelty is systematic: five named techniques addressing five named LLM failure modes for shell deception. Real contribution, but iteration not revolution.
The field fight
Should cyber deception systems use static rule-based emulation or dynamic LLM-driven generation?
Side A
Static emulation is predictable, auditable, and doesn't hallucinate — LLMs add attack surface
Held by: Cowrie, Kippo, traditional honeypots
Side B
LLM generation enables adaptive, long-lived deception that static systems can't sustain
Held by: ShellGames, this paper, LLM honeypot research line
Milestone tracker
Unlocks: Production-grade adaptive honeypots that waste real attacker time at scale
Current
0.898
cmd accuracy
ShellGames (lab)
Next
100+
hours mean session
Real attacker deployment
~1-2 years, gated by IRB and operational partnerships
Verified Facts (7)
ShellGames achieves 0.898 command accuracy on correctness (+5.3pp over baselines)
0.918 sequence-level accuracy on consistency (+36pp)
0.98 state tracking accuracy (+18.3pp)
0.95 accuracy on robustness (+37pp)
User study with n=20 participants
Five techniques: Automatic Chain-of-Thought and few-shot learning, memory management, speculative command execution, smart routing to sandboxed environment, subversion detection
Authors: Umberto Salviati, Fabio De Gaspari, Mauro Conti, Luigi Vincenzo Mancini
Practical Implications (4)
SOC teams could deploy ShellGames as a high-interaction honeypot that sustains attacker sessions long enough to capture TTPs, without maintaining real vulnerable hosts
The speculative execution technique for masking LLM latency is applicable to any real-time LLM system where response time matters — chatbots, coding assistants, interactive agents
The subversion detection module addresses a general problem: preventing prompt injection in constrained-domain LLM applications where valid inputs have known structure
Honeypot operators can use the benchmarking protocol as a standardized evaluation framework for comparing future LLM deception systems
More latest news
Read all