Shibboleth Traps: Honeypots That Fingerprint Which AI Agent Is Hacking You — and What Harness It Rides

Researchers design honeypot services that force AI attackers to reveal their LLM and harness through elicited behavior, hitting 90% model ID accuracy across 15 candidates.

Imagine you run a speakeasy and you suspect someone at the door is wearing a disguise. You don't rip off the mask — you ask them to pronounce a word only locals know. The way they say it tells you where they're really from. That's the core mechanism here: instead of trying to detect AI attackers by their exploit payloads, you design honeypot services that force them to produce linguistically and behaviorally distinctive outputs as a condition of getting what they want. The paper calls these "shibboleth traps," and the name is precise. The committed claim: by combining declarative traps (that elicit model self-identification and characteristic writing style) with behavioral probes (that reveal execution quirks of the surrounding harness), you can reliably identify both the LLM version and the orchestration framework behind an agentic cyberattack. Across 15 candidate models, the system achieves 90% top-1 accuracy on model identification. Across four harnesses, fixed probe rules hit 92% accuracy on harness fingerprinting. These are strong numbers for attribution in a domain where attribution is notoriously difficult. The architecture sits in the honeypot/deception-defense family, but the innovation is in what the honeypot elicits. Traditional honeypots log attack traffic passively. Shibboleth traps are active: they gate access to resources the agent seeks behind interactions that require the agent to produce text, make scheduling decisions, handle timeouts, and process masked secrets. The behavioral probes are particularly clever — they don't require prompt injection to work. They observe how a harness masks sensitive data in tool output, enforces command time limits, and schedules tool calls, all from side effects visible in subsequent requests. This means the defense works even against agents hardened against prompt injection. The integrity picture is encouraging but has gaps. The 90% model accuracy is reported as an average across 15 models, and the behavioral classifiers are evaluated on held-out harnesses or task families — a meaningful generalization test. The harness fingerprinting uses a separate experiment with fixed probe rules across four harnesses. The paper explicitly tests the adversarial case: when agents are instructed to conceal their identity, declarative signals degrade but behavioral evidence persists. That's the right experiment to run. What's missing is any independent replication, any pre-registration, and any reporting on whether the trap designs were iterated after seeing initial results — a potential source of overfitting the traps to the models tested. The ladder here is unusual because there isn't a deep existing baseline for LLM-agent fingerprinting specifically. Prior work on LLM watermarking and stylometric attribution exists, but fingerprinting agentic attackers through interactive honeypots is genuinely new territory. The closest relatives are passive honeypot systems and LLM output stylometry, neither of which combines declarative and behavioral channels in this way. The 90% and 92% numbers don't have a direct prior-art comparison to beat — they establish a new baseline. The milestone that matters is whether this scales to adversarial adaptation. Today's 90% accuracy is against agents that aren't specifically trained to evade shibboleth traps. The real test is a red-team loop: attackers learn the trap designs and adapt, defenders update traps, and the accuracy over rounds stabilizes at some equilibrium. The paper's concealment experiment is a first step, but the adversary had limited instruction, not architectural countermeasures. The next concrete number to watch: accuracy retention against agents with explicit anti-fingerprinting in their system prompts and harness configurations. The obvious experiment not run is a full red-team adversarial loop with trap-aware attackers. The honest read: this is a v0.1 paper (the authors say so explicitly), and the adversarial robustness experiment would require substantially more compute and a separate attacker-development effort. They likely plan it for v0.2 or a follow-up. The other gap is scaling the model candidate set — 15 models is enough to prove the concept, but real-world deployment faces dozens of models and rapid version churn. Whether the behavioral signals remain discriminative as the model zoo grows is an open question the authors acknowledge implicitly by fixing a candidate set.