Error-Correcting Codebooks Make LLM Watermarks Work on Short Texts — but Instruction-Tuned Models Fight Back

Codebook-search watermarking recovers 16-24 bit payloads from just 200 tokens with near-perfect accuracy. The catch: low-entropy instruction-tuned outputs resist embedding.

Imagine you're playing Battleship, but instead of calling out single grid squares, you shout entire row-column patterns that your opponent can decode even if a few squares get garbled. That's the core mechanism here: instead of embedding a raw n-bit message into generated text one bit at a time, you first encode the payload into a much longer u-bit codeword — one that's been chosen from a secret codebook precisely because every entry is maximally distant from every other entry. When noise corrupts a few bits during extraction, you just look up the nearest codeword and recover the original payload perfectly. The committed claim: codebook-based multi-bit watermarking recovers 16-24 bit payloads from 200-token LLM generations with near-perfect accuracy, outperforming direct embedding and BCH error-correction variants on short texts. This isn't the first multi-bit LLM watermark — it's a meaningful engineering improvement in the payload-vs-length tradeoff, the specific chokepoint that makes short-text attribution impractical with prior methods. The n-to-u expansion (e.g. 16 bits → 64 or 128 bits) gives you Hamming distance slack that raw embedding cannot. The architecture sits squarely in the logit-biasing family of watermarking: during token generation, the scheme partitions the vocabulary into green/red lists conditioned on preceding tokens and the target codeword bit, then biases logits to favor green-list tokens. This is a direct descendant of Kirchenbauer et al.'s (2023) green-list watermarking, extended with the error-correction layer from coding theory. The codebook itself is precomputed and stored as a shared secret — roughly 2^n entries of u bits each — making it a lookup-table approach rather than an algebraic one. The BCH variant trades that storage for on-the-fly parity checks but loses robustness in practice. On the ladder: the paper benchmarks against its own direct-embedding baseline and the BCH alternative, showing the codebook variant consistently recovers payloads where the others fail at short lengths. What's missing is head-to-head comparison against the strongest competing multi-bit schemes — SemStamp, MPAC, or the balance-marking approach from Christ et al. The authors acknowledge instruction-tuned models (Llama-3.1-8B-Instruct, Phi-3.5-mini-instruct) degrade performance substantially on 127-token texts because low-entropy output distributions leave fewer bits available for biasing. This is an honest admission, not buried — but it's also the regime where watermarking matters most (chatbot answers, short-form content). Integrity is decent but not airtight. The experiments use standard open-weight models (Llama-3.1-8B, Phi-3.5-mini, and their instruction-tuned variants) with C4 dataset prompts, and code is released on GitHub at submission. However, all evaluation is self-contained — no independent replication, no pre-registration, and the benchmark prompts were presumably chosen during development. The text quality analysis (perplexity ratios, zero-shot accuracy) is reasonable but limited to the authors' own pipeline. The paper is forthright about failure modes: insertion attacks defeat even the Levenshtein variant, and stronger paraphrasing collapses recovery rates. The milestone that matters: reliable multi-bit watermark recovery at ≤100 tokens on instruction-tuned models. This paper gets near-perfect at 200 tokens on base models but drops to ~60-85% accuracy at 127 tokens on instruct models for 16-bit payloads. Closing that gap — hitting >95% at 100 tokens on instruct models — would make codebook watermarking deployable for chatbot attribution, the commercially relevant use case. That's probably 1-2 iterations away if someone combines this with entropy-adaptive embedding. The experiment they didn't run: adversarial robustness against modern paraphrasers (GPT-4-level rewriting, not just light substitution). The paper tests word-level deletions and light edits but stops short of heavy neural paraphrasing. Honest read: they likely ran preliminary tests and the numbers weren't good — the Levenshtein variant helps with deletions but insertions and rewrites break the token-alignment assumption that the whole scheme depends on. This is the hard problem, and acknowledging it is more useful than hiding it.