RATIOS WITHIN
+1%
EAGER STEP
+4%
STEP
+9%
EAGER PATH RUNS
-9%
Imagine you're storing two versions of a document — one sorted alphabetically, one sorted by date — because your software expects both. Then someone notices that one sort order is just a simple transformation of the other: you only need to store it once, and derive the second view on the fly. That's what this paper does to Gemma 4's KV cache. The committed claim: in Gemma 4 31B's global attention layers, the value projection is the normalized key, and the key projection is the value scaled by a fixed weight (with RoPE on only one quarter of dimensions). This algebraic relationship means you can store a single 640-element row per token per KV head instead of the standard key (1024) plus value (1024). The paged attention kernel reads this compact row through two overlapping strided memory views — one for keys, one for values — with zero changes to the attention computation itself. The results are surgical. On a vLLM prototype with FP16 weights on two A100 GPUs, the compact cache produces bit-identical outputs to an emulation of the same arithmetic on the stock cache layout. Only 2 of 432 final answers differ from the stock vLLM engine, and those fall within run-to-run noise. Steady-state decode KV memory per request drops by a factor of 1.34× at 32K tokens and 1.40× at 64K tokens — directly translating into more concurrent requests per GPU. Throughput gains track the memory savings closely. With a 15 GiB block pool, the compact engine processed 1.38× the stock decode tokens per step for 16K-token prompts and 1.24× for 8K-token prompts. A predictive model of vLLM's block pool matched both ratios to within 1%, which is unusually clean validation for a systems paper. Eager decode throughput rose by 1.30× and 1.11× respectively. The honest limitations are stated upfront: the eager path runs 4-9% slower at equal batch size (you're trading per-step latency for total throughput via larger batches), and the method does not yet compose with 8-bit KV quantization at the default scale. That second limitation matters — most production deployments already use INT8 KV caches, so the practical win narrows considerably in those settings. The deeper significance is about closed-open alignment. Google's Gemma 4 technical report mentions this 37.5% cache reduction is possible, but no major open serving engine (vLLM, SGLang, TensorRT-LLM) actually implements it. This paper bridges the gap between architectural documentation and deployed systems — a surprisingly common no-man's-land where known optimizations go unimplemented for months or years because nobody writes the actual code. This is a clean systems contribution, not a research breakthrough. The math was implicit in the architecture spec. The value is in proving it works end-to-end in a real serving stack with bit-identical outputs, quantifying the throughput gain, and providing a prototype that others can integrate. For teams serving Gemma 4 at scale on constrained GPU memory, this is directly deployable knowledge.
The analogy
You're storing two photo albums — one sorted by date, one by location — because your app expects both indexes. Then you realize the location sort is just a deterministic reorder of the date sort: store one album, derive two views. That's the KV cache trick here: the key and value are algebraic transforms of each other, so you store one compact row and present two overlapping memory views to the attention kernel.
Dual-sorted album
Two index orders stored separately, but one is a deterministic reorder of the other — only one copy needed
Compact KV row
Key and value are algebraic transforms of a single 640-element row — two strided memory views replace two full 1024-element arrays
Honest assessment
This is good systems engineering, not a research breakthrough. The math was already in Google's architecture spec; the value is proving it works bit-identically in vLLM with clean throughput measurements and a predictive model that validates the gains. Teams serving Gemma 4 31B in FP16 should care today. Everyone else should note the pattern: model-specific algebraic redundancies left unimplemented in open engines are free performance waiting to be collected. The INT8 incompatibility and lack of cross-method comparison are the two real gaps.
What Is CLAIMS?
67
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
2/5
Ladder
3/5
Architecture
4/5
Integrity
4/5
Milestone
3/5
Successor
4/5
What is this paper actually committing to?
Implementation, not invention. The algebraic redundancy was documented in Google's report; the contribution is proving it works bit-identically in a real vLLM serving stack with measured throughput gains.
The field fight
Should open serving engines implement model-specific KV cache optimizations, or rely on generic compression (quantization, pruning, sliding window)?
Side A
Generic methods compose across models and are simpler to maintain; model-specific tricks are fragile
Held by: vLLM defaults, TensorRT-LLM, most production deployments
Side B
Architecture-aware optimizations leave free performance on the table; model designers encode exploitable structure that generic methods ignore
Held by: This paper, FlashInfer custom kernels, model-specific fusions
Milestone tracker
Unlocks: Serving Gemma 4 31B at significantly larger batch sizes on single-node GPU setups, reducing per-token cost
Current
1.40
× memory savings
FP16 compact cache
Next
~2.8
× combined savings
Compact + INT8 KV
Unknown — requires kernel-level changes to quantized paged attention
Verified Facts (9)
Gemma 4 31B global attention: value is the normalized key projection, key is the value scaled by a fixed weight, with RoPE on one quarter of dimensions
Stock serving engines store 1,024 values per KV head and token; compact layout stores one 640-value row
2 of 432 final answers differ from stock engine, within run-to-run noise
Steady decode KV memory per request falls by factor of 1.34 to 1.40 at 32K to 64K tokens
Decode engine with 15 GiB pool ran 1.38× stock decode tokens per step for 16K-token prompts (repeated to 0.06%)
1.24× for 8K-token prompts; predictive model matched both ratios to within 1%
Eager decode throughput rose by 1.30× and 1.11×
Eager step 4% to 9% slower at equal batch
Method does not yet stack with 8-bit KV cache at default scale
Practical Implications (4)
Teams serving Gemma 4 31B in FP16 can increase concurrent requests by 24-38% by adopting the compact KV layout with no accuracy loss
The strided-view approach requires no changes to attention kernels — only the cache allocation and read patterns change, lowering integration risk
This does NOT compose with INT8 KV quantization yet, so teams already using quantized caches see no benefit until that's resolved
The pattern generalizes: check whether your model's K and V projections have algebraic relationships that make one derivable from the other
More latest news
Read all