KV Cache

The KV cache is the stored set of key and value vectors a transformer computes for every token it has already processed, kept in memory so that generating the next token only requires attention computation against that new token — not a full recomputation over the entire sequence seen so far.

Autoregressive generation produces one token at a time, and each new token's attention step needs to look back at every previous token in the sequence. Without a cache, generating token 1,000 would mean recomputing the keys and values for tokens 1 through 999 all over again — work that was already done and thrown away the step before. The KV cache is simply keeping that work instead of redoing it.

How it works

The first time a token is processed, the transformer computes its key and value vectors at every layer and stores them. Every subsequent generation step computes the key and value for only the newest token, then attends against the full set of cached keys and values built up so far. This is why generation gets faster in practice than the naive recompute-everything approach would suggest — but it's also why the cache itself becomes a real memory cost: its size grows linearly with how many tokens are in the context, and it multiplies with however many requests are being served concurrently, since each one needs its own cache.

Why it matters

In a production serving system, the KV cache is frequently the actual ceiling on how many requests can run at once or how long a context window can be served — a limit that exists independent of how much memory the model's weights themselves take up. This is the direct motivation behind techniques like grouped-query attention (fewer distinct key/value heads to cache), cache quantization (storing cached values at lower precision, the same idea as model quantization applied to the cache instead of the weights), and paged-attention memory management — all three exist specifically because the cache, not the model, ran out of room first.

A worked example

A 7-billion-parameter model with 32 layers and 4,096 hidden dimensions, running at FP16, costs 512 KiB of KV cache per token for a single request (2 for key-and-value × 32 layers × 4,096 hidden dimensions × 2 bytes = 524,288 bytes). At a 4,096-token context, that's 2 GiB for one request. Serve 16 concurrent requests at that same context length and the cache alone needs 32 GiB — more memory than the model's own weights take up at FP16 (14 GB). The cache, not the model, is what ran out of room.

How Miatz teaches it

The systems track has learners compute the cache-size arithmetic above by hand for a model config of their choosing, then measure actual memory use running that model, to see how closely the formula predicts reality. It's paired deliberately with the quantization lesson, since cache quantization is the same core idea — trade precision for memory — applied to a different piece of the serving stack.