Will it fit
Pick a model, a GPU, and a quantization level, and watch a stacked fuel gauge fill with the real numbers: weights + KV-cache + overhead against your card's capacity line. Toggle quantization and the stack visibly shrinks; push the concurrency slider and watch the same GPU collapse under real traffic. It runs the actual arithmetic on your own hardware's number. Runs entirely in your browser (0 uploads, works offline).
How to use it
- Choose a model size and a GPU (or the 8 GB laptop), and a quantization level.
- Drag context length and concurrent users — the KV-cache block grows with both.
- Read the gauge and the X-ray: exactly how much VRAM each part needs, whether it fits, and how many concurrent users the card can actually serve.
What this clears up (the fundamentals)
- VRAM = weights + KV-cache + overhead — weights are just the start.
params × bytes-per-paramgives the weights; the cache and framework overhead are on top. - Quantization only shrinks the weights — going fp16 → int4 quarters the weight bytes but does nothing for the KV-cache, which is why quantized models still overflow under load.
- KV-cache is the scaling killer — it grows linearly with both context length and concurrent users. A model that loads fine at batch-1 collapses when real traffic arrives.
- "Any laptop runs a big model" is usually false — the arithmetic on your own GPU's number settles it. Even int4, a large model plus a long context plus a few users overflows a consumer card.
Where it's used
An infra/MLOps literacy tool — the fastest way to size a deployment before renting a GPU, and to kill the "just run it locally" myth with a real number. It's a miatz build-lab concept playable; learn to build it yourself.
FAQ
How is the weight memory calculated?
Parameters times bytes-per-parameter: fp16 is 2 bytes, int8 is 1, int4 is 0.5. A 7B model in fp16 is 7 × 2 = 14 GB of weights alone, before any cache or overhead.
What is the KV-cache and why does it grow?
It's the stored keys and values for every token in context, kept so the model doesn't recompute them. Its size is roughly 2 × layers × hidden × 2 bytes per token, multiplied by the context length and the number of concurrent users — so long contexts and many users both inflate it.
Why does quantization not fix everything?
Because it compresses only the weights. The KV-cache and runtime overhead stay the same, so a heavily-quantized model with a long context and real concurrency can still exceed the card.
Is anything uploaded?
No. All the arithmetic runs in your browser — nothing is transmitted or stored. Turn off your Wi-Fi and it still works.
Limits
A teaching model using standard approximations (fp16 KV-cache, a fixed overhead factor) and preset model dimensions — real deployments vary with attention type (GQA/MQA shrink the cache), paged attention, and framework. Use it for order-of-magnitude sizing, then confirm with your serving stack.
Related
Part of the Demystify Playgrounds. Explore the rest from the Playgrounds home.
Bookmark this page (Ctrl+D, or ⌘D on Mac) — it works offline the next time you need it.