How Big Is the Cache?

Serve real Llama models to many users at once: add up weights and KV cache against GPU memory, change the number of key/value heads, and see the memory-bandwidth limit on tokens per second.

Try it

8 min

Read the concepts behind it

The KV Cache
Prefill, Decode and the Memory Wall
Multi-Query and Grouped-Query Attention
Quantization