When a transformer processes a token, attention turns it into a key vector and a value vector at every layer. Those vectors do not change as generation continues, so recomputing them for the entire prompt on every new token would be pure waste. The KV cache keeps them.
The effect on cost is large. Without a cache, generating the thousandth token means re-reading nine hundred and ninety-nine tokens. With one, it means a single step against stored state. This is why the first token of a response is slow and the rest arrive quickly: prefill fills the cache, decode reads it.
The cache is also the reason long context is expensive. Its size grows linearly with sequence length and with batch size, and it lives in GPU memory alongside the weights. A serving stack that can hold sixty concurrent short conversations may hold only four long ones.
Most inference tuning work, from paged attention to cache quantization to prompt caching, is an attempt to make this structure cheaper.