LLM Inference & Serving: KV Cache, Batching, vLLM

LLM Inference & Serving

Inference is what happens between your prompt and the model's response. For a hosted API you never see it — you pay per token and someone else owns the GPUs. But once you're self-hosting an open-weights model, or trying to explain why your p95 latency tripled at 40 concurrent users, the mechanics stop being an implementation detail.

The central fact of LLM inference is that generation happens in two phases with completely different performance characteristics. Prefill processes your entire prompt in parallel and is compute-bound. Decode generates one token at a time, each requiring a full pass over the model weights, and is memory-bandwidth-bound. Nearly every serving optimization — batching, quantization, the KV cache, speculative decoding — exists to attack that second phase, where a GPU capable of hundreds of teraflops sits mostly idle waiting on memory.

TL;DR

Prefill (prompt → first token) is compute-bound and parallel; decode (each subsequent token) is memory-bandwidth-bound and sequential.
The KV cache stores attention keys/values so decode doesn't recompute history. It grows linearly with context and is usually what limits your concurrency.
Continuous batching — adding and evicting requests every step instead of per batch — is the single biggest throughput win, often 5–20×.
Quantization (FP8, INT8, INT4) shrinks weights and raises effective bandwidth, trading a small quality cost for large speed and capacity gains.
TTFT (time to first token) and TPOT (time per output token) are the metrics users feel; tokens/sec/GPU is the one finance feels. They trade off.
Speculative decoding uses a small draft model to propose tokens a big model verifies in one pass — real latency wins on predictable text.
Prompt caching reuses the KV cache for a shared prefix, cutting both cost and TTFT dramatically for repeated system prompts and long documents.

Quick Example

Serving an open-weights model with vLLM and measuring what matters:

For a hosted model, the same latency shape applies — you tune prompt structure and caching rather than GPU flags:

Core Concepts

Prefill and decode

The asymmetry is stark. Generating one token from a 70B model in FP16 requires reading ~140 GB of weights from HBM. On a GPU with ~3 TB/s of bandwidth that's ~47 ms of pure memory traffic for a single token — while the compute units are almost idle. Batch 32 requests together and you read those weights once for all 32, producing 32 tokens in roughly the same time. This is why batching is the dominant optimization: decode is bandwidth-bound, and batching amortizes the bandwidth.

The KV cache

Attention needs the keys and values of every previous token. Recomputing them each step would be quadratic; caching them makes decode linear. The cost is memory:

For a 70B-class model with grouped-query attention, that's roughly 0.3–1 MB per token. A single 8k-token conversation can hold ~2–8 GB of KV cache. Serving 50 concurrent long conversations needs more memory for the cache than for the weights — which is why the KV cache, not model size, usually sets your concurrency ceiling.

Techniques that shrink it:

Grouped-query attention (GQA) / multi-query attention (MQA) — share K/V heads across query heads. A model-architecture decision, already standard in modern open models.
PagedAttention — vLLM's contribution: allocate the cache in fixed-size blocks like OS virtual memory instead of contiguous per-request buffers. Eliminates the internal fragmentation that used to waste 60–80% of cache memory, and makes prefix sharing between requests nearly free.
KV cache quantization — store the cache in FP8 or INT8, halving its footprint at small quality cost.

Continuous batching

Static batching waits for a full batch, runs it to completion, and returns everything together. Because generation lengths vary wildly, most of the batch sits finished while the longest one grinds on.

Continuous (in-flight) batching evaluates the batch composition every decode step. Throughput improvements of 5–20× over static batching are typical, and it's the default in vLLM, TGI, TensorRT-LLM, and SGLang. If you're serving with a hand-rolled loop over model.generate(), this is the first thing to fix.

Prefix caching

Requests that share a prefix — the same system prompt, the same retrieved document, the same conversation history — can share KV cache blocks instead of recomputing prefill for each one.

The effect on both cost and TTFT is large — commonly a 5–10× TTFT reduction for long shared prefixes. It's a flag in vLLM (--enable-prefix-caching) and an explicit cache_control marker in hosted APIs. To benefit, structure your prompts stable-prefix-first: system instructions and documents before per-request variables. See Context Engineering.

Quantization

Quantization stores weights (and optionally activations and KV cache) in fewer bits. Since decode is bandwidth-bound, halving the bytes read per token roughly halves decode time — the speedup is nearly as large as the memory saving.

A practical rule: a 70B model at 4-bit fits on a single 48 GB GPU with room for a usable KV cache, where FP16 would need two 80 GB GPUs. That's often the difference between one machine and four.

Always measure quality on your own task. Perplexity barely moves under 4-bit quantization while structured output, long-context recall, and multi-step reasoning can degrade noticeably. Run your evals at each precision before shipping — this is exactly the kind of regression a benchmark average hides.

Latency, Throughput, and the Tradeoff

They pull against each other. Larger batches raise throughput and lower cost per token, but every request in a large batch decodes more slowly, so TPOT rises. The tuning question is never "make it fast" but "what's the latency budget, and what's the maximum throughput inside it?"

Rough targets for an interactive chat experience: TTFT under ~500 ms, TPOT under ~50 ms (≈20 tokens/sec, comfortably faster than reading speed). For batch and offline work, ignore latency entirely and maximize throughput with the largest batch that fits.

Chunked prefill is worth knowing here: a long prompt's prefill can block decode for every other request in the batch, causing latency spikes. Splitting prefill into chunks interleaved with decode steps smooths TPOT at a small cost to TTFT.

Speculative decoding

A small draft model proposes several tokens; the large model verifies them all in a single forward pass. Accepted tokens are free; rejected ones cost one wasted step.

Speedups of 1.5–3× on predictable text (code, structured output, formulaic prose) and much less on high-entropy creative text. Variants include n-gram/prompt lookup (no draft model, matches against the prompt — very effective for edit and summarization tasks) and Medusa/EAGLE-style multi-head prediction. Crucially, verification is exact: output quality is unchanged, unlike quantization.

Serving Stack

Multi-GPU strategies

Tensor parallelism — split each layer's matrices across GPUs. Needs fast interconnect (NVLink); use within one node to fit a model that doesn't fit on one GPU.
Pipeline parallelism — put different layers on different GPUs. Tolerates slower interconnect; adds pipeline bubbles.
Data parallelism — full model replicas behind a load balancer. The right answer for throughput once the model already fits.
Prefill/decode disaggregation — run prefill and decode on separate GPU pools, since they have opposite bottlenecks. Increasingly common at large scale.

Should you self-host at all?

Be honest about the economics. A hosted API charges per token with no idle cost, no GPU procurement, and no on-call for CUDA OOMs. Self-hosting only wins when you have sustained high utilization (GPUs idle at 3am still cost full price), hard data-residency requirements, a fine-tuned model you must serve, or extreme volume where per-token pricing exceeds amortized hardware. For most teams the honest calculation favors an API, and the right hybrid is often a hosted frontier model for hard requests plus a small self-hosted model for high-volume simple ones — see AI Gateway and Small Language Models.

Best Practices

Structure prompts so the stable part comes first

Cache hits require an exact prefix match. Put system instructions, tool definitions, and retrieved documents before anything that varies per request — a timestamp at the top of your system prompt silently disables prefix caching for every call.

Set `max_tokens` to something realistic

Serving engines reserve KV cache capacity based on the maximum possible output length. Defaulting every request to 4096 tokens when responses average 200 slashes your achievable concurrency for no benefit.

Measure with a load generator, not single requests

Single-request latency tells you nothing about behavior at concurrency 50, which is where queueing, batching, and cache pressure actually appear.

Right-size `--max-model-len`

Context length directly determines KV cache reservation. Setting 128k when your p99 conversation is 6k reserves memory you'll never use and cuts concurrency by an order of magnitude.

Quantize, then re-run your evals

FP8 on modern hardware is close to free and should usually be on. Below 8 bits, verify against your own task suite rather than published benchmark averages. See LLM Evaluation.

Stream, always

Streaming doesn't change total time but changes perceived latency enormously — a user reading the first sentence isn't waiting. It's also what lets you cancel abandoned generations and stop paying for them. See Server-Sent Events.

Route by difficulty

Most production traffic doesn't need your largest model. Classification, extraction, and routing run fine on a small model at a fraction of the cost and latency; escalate only the hard requests. This is usually a bigger cost win than any GPU-level tuning.

Monitor the queue, not just the GPU

GPU utilization near 100% during decode is expected and doesn't mean saturation. Track queue depth, TTFT p95, KV cache utilization, and preemption/swap counts — rising preemptions mean you're over-subscribed and requests are being evicted mid-generation. See LLMOps.

Common Mistakes

Benchmarking with one request at a time

Putting variable content at the front of the prompt

Sequential calls where the work is independent

Sizing GPUs for weights and forgetting the KV cache

Assuming quantization is free

Self-hosting a model you use twice an hour

FAQ

Why is the first token slow but the rest fast?

The first token requires prefill over your entire prompt — work proportional to prompt length. Subsequent tokens only process one new token against the cached keys and values. Doubling prompt length roughly doubles TTFT while leaving TPOT essentially unchanged.

What actually limits how many concurrent users I can serve?

Almost always KV cache memory, not compute. Total GPU memory minus model weights, divided by the per-request cache size (which scales with context length), gives your concurrency ceiling. Shortening context, enabling prefix caching, and quantizing the cache all raise it.

Does quantization hurt quality?

FP8 is close to lossless on hardware that supports it natively. 4-bit weight-only methods (AWQ, GPTQ) are usually acceptable but can degrade multi-step reasoning, tool calling, and long-context recall more than aggregate benchmarks suggest. Test on your own tasks — that's the only answer that generalizes.

Should I use vLLM or a hosted API?

Hosted unless you have a specific reason not to: sustained high utilization, data residency, a fine-tuned model, or volume where the arithmetic clearly favors owned hardware. Self-hosting adds GPU capacity planning, upgrades, and an on-call rotation that most teams underestimate.

What is PagedAttention?

vLLM's technique for allocating the KV cache in fixed-size blocks, borrowing the idea from OS virtual memory. It eliminates the fragmentation that plagued contiguous per-request allocation, lets requests share blocks for common prefixes, and raised achievable throughput by several times when it was introduced.

Does speculative decoding change the output?

No. The target model verifies every drafted token and rejects any it wouldn't have produced, so the output distribution is identical to standard decoding. Only speed changes — and only when the draft model guesses well.

How do I reduce cost without changing the model?

In order of typical impact: enable prompt/prefix caching, trim prompt length (retrieved context is usually the bloat), cap max_tokens, route easy requests to a smaller model, batch offline work, and cache full responses for repeated identical queries.