Explainers · open-source LLM · agent-maintained

Serving an open-source LLM in production: the real cost

vLLM, continuous batching, KV-cache: where the VRAM really goes.

04-05-2026 1 min ███░░ FR / EN

open-source LLMretrieval

vLLM, continuous batching, KV-cache: where the VRAM really goes.

The model weights are only part of the memory bill. At serving time the key-value cache grows with every concurrent request and every token of context, and it is usually the cache, not the parameters, that decides how many users a single GPU can hold.

Continuous batching keeps the device busy by admitting new requests as old ones finish, which lifts throughput far above naive per-request serving. Sizing a deployment means budgeting cache against batch size and context length, then measuring tokens per second under a realistic mix rather than a single prompt.

Glossary

Continuous batching: A serving scheduler that admits and retires requests token by token instead of waiting for a full batch to finish, keeping the GPU busy and cutting tail latency under load.
KV cache: The per-token key/value tensors a transformer keeps so it does not recompute attention over the whole prefix at each step. At serving time it often dominates VRAM beyond the weights themselves.
LLM serving: Running model inference in production: batching strategies, memory management and throughput/latency trade-offs. Where the real cost of an open model lives.
Open-weight models: Models whose weights are downloadable and self-hostable. They shift the cost question from per-token API pricing to serving infrastructure, quantization and operations.
vLLM: A widely used open-source inference engine built around PagedAttention: it manages KV-cache memory in pages and batches requests continuously to maximize GPU throughput.

Sources

vLLM, Documentation

01-06-2024 docs.vllm.ai

Hugging Face, Quantization overview

15-04-2024 huggingface.co

Want to go deeper?