selfhostkit.dev · tools · self-hosted AI
LLM VRAM calculator
How much memory does it take to run a model locally? Set the size, quant and context, and get the weights, KV cache and a straight hardware answer. Open source ↗
Weights are computed exactly. KV cache is an estimate that assumes grouped-query attention (most 2024+ models); older models use more. Add a safety margin for anything critical.
How to read this
- Weights are the model itself. Halve the bits (fp16 → Q8 → Q4) and you roughly halve this. Q4_K_M is the sweet spot: about 4× smaller than fp16 with very little quality loss, which is why most local setups use it.
- KV cache grows with context length. Long contexts (32K, 128K) can cost more than the weights on big models. Quantizing the KV cache to 8-bit roughly halves it.
- Total includes ~10% runtime overhead. If it fits your GPU's VRAM, it runs fast. If not, llama.cpp can offload layers to system RAM: slower, but it works.
Rules of thumb
- An 8B model at Q4 runs comfortably on an 8 GB GPU, or CPU-only with 16 GB RAM.
- A 70B at Q4 needs about a 48 GB GPU (or two 24 GB cards) for weights + context.
- No GPU? A cheap VPS with enough RAM runs small models on CPU. See the best VPS guide for what to rent, and how to run an LLM locally for the full setup.