TL;DR — A model’s memory footprint is, to a first approximation, parameter count × bytes per parameter. Both numbers are public: the parameter count is usually in the model’s name, and the precision is in the config.json of its Hugging Face repository (torch_dtype). Llama-2-7B in FP16 is ~14 GB of weights — but weights alone don’t serve requests. Budget for the KV cache and runtime overhead too, or your “it fits” GPU will OOM on the first long prompt.
Why this matters
Serving an LLM is not like deploying a web app. The binary you ship is tens of gigabytes of floating-point numbers, all of which must sit in GPU memory (VRAM) before the model can generate a single token. Get the estimate wrong and you either buy a GPU the model can’t fit on, or you overprovision and burn money on idle VRAM.
The good news: you can estimate the footprint on the back of a napkin, from information the model card already gives you. That’s the first challenge in serving LLMs, and the subject of this post.
1. Estimate the model size
When calculating the memory footprint of a model, there are two key variables:
- The model’s parameter count
- The data type (precision) of its parameters
Finding the parameter count
Take a model from Hugging Face — a model’s number of parameters can often be found directly in its name. For example, Llama-2-7b-chat-hf: the 7b indicates the model contains approximately 7 billion parameters in its weights. The same convention holds across most open models: Mistral-7B, Llama-2-13b, Falcon-40B, Llama-2-70b.
Finding the precision
The data type is found in the config.json file of the model’s Hugging Face repository:
config.json
{
"_name_or_path": "meta-llama/Llama-2-7b-chat-hf",
"architectures": [
"LlamaForCausalLM"
],
"bos_token_id": 1,
"eos_token_id": 2,
"hidden_act": "silu",
"hidden_size": 4096,
"initializer_range": 0.02,
"intermediate_size": 11008,
"max_position_embeddings": 4096,
"model_type": "llama",
"num_attention_heads": 32,
"num_hidden_layers": 32,
"num_key_value_heads": 32,
"pretraining_tp": 1,
"rms_norm_eps": 1e-05,
"rope_scaling": null,
"tie_word_embeddings": false,
"torch_dtype": "float16",
"transformers_version": "4.32.0.dev0",
"use_cache": true,
"vocab_size": 32000
}Specifically, the torch_dtype attribute reveals the precision used for the model weights. Here the value is float16 (FP16), meaning the model uses 16-bit floating-point numbers — 2 bytes per parameter — for storage and computation.
Each precision has a fixed cost per parameter:
| Data type | Bits | Bytes per parameter |
|---|---|---|
| FP32 | 32 | 4 |
| FP16 / BF16 | 16 | 2 |
| INT8 / FP8 | 8 | 1 |
| INT4 | 4 | 0.5 |
The memory equation
With those two variables, the estimate is a single multiplication:
\[ \text{weights memory} \approx \text{parameters} \times \text{bytes per parameter} \]
For Llama-2-7B in FP16:
\[ 7 \times 10^9 \ \text{params} \times 2 \ \text{bytes} = 14 \times 10^9 \ \text{bytes} \approx \textbf{14 GB} \]
You can sanity-check this against the repository itself: the model’s .safetensors shards on Hugging Face add up to roughly the same number. As a rule of thumb, a 7B model in FP16 needs ~14 GB just for weights — already too big for a 12 GB consumer card, and a snug fit on a 16 GB one.
2. Weights are not the whole story: the KV cache
If weights were the only cost, a 24 GB GPU would serve Llama-2-7B with 10 GB to spare. In practice that headroom disappears fast, because inference needs a KV cache: for every token in every active sequence, the model stores the attention key and value tensors of every layer so it doesn’t recompute them at each generation step.
The per-token cost comes from the same config.json fields:
\[ \text{KV per token} = 2 \times n_\text{layers} \times n_\text{kv-heads} \times d_\text{head} \times \text{bytes per value} \]
For Llama-2-7B (32 layers, hidden size 4096, FP16):
\[ 2 \times 32 \times 4096 \times 2 \ \text{bytes} = 512 \ \text{KB per token} \]
At the model’s full 4096-token context, that is ~2 GB for a single sequence — and serving is rarely a single sequence. Ten concurrent requests at full context want ~20 GB of KV cache on top of the 14 GB of weights. This is exactly why servers like vLLM exist: PagedAttention manages this cache in blocks so memory isn’t wasted on unused context, but the physical budget still has to be there.
3. Add runtime overhead
Beyond weights and KV cache, the serving runtime itself consumes VRAM: CUDA context, activation buffers for the batch in flight, framework workspace. A practical planning rule:
\[ \text{VRAM needed} \approx (\text{weights} + \text{KV cache}) \times 1.2 \]
Putting it together for Llama-2-7B FP16 with a modest batch of concurrent requests, you land in the 20–24 GB range — one A10G/L4-class GPU is tight, one A100/L40S is comfortable.
4. When it doesn’t fit: your two levers
When the memory equation exceeds the GPU you have, there are two standard ways out:
- Lower the precision (quantization). Serving Llama-2-7B in INT8 halves the weights to ~7 GB; INT4 (GPTQ/AWQ) brings them to ~3.5 GB. Quality degrades gracefully for most workloads, and for chat-style use INT4 is often indistinguishable. The table in section 1 makes the effect easy to predict: same parameter count, fewer bytes per parameter.
- Split the model (tensor parallelism). A 70B model in FP16 is ~140 GB of weights — no single GPU holds that. Inference servers shard the weights across 2, 4, or 8 GPUs, at the cost of inter-GPU communication on every forward pass.
Both levers move the same formula, which is the point of this post: the formula comes first. Estimate before you provision.
Wrapping up
Before serving any model, read two numbers off its Hugging Face page — the parameter count in the name, and torch_dtype in config.json — and multiply. Then remember the two costs that don’t show up in the model card: the KV cache (a function of your context length and concurrency) and ~20% runtime overhead.
That answers the capacity question. The next challenges in serving LLMs — throughput, latency, and batching, and how servers like vLLM squeeze more requests out of the same VRAM — are topics for a follow-up post.