Choosing a Local Coding AI Model by VRAM Capacity

The most common mistake when running local coding AI is a single one: calculating VRAM from the model weights alone. In practice, weights + KV cache + runtime overhead all load together, and once VRAM fills up the system falls back to virtual memory (swap), where token generation slows by tens of times or the process crashes.

This piece collects notes from repeated runs of the same prompt across setups from 8GB up to 128GB.

1. The VRAM Formula

Required VRAM = weights + KV cache + runtime overhead + headroom

In practice, the items that mismatch most are KV cache and headroom. You must not assume that 15GB of weights fits a 16GB card. An 8K context alone adds 2GB of KV cache, and the desktop is already eating 1.5GB.

2. What Weights Are

Weights are the lump of numbers that stores the parameters a model learned (the connection strengths between neurons). The 4B, 27B, and 125B in a model name are exactly the parameter count (in billions).

The size is determined by the product of two values.

Weight size ≈ parameter count x (quantization bits ÷ 8) + a small amount of metadata

I actually measured how much the same 27B model changes across quantization levels.

There is a reason the measured size is always a bit larger than the theoretical one. "K-quants" such as q4_K_M do not press every layer down to 4 bits identically. Sensitive parts like the embedding, output layer, and normalization are kept at q6 or q8. So a 4-bit model's file is about 5% larger. That is not waste; it is a device for protecting quality.

One more thing: more parameters is not automatically smarter. It only means more representational capacity. For coding work, 27B q4 is generally better than 9B q8. But that holds only when VRAM allows.

3. KV Cache — Working Memory That Grows as the Conversation Grows

The KV cache is the working memory a model keeps so it does not recompute earlier tokens. The problem is that it grows in direct proportion to context length.

KV cache ≈ 2(K,V) x layers x KV heads x head dim x 2 bytes x tokens

Computing this directly for a 27B model (64 layers, 8 KV heads from GQA, head dim 128) gives about 0.25MB per token.

Here is the practical lesson. "The weights fit" and "it is usable" are different statements. Put 27B q4 (15.2GB) on a 24GB card and, after subtracting runtime and headroom from the remaining 8.8GB, 32K context is in fact the limit. To fit an entire codebase, you have to drop the model a tier.

GQA (Grouped Query Attention) and MLA reduce the number of KV heads to ease this burden, and quantizing the KV cache itself to q8 halves it. The accuracy loss is not perceptible.

4. Runtime Overhead — the Fixed Cost Nobody Mentions

There is an item that consumes VRAM without being weights or KV cache.

• CUDA/Metal context and kernel library workspace
• Batch and graph buffers
• Tokenizer and input/output buffers

With llama.cpp (CUDA 12.6) and a 27B model, the measurement was 0.9-1.4GB. It swings with batch size and context. ollama adds a little more management overhead on top.

5. What Each Quantization Level Costs Against the Original

Smaller numbers are lighter, but quality loss does not arrive linearly — it arrives like a cliff. This is the result of repeating a "refactor three files and pass the tests" request 10 times on the same 27B model.

Measurement details.

• q4_K_M: 8 of 10 runs fully succeeded. Even the 2 failures were at the level of a trivial typo.
• q3_K_M: 5 of 10 succeeded. Errors such as missing function signatures and inventing imports that do not exist ran about 3x those of q4_K_M.
• q2_K: 2 of 10 succeeded. Most of the time the code was syntactically valid but missed the requirements.
• q4_0 / q4_K_S: Not much different from q4_K_M, but a bit rougher on long contexts.

To put it together: 4-5 bits is the ideal point for cutting size without degrading performance, and below 3 bits code-generation quality falls off sharply. When VRAM is short, it is better to go one model size smaller than to lower the bits.

6. Recommended Local Coding AI Models by VRAM Capacity

The table below is based on measured footprint, summing weights + KV cache (at 8K) + runtime.

A few measured points to add.

• 9B q4 (7.0GB) on an 8GB card is razor-thin. If the desktop consumes even 1GB, swapping begins.
• On a 12GB card, 9B at 8-bit felt more stable. It uses 4GB more than q4, but quality rises and swapping disappears.
• With MoE, speed and memory are decoupled. A 35B MoE activates only 3B per token and is fast even on 32GB, but VRAM takes the entire 35B.

7. Three Key Points

1. Choose a model with 'headroom' rather than the biggest model. A model one tier smaller that holds context comfortably is far better for real development productivity than a large model that barely fits in VRAM. Once swapping starts, token speed drops to a tenth.

2. Quantization's optimum is 4-5 bits. The 4-bit range is the ideal point for cutting size without degrading performance. Below 3 bits, code-generation quality falls off sharply, so when VRAM is short, reduce model size rather than lowering bits.

3. Understand what MoE is. An MoE model activates only part of its parameters per token, so it is fast, but it still occupies VRAM equal to the entire model. "It's only 3B active, so 4GB should do" is a misconception.

8. Measurement Method and Variability

• Tools: llama.cpp (CUDA/Metal backend), ollama
• Measurement: nvidia-smi / active VRAM, tokens per second (TPS), increasing context until an OOM occurs
• Conditions: temperature fixed at 0.2, the same prompt repeated 10 times
• Caveat: driver and backend versions and batch settings swing things by hundreds of MB. Browsers, IDEs, and monitors use VRAM too, so real headroom is smaller than the table above (on a 24GB card the desktop occupies 1.2-2.5GB).

These figures are reference values measured in a single environment; actual performance varies with hardware configuration and software versions.

Related Posts

• GPU VRAM Allocation Structure and the KV Cache Bible
• A Complete Guide to Local AI Model Recommendations by Graphics Card
• Local AI Quantization Formats Explained
• Unified Memory 128GB vs 192GB
• The Reality of DGX Spark

2 (K,V) x 64 layers x 8 KV heads x 128 head dim x 2 bytes = 262,144 bytes = 0.25 MB per token