Local AI Quantization Formats Explained: GGUF, EXL2, AWQ, GPTQ, and GGML

Bottom Line First

Format choice is decided by "which GPU you use." On an NVIDIA consumer GPU, EXL2 is fastest, while GGUF is the standard when you need broad compatibility. In a server environment, AWQ is the answer. GPTQ is legacy, and GGML is already dead. This article analyzes the actual benchmark numbers for each format and the concrete per-model sizes.

1. Core Architecture Comparison by Format

2. GGUF: The Standard for Local Agents

GGUF is a format that solves GGML's fatal flaw (broken backward compatibility) by standardizing the metadata header into Key-Value (KV) form.

Why GGUF Is the Standard

1. Universality: Works on CPU, Apple Silicon, NVIDIA, AMD, mobile, and even Raspberry Pi
2. Metadata encapsulation: Model architecture, token info, and context limits are contained in a single file
3. Offloading: When VRAM is short, the excess spills to CPU RAM so it runs, if slowly
4. Community support: Default support on every platform, including HuggingFace, Ollama, and LM Studio

Quality Benchmark by GGUF Quantization

For a 7B model, perplexity increase vs. FP16:

Key point: The perplexity difference between Q4_K_M and Q5_K_M is on the order of 0.04, hard to notice in everyday use. However, for multi-step math/reasoning, Q5_K_M shows a measurable improvement.

Recommendation by Use Case

3. EXL2: Extreme Speed on NVIDIA

EXL2 (ExLlamaV2) performs variable-bit quantization in fractional units rather than being locked to whole-bit counts. It is designed to use NVIDIA CUDA kernels to the fullest, so even for the same 4-bit model it is 1.5-2x faster than GGUF.

File Size by bpw Level

GGUF vs EXL2 TPS Benchmark

Llama 3.1 8B, single-stream decode (512-token output):

Key point: EXL2 4.0 bpw is about 45% faster than GGUF Q4_K_M on an RTX 4090. However, being even 1MB short on VRAM causes an OOM crash, so you must match the exact bpw file to your GPU's VRAM boundary.

Caveats

• NVIDIA GPU only (unusable on AMD or Apple Silicon)
• Narrow model selection (few community quantized models)
• VRAM boundary calculation is mandatory (e.g., RTX 4060 Ti 8GB -> 4.0 bpw 7B is tight, 4.5 bpw is OOM)

4. AWQ: The Standard for Server Agents

AWQ (Activation-aware Weight Quantization) observes the activation channels of tensors during inference and selectively preserves the weights with high importance (about 1%). It has an edge over GPTQ in both precision and speed.

AWQ vs GPTQ Benchmark

Llama-2-13B 4-bit quantization comparison (RTX 3090):

AWQ's Mathematical Advantage

• Unlike GPTQ, which trims every layer uniformly, it preserves important weights, so its defense of coding, math, and multi-step reasoning benchmark scores is far superior
• MMLU basis: AWQ -1.2, GPTQ -1.5 (vs. FP16)
• HumanEval Pass@1: AWQ 51.8%, GPTQ 46% (Llama 3.1 8B basis)

Usage Scenarios

• Mainly used in large-scale pipelines such as vLLM and TGI
• Optimal for standing up a stable API server in a Linux container environment
• Preferred when building multi-GPU infrastructure

5. GPTQ: The Legacy Format

GPTQ (Generalized Post-Training Quantization) is a traditional second-derivative-based post-training quantization technique. Since AWQ appeared it has fallen somewhat behind in computational precision and speed, and it is no longer used for new models.

GPTQ's Limits

• Slightly higher perplexity than AWQ (4.338 vs 4.325)
• Lower score than AWQ on the HumanEval coding benchmark (46% vs 51.8%)
• The community prefers AWQ when quantizing new models

Cases Where It Is Still Used

• When using a model that has already been quantized to GPTQ
• In legacy environments where AutoGPTQ is installed
• When vLLM's Marlin kernel accelerates GPTQ

6. GGML: The Dead Format

GGML (Georgi Gerganov Machine Learning) is the direct ancestor format of llama.cpp. Due to structural limits, it had the fatal flaw of breaking backward compatibility every time the model metadata architecture changed, so it is no longer used and has been entirely replaced by GGUF.

GGML's Historical Significance

• As llama.cpp's first format, it announced the beginning of the local LLM ecosystem
• With the move to GGUF, every backend dropped GGML support
• Most models downloadable from HuggingFace have also completed the GGUF migration

7. Concrete Format/Size Comparison by Model

Qwen3-4B

K2-Horizon-3.7B

Gemma 4 E4B (8B parameters)

Llama 3.1 8B (comprehensive comparison)

8. Format Selection Decision Table by Hardware

Model Recommendation by VRAM Tier (September 2026)

9. The Community Consensus as of 2026

The 2026 consensus of the r/LocalLLaMA community of 749k+ members:

1. GGUF is the "lingua franca" — works on all hardware, the widest-reaching format
2. EXL2 is fastest on NVIDIA consumer GPUs — but NVIDIA-only with a narrow model selection
3. AWQ replaces GPTQ — slightly better quality at the same bit width
4. GPTQ is legacy — used only for existing models; use AWQ for new models
5. Q4_K_M is the default starting point — "good enough quality for most users"
6. Bigger model + Q4 > smaller model + Q8 — parameters matter more than precision

10. Latest llama.cpp Development (2026)

1. Imatrix quantization standardization: The llama-imatrix tool enables more accurate low-bit quantization after calibration
2. Rise of IQ4_XS: Quality on par with Q4_K_M, about 10% smaller (about 4.17 vs 4.58 GB for 8B)
3. KV cache quantization: The --cache-type-k q8_0 --cache-type-v q8_0 flags save 50% of the KV cache
4. Per-tensor selective quantization: The --tensor-type option keeps sensitive attention vectors at higher precision
5. Gemma 4 official QAT model: Google itself provides GGUF Q4_0 checkpoints

11. Key Summary

1. GGUF is the universal standard: Works in every environment down to CPU, Apple Silicon, AMD, and mobile
2. EXL2 is the fastest on NVIDIA: 45% faster than GGUF, but NVIDIA-only and VRAM calculation is mandatory
3. AWQ is the server standard: Replaces GPTQ and keeps high quality by preserving important weights
4. GPTQ is legacy: Not used for new models
5. Q4_K_M is the default starting point: Good enough quality for most users
6. Bigger model + Q4 > smaller model + Q8: Parameters matter more than precision