Quantization for LLMs Explained (2026 Guide)

Quantization shrinks a large language model by storing its weights in lower precision—4-bit or 8-bit instead of the usual 16-bit—cutting memory use and speeding up inference with surprisingly little quality loss. With quantization for LLMs explained clearly, you'll understand how a 70-billion-parameter model that needs 140 GB of memory can be squeezed onto a single consumer GPU. This guide covers how quantization works, the main formats (GGUF, AWQ, GPTQ, FP8, and bitsandbytes), how to choose between them, and the pitfalls that quietly degrade quality.

What quantization is and why it matters

A model's weights are just numbers, and by default they're stored as 16-bit floating-point values (FP16 or BF16)—two bytes each. Quantization stores them in fewer bits: 8-bit integers (one byte), 4-bit (half a byte), or lower. Fewer bits per weight means a smaller model, and the savings are dramatic:

The practical impact is enormous. A 70B model needs roughly 140 GB at FP16—several data-center GPUs—but quantized to 4-bit it drops to about 40 GB, fitting on a single high-end consumer card. Quantization is what turns "requires four A100s" into "runs on one RTX 4090."

Crucially, the most common form—post-training quantization (PTQ)—requires no retraining. You take an existing model and compress it, often with a single line of code, which makes quantization the lowest-effort of the model-compression techniques (the others being pruning and distillation). The whole game is the tradeoff between size and quality: smaller is cheaper and faster, but push the precision too low and the model's outputs degrade.

How quantization works

At its core, quantization maps a wide range of high-precision weight values onto a much smaller set of low-precision buckets. Done naively, this loses information and hurts quality; done well, it preserves nearly all of the model's behavior because neural network weights tolerate a surprising amount of rounding.

The sophistication is in how you round. Modern methods don't treat all weights equally—they recognize that a small fraction of weights matter disproportionately. AWQ, for example, identifies roughly the 1% most important weights (by activation magnitude) and keeps them at higher precision while aggressively quantizing the rest. Many methods also use a small calibration dataset to measure how weights behave on real inputs and quantize accordingly.

There are two broad timing strategies. Post-training quantization (PTQ) compresses an already-trained model and dominates in practice because it's fast and easy. Quantization-aware training (QAT) simulates the rounding during training so the model learns to be robust to it, achieving better quality at very low bit-widths—but at much higher cost. As a rule, 8-bit quantization is nearly lossless, 4-bit costs only a small amount of quality with good methods, and below 4-bit quality starts to fall off meaningfully.

The main formats and methods

The ecosystem has settled around a handful of formats, each tuned for a different deployment target:

GGUF (from the llama.cpp project) is the dominant format for local use—it runs on CPUs, Apple Silicon, and NVIDIA GPUs, packages everything (weights, tokenizer, metadata) into one self-contained file, and uniquely supports splitting a model across GPU VRAM and system RAM. Its quality levels run from Q2_K to Q8_0. AWQ (Activation-aware Weight Quantization) shines for GPU serving through vLLM, offering the best quality-to-speed ratio and strong results on reasoning-heavy tasks. GPTQ is a comparable INT4 method, a fine fallback when an AWQ build isn't available. FP8 is the native 8-bit format on the newest NVIDIA hardware (H100 and Blackwell), delivering near-FP16 quality at the fastest speeds and cutting latency substantially. And bitsandbytes, with its NF4 (Normal Float 4) format, is the standard for fine-tuning—its on-the-fly 4-bit quantization is what powers QLoRA.

How to choose the right one

The right format follows directly from what you're doing:

• Running locally on a CPU, laptop, or single GPU? Use GGUF via Ollama or llama.cpp. Start with Q4_K_M, the 2026 sweet spot: about a 4× memory reduction versus BF16 with under 2% quality loss on standard benchmarks. Step up to Q5_K_M or Q6_K if you have VRAM headroom; avoid Q2_K unless you're desperate for memory.
• Serving many users on GPUs in production? Use AWQ with vLLM for the best throughput-to-quality balance, or FP8 if you're on H100/Blackwell hardware for near-lossless speed.
• Fine-tuning? Use bitsandbytes 4-bit (NF4). This is exactly the quantization underpinning QLoRA, which lets you fine-tune open source LLMs on modest hardware—and it's a big part of why LoRA-based methods beat full fine-tuning on cost. You load the base in 4-bit, train adapters on top, and fine-tune on a carefully prepared dataset—whether hand-curated or generated synthetically—without ever needing the full-precision model in memory.

You rarely need to quantize models yourself: Hugging Face contributors (bartowski, mradermacher, and historically TheBloke) publish GGUF, AWQ, and GPTQ builds of popular models within hours of release, and Ollama's library serves pre-quantized GGUF (mostly Q4_K_M) by default. One important note: a modern "K-quant" like Q4_K_M is meaningfully better than the legacy Q4_0—there's almost never a reason to use a _0 quant in 2026.

Quantization vs distillation, and common pitfalls

Quantization is easy to confuse with model distillation, since both shrink models—but they work differently. Quantization reduces the precision of an existing model's weights, leaving its architecture intact; distillation trains a new, smaller model to mimic a larger one. They're complementary and stackable: a common production recipe distills a model down to a smaller student, then quantizes that student for even more savings.

The pitfalls to watch:

• Pushing precision too low. Below 4-bit, quality degrades fast. Reasoning and long-context tasks suffer most from aggressive quantization, so always validate a quantized model on your actual tasks, not just generic benchmarks.
• Forgetting the KV cache. Quantizing weights shrinks the model, but the KV cache that holds your context isn't reduced by default—and it grows with context length. At 32K context on a 70B model, the KV cache alone can exceed 20 GB. Leave headroom for long contexts.
• Using legacy quant types. Old Q4_0-style quants are strictly worse than modern K-quants like Q4_K_M. Check what you're downloading.
• Expecting it to change what the model knows. Quantization is purely an efficiency technique—it makes the same model smaller and faster, not smarter or better-informed. If you need a model to know different things, that's a job for fine-tuning or retrieval, so it's worth settling when to fine-tune versus use RAG separately.

Frequently asked questions

What does quantization do to an LLM? It stores the model's weights in lower precision—4-bit or 8-bit instead of 16-bit—which shrinks the model's memory footprint and speeds up inference. A well-quantized model keeps almost all of its original quality while using far less memory, letting large models run on cheaper hardware.

Does quantization reduce model quality? A little, depending on how aggressive it is. 8-bit quantization is nearly lossless, and good 4-bit methods like Q4_K_M lose under about 2% on standard benchmarks. Below 4-bit, quality drops more noticeably, and reasoning or long-context tasks are the most sensitive—so validate on your real workload.

What's the difference between GGUF, AWQ, and GPTQ? GGUF is the format for local and CPU/mixed inference via llama.cpp and Ollama. AWQ and GPTQ are INT4 formats for GPU serving, with AWQ generally giving the best quality-to-speed ratio through vLLM. Choose based on where you're running: GGUF for local, AWQ (or GPTQ) for GPU production.

Which quantization should I use? For local use, GGUF Q4_K_M is the best default. For GPU production serving, AWQ with vLLM, or FP8 on the newest NVIDIA GPUs. For fine-tuning, bitsandbytes 4-bit (NF4), which powers QLoRA. Match the format to your deployment target rather than chasing a single "best" option.

Is quantization the same as distillation? No. Quantization lowers the numerical precision of an existing model's weights, while distillation trains a new, smaller model to imitate a larger one. They address size and cost differently and are often combined—distill to a smaller model, then quantize it for additional savings.

The takeaway

With quantization for LLMs explained, the essential idea is that storing weights in fewer bits makes a model dramatically smaller and faster while keeping nearly all its quality—turning models that once needed a server rack into ones that run on a single GPU. Your next step is simple: for local use, pull a Q4_K_M GGUF build and run it; for GPU serving, reach for AWQ with vLLM; and for fine-tuning, use 4-bit bitsandbytes with QLoRA. Just validate the quantized model on your own tasks, mind the KV cache on long contexts, and remember that quantization changes a model's size, never what it knows.