If you have spent any time reading about running AI models on your own machine, you will have bumped into the word quantization within minutes. It is everywhere — in model names, in download pages, in forum advice. And yet it is rarely explained properly. People throw around terms like 4-bit, INT8 and FP16 as though everyone already understands them. This article fixes that. Quantization is the single most important idea in local AI, and once it clicks, almost everything else about running models at home makes sense.
The short version is this — quantization is the trick that lets ordinary people run enormous models on affordable hardware. Without it, the kind of consumer graphics card you might put in a Lagos workstation simply could not hold a capable model. With it, the same card runs something genuinely useful. If you want the hardware side of that story, our guides on how much GPU VRAM you actually need and serving local models with llama.cpp pair naturally with what follows here.
What quantization actually is
An AI model is, at heart, a gigantic collection of numbers called weights. A large model has billions of them. Every time the model thinks, it is doing arithmetic with these numbers. The question quantization answers is a simple one — how precisely do we need to store each number?
By default, models are trained and stored at high precision. Each weight is typically a 16-bit floating point number (FP16), and sometimes a 32-bit one (FP32). That precision is generous. It means each individual number can be represented very finely, with many decimal places of accuracy. The cost of that generosity is size — billions of numbers, each taking up 16 or 32 bits, adds up to an enormous memory footprint.
Quantization stores those same numbers using fewer bits. Instead of 16-bit precision, you might use 8-bit integers (INT8), or even 4-bit (INT4). You are rounding each weight to a coarser approximation of its original value. The payoff is that the model shrinks roughly in proportion to the bit reduction. Going from FP16 down to 4-bit is roughly a fourfold reduction in size — a model that needed a certain amount of memory now needs about a quarter of it.
Why it matters so much
That size reduction is not a minor convenience. It is the entire reason local AI is possible for normal people. Consider what it does:
- Models fit where they could not before. A model that would never squeeze onto a consumer graphics card at full precision suddenly fits comfortably once quantized. This is why running something like Llama 70B locally is achievable at all — at full precision it is hopeless on consumer hardware, but quantized it becomes a real option.
- It speeds things up. Inference on these models is largely limited by how fast data can move through memory rather than raw calculation. Smaller weights mean less data shuttling back and forth, so a quantized model often runs faster as a happy side effect of being smaller.
- It lowers the cost of entry. Less memory required means a cheaper card does the job. For a Nigerian builder weighing every naira, that difference can be the line between a build that is affordable and one that is out of reach.
In plain terms, quantization is the great democratiser. It takes models that were the preserve of well-funded data centres and brings them within reach of a single workstation in Ibadan or Abuja.
The trade-off you are making
Nothing is free. When you store a number with fewer bits, you are approximating it more coarsely, and that approximation introduces a small amount of error. Across billions of weights, those errors can in principle add up to a loss of quality — the model becomes a slightly less faithful copy of its full-precision self.
Here is the crucial insight, though — for modern quantization methods, the loss is usually small and graceful down to about 4-bit. The techniques used today are clever about which numbers to preserve carefully and which can be approximated more loosely. In practice, a well-made 4-bit model is often nearly indistinguishable from the full version for everyday tasks. You would struggle to tell the difference in ordinary conversation, writing help or summarising.
The trouble starts when you push too far. Drop down to 3-bit or 2-bit and quality degrades noticeably. The model gets measurably worse — it makes more mistakes, loses coherence, and starts to feel, bluntly, less intelligent. So there is a genuine sweet spot, and the art of choosing a quant is the art of finding it.
Where quantization breaks down
Some situations punish aggressive quantization more than others, and it helps to know which:
- Very low bit-widths. Below 4-bit, the errors stop being negligible. There is a steep drop in usefulness as you head towards 2-bit territory.
- Precise reasoning, maths and code. Tasks where small errors compound are the most sensitive. A tiny slip early in a chain of reasoning, or a single wrong token in code, can derail the whole answer. These benefit from keeping more precision.
- Smaller models. Big models tolerate aggressive quantization far better than tiny ones. A large model has redundancy to spare, so trimming precision hurts less. A small model has less slack, so the same trimming bites harder.
The practical takeaway is that the right level of quantization depends on both the model and the job. A big model doing casual work can be squeezed hard. A small model doing careful maths should be kept generous.
The precision ladder
It helps to picture the options as a ladder, from most generous to most aggressive. Each rung trades size for quality:
- FP16 (full precision). Largest footprint, maximum fidelity. Use it when you have the VRAM to spare and need every last drop of quality.
- 8-bit. Roughly half the size of FP16 with quality loss so slight it is rarely noticeable. A safe, conservative choice.
- 5-bit. Smaller again, still excellent quality. A comfortable middle ground for those with a little memory headroom.
- 4-bit. The popular sweet spot — about a quarter of the full size, with quality that holds up remarkably well for everyday use. This is where most local users live.
- Below 4-bit (3-bit, 2-bit). Smallest footprint, but now the quality cost is real and visible. Reserve these for when memory is so tight you have no other option, and accept the model will feel duller.
For most local use, a 4-bit or 5-bit quant is the right answer. You get the big memory savings with minimal quality loss. Reach for 8-bit or full precision only when you genuinely have the VRAM and need maximum fidelity. And for Nigerian builders especially, this ladder is good news — quantization is precisely what makes capable local AI affordable on modest or used hardware, rather than requiring an expensive top-tier card.
How this connects to the rest of local AI
What you have read here is the general concept. In practice you will meet quantization through specific formats and labels. The most common for local use is GGUF, with its family of quant levels — Q4, Q5, Q8 and so on — which we cover in detail in our piece on GGUF Q4, Q5 and Q8 quants explained. Those labels are just concrete implementations of the precision ladder above.
Quantization is not only about running models, either. It is also the foundation of efficient fine-tuning — the technique known as QLoRA, which we explore in our LoRA and QLoRA training hardware deep dive, works by quantizing a model so it can be adapted on affordable hardware. And if you want to see these ideas in action across different model families, our guide to running Mistral and Mixtral locally shows how quant choices play out in real builds.
Frequently Asked Questions
Does quantization change the model permanently, or can I get the full version back? Quantizing produces a separate, smaller file — it does not destroy the original. The full-precision model still exists wherever it was published, and you can always download a different quant level or the full version if you decide you need more fidelity. You are choosing which copy to run, not altering the original.
Will a 4-bit model give me wrong answers I would not get otherwise? For everyday tasks, the difference is usually too small to notice. Where it can show is in demanding work — long chains of reasoning, precise maths or code — where small approximation errors have more room to compound. If your work is sensitive in that way, stepping up to 5-bit or 8-bit is a sensible insurance policy.
If quantization is so good, why not always use the lowest bit-width? Because quality falls off a cliff once you go below about 4-bit. The savings from 4-bit to 2-bit are far smaller than the savings you already banked getting down to 4-bit, while the quality cost grows sharply. You would be trading away a lot of intelligence for a little memory — rarely a good deal.
The One Thing to Remember
Quantization stores a model's numbers with fewer bits, which shrinks it dramatically while losing very little quality down to about 4-bit. That single trade-off is what lets a big, capable model run on an ordinary, affordable machine — and for anyone building locally on a budget, a 4-bit or 5-bit quant is almost always the right place to start.
Ready to put this into practice? Build a workstation tuned for local AI with our configurator, or get in touch and we will help you match the right card and memory to the models you actually want to run.