You have decided to run a quantised model to fit it onto hardware you can actually afford. Good. But then you go looking for the download and you hit a wall: the same model is offered as AWQ, GPTQ, and GGUF, each in several sizes, and nobody tells you which one you are supposed to grab. Pick wrong and you either waste a day fighting a runtime that refuses to load the file, or you leave performance on the table.
The short version is that these are three different formats — three different ways of taking a full-precision model and squeezing it down — and they suit different setups. If the idea of quantisation itself is still fuzzy, start with our plain-English guide to quantising local AI models, then come back. This piece assumes you already know roughly what 4-bit means and just need to choose between the formats, with an eye on the modest and mixed hardware that is common here in Nigeria. We will also touch on how much GPU VRAM you actually need along the way.
They quantise the same model in different ways
All three formats start from the same place: a model trained in 16-bit precision that is too big to run comfortably. Quantisation shrinks the numbers that make up the model down to fewer bits — typically 4 bits — so the file is smaller and needs far less memory. What differs is how each format does the squeezing, which runtimes can load the result, and therefore which hardware it suits.
That last point is the one most guides bury. The deciding factor in practice is rarely some abstract quality score. It is whether the inference engine you intend to use supports the format well. Choose the runtime first, or at least keep it in mind, and the format often picks itself.
GGUF: the flexible all-rounder
GGUF is the native format of llama.cpp and the wide ecosystem of tools built on top of it — Ollama, LM Studio, and most of the friendly desktop apps people actually use at home. If you are running a model on your own machine for your own use, this is very likely the format you want.
- What it is: the file format of the llama.cpp project, designed for flexible local inference.
- Runs where: CPU only, GPU only, or — crucially — a mix of both, offloading as many layers to the GPU as fit and running the rest on the processor.
- Best for: single-user local use on consumer or mixed hardware; people on llama.cpp, Ollama, or LM Studio.
The standout strength is flexibility. GGUF will run on a CPU with no dedicated GPU at all, and it will happily split a model across a modest GPU and system RAM through partial offload. That matters enormously if your VRAM is limited — you are not forced to fit the entire model on the card. It also comes in a huge range of quant levels, from aggressive small sizes to near-lossless larger ones; our breakdown of the Q4, Q5 and Q8 GGUF quants covers how to choose among them. For the practical mechanics of running it, see our guide to serving local models with llama.cpp.
GPTQ: the earlier GPU-resident format
GPTQ is an older post-training quantisation method built specifically for running quantised models on the GPU. Unlike GGUF, it is not designed to spill onto the CPU — the expectation is that the whole quantised model lives in VRAM, where it runs fast.
- What it is: a GPU-focused quantisation method, one of the first widely adopted for 4-bit inference.
- Runs where: on the GPU, with the model resident in VRAM; supported by several GPU inference runtimes.
- Best for: GPU serving setups where the format your engine supports best happens to be GPTQ.
GPTQ delivers good GPU speed and was for a long time the default for GPU-based 4-bit inference. It is still widely available and perfectly usable. That said, it is gradually being overtaken in popularity by both AWQ on the GPU side and GGUF on the flexible side. If you are choosing fresh today and your engine supports the alternative, AWQ is usually the more modern pick — but if your stack is already on GPTQ, there is no urgent reason to abandon it.
AWQ: the modern GPU serving choice
AWQ stands for Activation-aware Weight Quantisation. The clever idea is that not all weights matter equally; AWQ identifies and protects the most important ones during quantisation, which often yields slightly better quality than GPTQ at the same bit-width. It is a newer, GPU-focused method aimed squarely at efficient serving.
- What it is: a newer GPU quantisation method that protects the most influential weights.
- Runs where: on the GPU, with the model in VRAM; well supported by high-throughput serving engines.
- Best for: serving a model to multiple users or requests efficiently on a GPU.
AWQ's natural home is a high-throughput engine like vLLM, which is built for serving many concurrent requests rather than a single chat session. If that is your situation — an internal tool, an API, several people hitting the same model — AWQ on a serving engine is the combination to reach for. Our guide to serving with vLLM walks through the hardware side, and if you are eyeing a large model, our notes on running Llama 70B locally are worth a read first.
The practical decision
Strip away the detail and the choice comes down to how and where you are running the model.
- Choose GGUF if you run locally on consumer or mixed hardware, want the freedom of CPU-only or partial-GPU offload, or use llama.cpp, Ollama, or LM Studio. This covers the majority of home and small-team users.
- Choose AWQ or GPTQ if you are serving on the GPU with an engine like vLLM and want GPU-optimised throughput, with the whole model resident in VRAM.
- Between those two, prefer AWQ as the more modern method — generally the better quality-per-bit of the pair — unless your runtime or existing pipeline pushes you toward GPTQ.
Tie it back to hardware and it is simple. GGUF is flexible, CPU-friendly, and forgiving of a modest card — which is exactly why it suits the mixed and budget-conscious setups many people here in Nigeria run, where a modest GPU might be paired with plenty of system RAM rather than a flagship card. AWQ and GPTQ assume a GPU big enough to hold the whole model, so they belong in deliberate serving builds where you have specified the VRAM to match.
Frequently Asked Questions
Is one format clearly more accurate than the others at 4-bit? Not in a way you should overclaim. All three can reach similar low bit-widths, and the quality gap at 4-bit is usually small and depends on the specific method and model. AWQ's weight-protection often gives it a slight edge over GPTQ, but do not expect a night-and-day difference. The real decider is runtime support, not a benchmark.
Can I just convert between formats? In principle you re-quantise from the original model rather than convert one quantised file into another, and the tooling differs per format. It is almost always easier to download the format you need directly — most popular models are published in all three — than to convert anything yourself.
I am a single user at home. What should I pick? GGUF, almost certainly. It runs on whatever you have, splits across CPU and GPU when VRAM is tight, and works with the friendliest tools. Reach for AWQ or GPTQ only when you move to GPU serving for multiple users or requests.
The One Thing to Remember
Pick the format your runtime supports best, not the one with the nicest reputation. GGUF for flexible local use on mixed or modest hardware; AWQ (or GPTQ) for GPU-resident serving with an engine like vLLM. The format follows the setup, and the setup follows what you are actually trying to do.
Not sure which hardware your chosen format needs? Build a spec in our configurator to size the VRAM and RAM around your plan, or get in touch and we will help you match the format, the runtime, and the machine to your budget.