You have decided to run an AI model on your own machine. You find the model you want, click through to the download page, and suddenly you are staring at a list of fifteen files with cryptic names like Q4_K_M, Q5_K_M, Q8_0 and Q3_K_S. They are all the same model, yet they range from a few gigabytes to dozens of gigabytes. Which one do you actually download? Picking wrong means either a model that will not fit in your memory or one that is needlessly large and slow. This guide decodes those names so you choose correctly the first time.
GGUF is the model file format used by llama.cpp and the wider ecosystem of tools built on it, so almost every local model you download for CPU or mixed CPU and GPU running will be a GGUF file. If you want the deeper background on why these files come in so many sizes, our guide to quantisation for local models explains the underlying idea, and our llama.cpp serving guide covers actually loading these files on Nigerian hardware. Here we focus narrowly on reading the names.
The number is bits per weight
The first thing to understand is the number after the Q. It tells you, roughly, how many bits are used to store each weight in the model. A model's weights are the millions or billions of numbers that hold what it has learned, and the original model stores each one at full precision. Quantisation shrinks those numbers down to fewer bits so the file is smaller and needs less memory.
Higher number means more bits, which means larger file, more memory needed, and quality closer to the original. Lower number means fewer bits, smaller file, less memory, and a gradual loss of quality. In plain terms:
- Q8 is around 8 bits per weight. It is the largest of the common options and sits so close to the original model that most people cannot tell the difference in output.
- Q5 is around 5 bits per weight. A strong middle ground that keeps most of the quality while taking noticeably less space than Q8.
- Q4 is around 4 bits per weight. The smallest of the everyday choices and the popular sweet spot, fitting comfortably on modest hardware with only a small, often unnoticeable, dip in quality.
Below Q4, into Q3 and Q2 territory, the quality drop becomes much more obvious, with the model making more mistakes and writing less coherently. Those tiny quants exist for desperate situations where the model simply will not fit any other way. The number, then, is the dial that trades size and memory against quality, which is exactly the trade-off our quantisation explainer covers in detail.
The K means K-quants, and you want them
Most modern GGUF files have a K in the name, as in Q4_K_M. That K marks them as K-quants, a smarter quantisation method than the older, simpler approach. Instead of squashing every part of the model with the same crude precision, K-quants are cleverer about where they spend their bits, giving more precision to the parts of the model that matter most and less to the parts that can spare it.
The practical result is that a K-quant gives you better quality than an old-style quant of the same size. There is almost never a reason to pick a non-K quant when a K version exists. When you see a file with a plain number and no K, such as Q4_0, treat it as a legacy option and reach for the K-quant instead. The one common exception you will still see is Q8_0, where the model is already so close to the original that the older method makes little practical difference.
The S, M and L suffix is the size within a level
The final piece is the letter at the end: S, M or L. These stand for Small, Medium and Large, and they describe variants within a single quant level. They exist because even at, say, 4 bits, there is some room to spend a little more or a little less precision on certain layers.
- _S (Small) is the leanest version of that quant level, a touch smaller and slightly lower quality.
- _M (Medium) is the balanced default and the one most people should pick.
- _L (Large) spends a bit more space for a bit more quality.
This is why Q4_K_M is the single most recommended file you will see across the local AI world. It is the medium variant of the 4-bit K-quant: small enough to fit a huge range of hardware, smart enough to hold its quality well, and balanced enough that you rarely need to think harder than that.
Decoding a full name: Q4_K_M
Put the three pieces together and any GGUF name becomes readable. Take Q4_K_M and break it apart:
- Q4 means roughly 4 bits per weight, the small everyday sweet spot for size and memory.
- K means it is a K-quant, the modern method that allocates precision cleverly for better quality at the same size.
- M means the Medium variant within that 4-bit level, the balanced middle pick.
Run the same reading on Q5_K_M and you get the 5-bit K-quant in its medium variant, a clear step up in quality and size. Q8_0 reads as the roughly 8-bit version using the older method, which is fine here because it is already near-original quality. Once the pattern clicks, you never have to guess at a file name again.
How to pick based on your memory
The right quant depends mostly on how much memory you have to spare, whether that is GPU VRAM, system RAM, or a mix of both. Work out roughly how big a file your hardware can hold, then climb this ladder as far as your memory allows. Our VRAM guide helps you size that budget honestly.
- Q8 when you have memory to spare and want the maximum fidelity, with output essentially indistinguishable from the original model.
- Q5_K_M when you want a high-quality result but Q8 is a stretch, the comfortable upper-middle choice.
- Q4_K_M as the everyday sweet spot that fits the widest range of machines, including the modest and used hardware most people in Nigeria are working with.
- Q3 or below only when nothing larger will fit and you accept a real quality cost in exchange for getting the model running at all.
These choices matter most on the large models, where the difference between quants can be several gigabytes. If you are eyeing the big ones, our guides on running Llama 70B locally and running Mistral and Mixtral locally lean heavily on getting the quant choice right to fit them on affordable hardware.
The simple rule and a note on other formats
If you remember nothing else, use this: start with Q4_K_M. It is the safest default for almost everyone. From there, move up to Q5_K_M or Q8 only if you have spare memory and want a little more quality, and move down to Q3 or lower only if Q4_K_M genuinely will not fit. That single habit will serve you well across nearly every model you download.
One thing not to confuse: GGUF is not the only quantised format out there. You will also see AWQ and GPTQ files, which are different formats aimed mainly at running entirely on a GPU, with their own naming and tooling. They are not better or worse in the abstract, just suited to different setups. Our AWQ versus GPTQ versus GGUF comparison covers when each one makes sense, so reach for it if you are deciding between formats rather than just between GGUF sizes.
Frequently Asked Questions
Is Q4_K_M good enough, or am I sacrificing too much quality? For the vast majority of uses, Q4_K_M is more than good enough. The quality loss against the full model is small and usually unnoticeable in everyday chat, drafting and coding help. You only really feel the gap on the most demanding tasks, and even then the step up to Q5_K_M closes most of it. Start at Q4_K_M and only climb if you can see a reason to.
Why are there both Q4_K_S and Q4_K_M if they are the same 4 bits? Because the S, M and L variants spend their bits slightly differently across the model's layers even at the same nominal bit level. Q4_K_S is a touch smaller and a touch lower quality than Q4_K_M, useful when you need to squeeze a model into a memory budget that Q4_K_M just misses. If both fit, Q4_K_M is the better pick.
Can I run Q8 on a modest used PC? It depends entirely on the model's size and your memory. Q8 of a small model may fit a modest machine comfortably, while Q8 of a large model will need far more memory than most affordable builds have. This is exactly why Q4_K_M exists, and why most people on used or mid-range hardware in Nigeria settle there. Check your memory against the file size before committing to Q8.
The One Thing to Remember
A GGUF name is just three facts stacked together: the number is bits per weight and trades size against quality, the K means it is a modern K-quant you should prefer, and the S, M or L is the small, medium or large variant within that level. Put them together and Q4_K_M reads as a balanced 4-bit K-quant, the everyday default that fits the most hardware. Start there, climb only if you have memory to spare, and you will rarely choose wrong.
Not sure which GPU or how much memory your chosen quant needs? Build a machine sized for the models you actually want to run with our configurator, or contact us and we will help you match the right hardware to the right quant for your budget.