If you've shopped for hardware to run large language models locally, you've seen the headline metric: tokens per second (tok/s). Higher is faster, and it's the number every benchmark quotes. But tok/s on its own hides almost everything that decides whether a local LLM is usable — whether the model even fits in your VRAM, how fast it digests your prompt versus how fast it writes the reply, and how badly speed falls as the conversation grows. This article explains what tokens per second actually measures, the two different speeds it hides, and why VRAM is the first thing to check.
It's part of reading performance numbers honestly — see our hub on understanding PC benchmark scores, and it pairs with the local LLM inference rig build and how much VRAM you need.
What Tokens Per Second Measures
A token is a chunk of text — roughly three-quarters of a word on average, so 100 tokens is about 75 words. Tokens per second is how many of those chunks the model processes each second. At 40 tok/s the model produces text roughly twice as fast as a fast human can read, which feels instant; at 5 tok/s it crawls out word by word and feels painful in a chat. So the metric is real and useful — but the single number you see on a spec sheet is almost always just one of two very different speeds.
The Two Speeds Hidden in One Number
- Prompt processing (prefill): how fast the model reads your input — the question, the pasted document, the conversation so far. This is highly parallel and usually very fast, often hundreds or thousands of tok/s.
- Generation (decode): how fast the model writes its reply, one token at a time. This is the slower, sequential speed — and it's the number you actually feel waiting for an answer.
When a benchmark quotes "60 tok/s," it almost always means generation speed, because that's the bottleneck. But if you feed the model a huge prompt — a long document to summarise — the prefill time matters too, and a benchmark that only reports generation hides the pause before the first word appears (the "time to first token"). For chat the generation number dominates; for summarising large inputs, watch both.
Why VRAM Decides Everything First
Here's the caveat that overrides raw speed, exactly as it does for image generation: VRAM decides whether the model runs at all. A model's parameters must fit in memory, and the size depends on the quantisation — the precision the weights are stored at. A 7-billion-parameter model at 4-bit quantisation needs roughly 4–5 GB; the same model at higher precision needs far more, and a 70-billion model can need 40 GB or more even quantised. If the model doesn't fit in your GPU's VRAM, layers spill to system RAM and run on the CPU, and generation speed collapses from tens of tok/s to low single digits. At that point a card's headline tok/s is irrelevant — you're not running on the GPU anymore. This is why VRAM tier is the first thing to check, the same lesson as our Stable Diffusion it/s explainer and what VRAM does for AI work.
Why Speed Falls as Context Grows
One more reality the headline number hides: tok/s is not constant. As a conversation or document grows, the model has to attend to more prior tokens each step, and the key-value cache that stores that context consumes more VRAM. So generation slows down as the context fills, and a long session can run noticeably slower than the benchmark's short-prompt figure. A model that benchmarks at 50 tok/s on a one-line prompt may settle far lower deep into a long chat — and if the growing context pushes you past your VRAM limit, it slows sharply.
How to Read LLM Benchmarks
- Confirm the model and quantisation fit your VRAM first: match the model size and quant level to your card before comparing any speed numbers.
- Separate prefill from generation: the quoted tok/s is usually generation; if you process big prompts, check time-to-first-token too.
- Check the context length tested: a short-prompt figure flatters; ask what the speed is at the context length you'll actually use.
- Compare like-for-like: same model, same quantisation, same context — otherwise the numbers aren't comparable.
Frequently Asked Questions
What is a good tokens-per-second speed for a local LLM? For interactive chat, generation around 15–20 tok/s feels comfortable and anything above ~40 tok/s feels instant, since that outpaces reading speed. Below roughly 5–7 tok/s the reply crawls out and feels frustrating. The figure that matters is generation speed at the context length you actually use, not a short-prompt headline.
Why does VRAM matter more than tokens per second? Because VRAM decides whether the model runs on the GPU at all. The weights — sized by parameter count and quantisation — must fit in VRAM; if they don't, layers offload to system RAM and CPU, and speed collapses to a fraction of the GPU figure. A card with enough VRAM at moderate tok/s beats a faster card that can't fit your model.
Why does my local LLM slow down in long conversations? Because each new token must attend to all prior tokens, and the context cache grows in VRAM as the chat lengthens. Generation speed therefore drops as context fills, and if the growing context pushes past your VRAM limit it slows sharply. Benchmarks run on short prompts hide this, so check the speed at a realistic context length.
The One Thing to Remember
Tokens per second is real but partial: it usually reports only generation speed, hides the prompt-processing (prefill) speed and the time to first token, and falls as context grows. Above all, it's meaningless until the model and its quantisation fit your VRAM — spill to system RAM and speed collapses. Confirm the fit first, then compare generation tok/s at a realistic context length, like-for-like on the same model and quant.
Building a rig to run LLMs locally? Configure an AI workstation online → or talk to our team → and we'll size the VRAM to the models you plan to run, then chase the speed. For business deployments see our AI inference server guide.