Why you would bother
Four good reasons and a lot of bad ones. Be honest about which you have.
The reasons that hold up
The data cannot leave. Medical records, legal files, a client's source code, anything under a contract that forbids third-party processing. This is the big one, and it is an absolute rather than a preference — no amount of cheapness or cleverness substitutes.
There is no network. A factory floor, a ship, a vehicle, a laptop on a train.
The volume is enormous and the task is narrow. Classifying ten million records with a small fine-tuned model is genuinely cheaper on your own hardware.
You are learning. Entirely valid, and much the most common honest answer.
The reason that usually does not hold up is cost. A card capable of running a decent model is a four-figure purchase that then sits idle most of the day, and the electricity is not free either. Unless you are saturating it, the arithmetic rarely favours owning one — the hosted equivalent of a hobby project's traffic is often a few pounds a month.
"It runs locally so it is private" is only true if it actually runs locally. A great many desktop AI apps are a local interface to a hosted model. If privacy is the requirement, the thing to verify is that the weights are on your disk and the network is not being touched — not that the icon is in your dock.
What the size actually means
The B in 7B, and why it decides almost everything else.
Billions of numbers, and each one takes up room
A model is a very large pile of learned numbers. A 7B model has seven billion of them. Stored at full precision that is two bytes each, so about 14 GB before anything else — which is why model size and memory are the same conversation.
1–4B phones, laptops, quick classification 7–8B the sweet spot for one consumer card 13–32B noticeably better, needs a good card 70B+ serious hardware or several cards
Bigger is better at the things that need breadth: following an awkward instruction, reasoning across several steps, writing that does not read like a form letter. For narrow, well-defined jobs — is this sentence angry, pull the date out of this line — a small model is often indistinguishable and runs twenty times faster.
Some models list two numbers, like 8×7B or 30B-A3B. The first is how much you must store, the second is how much runs for any given token. They are fast like the small number and hungry like the big one — excellent when you have plenty of memory and modest bandwidth, useless when memory is the thing you are short of.
Quantization
How a 70B model fits on a gaming card, and what it costs you.
Store each number in fewer bits
The weights arrive at 16 bits each. Round them to 8 and the file halves. Round to about 4 and it quarters. The model gets slightly worse at everything, and — this is the surprising part — far less worse than you would expect.
Four-bit is the usual landing spot. It is roughly a quarter of the size for a small enough quality loss that most people cannot pick it out in ordinary use.
Reading a GGUF filename: Q4_K_M means four-bit, K-quant, medium. The letter
after K is the variant — S, M and L trade size against fidelity, and M is the sensible
default. Q8_0 is eight-bit and effectively lossless. Q2_K exists
and you should treat it as a curiosity.
Not on the benchmarks people quote — those hold up well down to four bits. It shows on long multi-step reasoning, on strict format-following, and on languages the model was weakest at to begin with. If you are quantizing hard and your task is any of those three, test on your own examples rather than trusting a leaderboard.
And prefer a bigger model quantized harder over a smaller one at full precision. A 13B at four bits generally beats a 7B at eight, for roughly the same memory.
Will it fit
Three things go in the card, and people usually forget the second one.
Weights, cache, overhead
The weights are the model itself: parameters × bits ÷ 8. A 7B at four bits is about 4 GB.
The KV cache is working memory for the conversation, and it grows with every token in the context. At long context this can be larger than the model. This is the one that catches people: it fits, they raise the context, and it stops fitting.
Overhead is the runtime itself and scratch space — call it most of a gigabyte.
Modern models use grouped-query attention, which shrinks the cache a great deal compared with older ones — a 70B needs roughly 320 KiB per token rather than the megabytes an older architecture wanted. You can also quantize the cache itself to 8 bits and halve it again, for very little cost.
If it does not fit, the runtime will put some layers in ordinary system memory and carry on. It works, and it is roughly an order of magnitude slower for the portion that spilled, because system memory is far slower than the card's own. A model that is 90% on the card is not 90% as fast — the arithmetic on the next section explains why.
How fast it will be
One formula predicts this well, and it is not about how fast your chip computes.
Generation is limited by memory, not maths
To produce one token, the machine reads every weight in the model. Once. So the ceiling is how fast it can read its own memory:
tokens per second ≈ memory bandwidth ÷ model size
A 4 GB model on a card with 1000 GB/s of bandwidth tops out around 250 tokens a second. The same model on system memory at 80 GB/s manages about 20. The compute barely enters into it.
Which explains several things that otherwise look odd. Quantizing makes generation faster, because there is less to read — not because the arithmetic got cheaper. An older card with fast memory beats a newer one with slow memory. And a model that half-fits is dreadful, because every token now waits on the slowest part of the path.
Processing what you sent is compute-bound and highly parallel, so it runs hundreds of times faster per token than generation. That is why a long prompt costs you a pause before the first word and then normal speed afterwards, and why the two numbers are always quoted separately.
It is also why serving several people at once is nearly free: the weights get read once for the whole batch. One user gets the worst deal your hardware has to offer.
What to run it with
Two for trying things, two for serving them.
For getting started
Ollama is the shortest path from nothing to a running model:
ollama run llama3.1 downloads it, loads it, and gives you a prompt. It also
exposes an API on a local port, so it is a perfectly reasonable thing to build against.
LM Studio is the same idea with a window, a model browser and sliders. Good for seeing what the settings do.
Both are wrappers around llama.cpp, which is the engine doing the actual work. Reach for it directly when you want a flag the wrapper does not expose, or you are embedding a model in something you ship.
For serving other people
vLLM is what you use when the thing has users. It batches concurrent requests properly, manages the KV cache far better, and gets multiples of the throughput the desktop tools do under load.
The cost is that it wants full-precision or specific quantized formats and a real GPU. It is server software, not a desktop app.
The rule of thumb: one user at a time, use Ollama and stop thinking about it. Many users at once, use vLLM, because the desktop tools will serialise them and your queue will grow until somebody notices.
Talking to it from code
The good news: almost nothing changes.
Everything speaks the same dialect
Every local runner worth using exposes an OpenAI-compatible endpoint. So swapping a hosted model for a local one is a base URL and a model name, not a rewrite:
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:11434/v1", # Ollama, LM Studio, vLLM…
api_key="not-needed", # required by the client, ignored
)
msg = client.chat.completions.create(
model="llama3.1:8b",
messages=[{"role": "user", "content": "Summarise this ticket in one line."}],
)Which makes a useful pattern easy: develop against a local model, ship against a hosted one, or fall back to local when the network is gone.
Compatible does not mean identical. Tool use, structured outputs and long-context behaviour vary a lot between local models, and a small one will happily ignore a schema that a hosted model would honour. Test the features you actually depend on rather than assuming the endpoint's shape guarantees the behaviour.
Keep the two behind one small interface of your own — a function that takes messages and returns text. Then the local model, the hosted model and a stub for tests are three implementations of the same thing, and none of your application code knows or cares which it got.
Will it fit?
Pick a model, a quantization and a context length, then pick your hardware. Everything on this page, as one bar and one number. Estimates, not promises — but the arithmetic is the real arithmetic.
024 GB
- Weights 0
- KV cache 0
- Overhead 0
- Total 0 of 0
Generation
—
Feels like
—
reading pace is about 7 tok/s
Q4_K_M on a 24 GB card and
watch it overflow, then drop to 32B and watch it land — that is the real decision most people
face. Then take an 8B that fits comfortably and drag the context from 8K to 128K: the weights
never move, and the KV cache eats the card on its own.
What actually goes wrong
Mostly expectations, and one arithmetic error everyone makes once.
Sizing for the weights and forgetting the context
It loads, it runs, everyone is delighted. Then a long document arrives and it falls over, or slows to a crawl as layers spill into system memory.
Size for the longest context you intend to allow, not for an empty chat. Most runners let you cap it — set that cap deliberately rather than discovering it in production.
Expecting hosted-model behaviour
An 8B model is not a small version of a frontier model. It is a different thing that is worse at instructions, worse at tools, and much worse when the task has several steps.
Give it narrow, well-specified jobs and it is excellent. Ask it to be a general assistant and you will spend a fortnight on prompts before concluding what you could have concluded on day one.
Benchmarking with one request at a time
Single-stream tokens per second says almost nothing about what happens when ten people arrive at once.
If it is going to serve users, measure it under concurrency with the server you will actually deploy. The gap between Ollama and vLLM at ten concurrent requests is not a few per cent.
Quantizing to Q2 because it fits
It loads, it generates fluent text, and it is quietly much worse at anything requiring care. Fluency is the last thing to degrade, which makes it a terrible signal.
Below four bits, test on your own examples before believing anything. A smaller model at Q4 is usually the better trade than a big one at Q2.
Assuming local means private
Plenty of desktop AI apps are a local front end to a hosted model. Some local runners phone home for updates and telemetry.
If the requirement is real, verify it: the weights on your disk, and the process making no outbound connections. Watch the traffic rather than trusting the marketing.