How AI models work, and what it takes to run one on your own hardware

Under the hood a model is a huge collection of numbers arranged in grids. Each single number is a weight. When people say a model “is 70B,” they mean it holds 70 billion weights and literally nothing else. There is no hidden rulebook and no code full of “if the user asks X, say Y.” The entire behavior of the model lives in the specific values of those 70 billion numbers.

Those positions are what training produced. You start with all the knobs random (the model babbles nonsense), show it an enormous amount of text, and every time it guesses the next word wrong an automatic process nudges the knobs toward a better guess. Do that trillions of times and the knobs settle into positions that produce fluent, useful output. Then you freeze them. That frozen configuration is the finished model, and downloading a model means downloading that list of final positions.

Using the model means feeding your text in, passing it through all those weighted connections (multiply, add, repeat, billions of times), and reading out the most likely next token. A token is roughly a word or word-piece, the unit models read and write in. The model then appends that token and runs the whole pass again for the next one, a word at a time. This process of using the trained model is inference, and two questions fall straight out of it: how much space do all those numbers take, and how fast can you move them through the processor. Those are the only two that decide your hardware.

A weight is a number, and a number can be written with more or less exactness. That choice controls how much space the model takes. A weight might really be 0.1836274. You can store all of it, or round it to 0.184, or roughly to 0.2. Each rounding is smaller to store but slightly less exact. Precision is how much detail you keep per weight, and deliberately keeping less is called quantization. It is the main lever for making a large model fit a smaller box.

This works because most weights do not need their last few digits. The model’s behavior comes from billions of numbers working together, and small rounding errors tend to cancel out across the crowd. So you can shrink every weight and the model barely notices, right up until you shrink too far.

FP means “floating point” (a decimal-style value); INT means “integer.” The number is the bit count: FP16 uses 16 bits (2 bytes) per weight, FP8 uses 8 bits (1 byte), INT4 uses 4 bits (half a byte). Fewer bits means coarser rounding and a smaller file. The same idea appears in CPU tools under the label Q4 (4-bit) or Q8 (8-bit).

Multiply it out and the stakes are clear: 70 billion weights at 2 bytes is about 140 GB, but only about 35 GB at INT4. The same model, a quarter of the footprint, purely from how precisely each weight is stored. That single fact decides half the fit calls you will see in the fit guide, where a table marking a model FP8 or INT4 only is naming the lowest precision you must drop to for that model to fit a given card. Any precision means the model is small enough that even full FP16 fits with room to spare.

There is a second thing taking up memory while a model runs, and it is the one that surprises people: the model’s running memory of the conversation so far, called the KV cache.

A model writes one token at a time, and to choose each new token it looks back at everything said so far. The naive approach would re-read the entire conversation from scratch for every new token, which gets slower with every word and quickly becomes unusable.

That is what the fit notes mean. Load a large model and the weights take most of the card, leaving only a sliver for the notepad:

• “full context”: the weights are small, so there is plenty of room for a long notepad, meaning long conversations and many users.
• “limited KV” or “tight KV”: the weights took most of the card, so little room is left, meaning shorter conversations and fewer concurrent users.
• “image tokens eat KV”: a vision model turns each image into hundreds or thousands of tokens, so one picture fills pages of notepad and squeezes context hard.

“Context” is how long the prompt plus output is, measured in tokens. Longer context means a bigger notepad and more memory. This is why a model can fit yet still leave you only a small usable conversation window. All of these models are Transformers; the notepad exists because of a mechanism called attention, where each token looks back at every previous one.

So far “size” has been a single number: 70B means 70 billion weights, all used for every token. That is a dense model, and for it “how much to store” and “how much work per token” are the same number.

A Mixture-of-Experts (MoE) model breaks that, and it is the most important idea here, because it is why the same model can be a poor fit for a GPU and a great fit for a CPU. An MoE is split into many specialist sub-models called experts, with a small router that, for each token, picks just a couple of experts to do the work. The rest sit idle for that token.

Look at the top two rows. DeepSeek-R1 is larger than Llama 3.1 405B on paper (671B vs 405B), yet does far less work per token (about 37B vs 405B) because it is MoE. The headline number and the per-token cost point in opposite directions. That is exactly why the two boxes disagree about which model is easy to run.

“8x7B” is MoE shorthand for “8 experts of about 7B each,” not 8 times 7. Shared layers make the real total 46.7B, and only about 2 of the 8 experts fire per token (about 13B active).

The two machines in play are the GPU (a chip with thousands of parallel cores, built for this kind of math, but with limited on-board memory) and the CPU (the general-purpose processor in every computer, fewer cores, but paired with large, inexpensive system memory). Two resources decide everything: capacity (where do all the weights physically live?) and bandwidth (how fast can you stream the ones you need into the cores, per token?). Every hardware decision comes down to which of these runs out first, and the GPU and CPU run out of different ones.

Now watch MoE flip. On a GPU, an MoE must load every expert into memory (the router can call any of them next token), so it pays its full total size and is penalized. On a CPU, that same MoE only touches its active experts per token while the plentiful memory holds the rest, so it is rewarded. Same model, opposite outcome:

On a GPU the total size is what bites; on a CPU the active size is what bites. MoE is the architecture where those two numbers differ, which is why it is penalized on the GPU and rewarded on the CPU.

Every request runs in two phases that behave very differently:

• Prefill: the model reads your entire prompt. Because all the prompt’s tokens are known, it processes them in parallel, in one pass. This is fast, limited by raw compute, not memory.
• Decode: the model writes the answer one token at a time, and each new token streams the relevant weights through the processor again. This is serial, limited by memory bandwidth, and it is the slow part. It is what “tokens per second” measures.

Two earlier points now connect. “Image tokens eat KV” means an image becomes thousands of prompt tokens, a big prefill cost and a big chunk of notepad. And the whole CPU speed story is really a decode story: the bandwidth limit bites while writing tokens, so a model that decodes too slowly is a background-job model no matter how quickly it read the prompt.

“How fast is it” is two questions, and sizing a box depends on which you mean:

• Latency: how quickly a single response streams back for one user. This is what a live chat feels like.
• Throughput: how many tokens the box produces per second across all users at once. This decides how many people you can serve on one machine.

The lever connecting them is batching. Because decode spends most of its time waiting on memory, a GPU can generate for many requests at the same time almost as cheaply as for one, so serving 20 users can cost barely more per user than serving one, up to a point. That greatly improves throughput and the economics of the box, but each simultaneous user needs their own KV-cache notepad, so batching spends the very memory the weights are already competing for.

So “many users” is really a throughput-and-batching question, and it trades directly against context length for the same memory. “Can one person chat with a 70B model?” and “can I serve my whole team on this box?” are different questions, and the second is usually the one that sizes the hardware.