LLM inference is not one workload. It is two with opposite resource profiles, and provisioning them as one uniform GPU fleet means paying for the wrong resource twice.
Most inference platforms are built as if serving a model were a single, uniform thing: pick a GPU, replicate it, put a load balancer in front. That model is convenient, and it is also the reason so many inference deployments feel simultaneously over-provisioned and slow. Generating text with a transformer is actually two distinct phases that stress opposite parts of the hardware. Prefill, the pass that reads your entire prompt and builds the initial cache, is compute-bound: it does a large batch of matrix math in parallel and pushes the tensor cores hard. Decode, the loop that emits one token at a time afterward, is memory-bandwidth-bound: each step does very little arithmetic but must stream the entire growing key/value cache out of GPU memory. One phase wants FLOPs. The other wants bandwidth. A single GPU model cannot be the cheapest answer to both.
The pivot between the two phases is the KV cache, the stored key and value tensors for every token already processed. Prefill creates it. Decode reads it on every step and appends to it. Its size grows with context length and concurrency, and reading it is what makes decode bandwidth-bound rather than compute-bound. Once you see inference as prefill-then-decode with the cache as the handoff, the provisioning question changes. You stop asking which single GPU to buy and start asking how to size two pools, one for each phase, and what has to move between them.
This is a design-philosophy argument, not a product tour. The claim is falsifiable: for sustained mixed-context serving at a fixed hardware budget, two purpose-fit pools beat one uniform fleet, and a per-GPU-hour metered cloud structurally cannot offer the split because its meter punishes exactly the pattern that makes disaggregation pay. OpenMetal’s heterogeneous GPU catalog and whole-card, fixed-cost delivery are what make the split practical to own rather than merely describe.
Key Takeaways
- Prefill and decode have opposite bottlenecks. Prefill processes the whole prompt in parallel and saturates compute; decode emits one token per step and is bound by the bandwidth of reading the KV cache out of HBM. Sizing one GPU for both means overbuying one resource.
- The KV cache is the phase boundary. It scales as roughly 2 x layers x kv_heads x head_dim x precision x sequence_length x batch, so decode throughput tracks memory bandwidth, and the cache is what has to move if you split the phases apart.
- A uniform fleet pays the compromise twice. Provision the fleet for prefill and its bandwidth sits idle during decode; provision it for decode and its compute sits idle during prefill. Either way, part of every card is the wrong resource for half the work.
- Two pools can each be fit to a phase. A compute-dense card (RP6000, 96 GB GDDR7) suits prefill; a bandwidth-and-HBM-dense card (H200 NVL, 141 GB HBM3e at 4.8 TB/s) suits decode. The KV cache moves between them as data over the private network (20 Gbps default, 40 Gbps upgradeable).
- Fixed-cost, single-tenant delivery is what makes the split ownable. Two specialized pools only pay off if you can run both hot around the clock. On a per-GPU-hour meter that is the most expensive thing you can do; on OpenMetal’s whole-card fixed monthly price it costs the same whether the pools are busy or idle.
Two Phases, Two Bottlenecks
Start with what the hardware is actually doing. During prefill the server takes the full prompt and runs it through every layer in one parallel pass, computing the key and value tensors for all prompt tokens at once. This is dense matrix multiplication across the whole sequence, so it keeps the tensor cores busy and its cost scales with prompt length times model size. Prefill is the compute-heavy phase, and on a long prompt it is where the arithmetic lives.
Decode is the opposite. Having built the cache, the model now generates output one token at a time, and each step attends over every prior token by reading the stored keys and values back out of memory. The arithmetic per step is small. The memory traffic is not: the model streams the entire KV cache, plus the weights, out of HBM for every single token. That is why decode throughput tracks memory bandwidth almost linearly and barely responds to added FLOPs. You can watch this signature directly: as you raise batch size, tokens per second plateaus while the streaming multiprocessors sit well below saturation. The card is not compute-starved. It is waiting on memory.
The KV cache is what makes the two phases diverge, and its size is not incidental. It grows as roughly two (keys and values) times the layer count times the number of key/value heads times the head dimension times the precision, multiplied by the sequence length and the batch. Grouped-query attention shrinks the head term, quantizing the cache shrinks the precision term, but the structural fact holds: the cache grows with context and concurrency, and decode has to read all of it, every step. That single data structure is why decode is bandwidth-bound, and it is the thing that has to travel if you ever want to run the two phases on different hardware.
The Uniform-Fleet Compromise
Now consider the conventional design: one GPU model, replicated, serving both phases on every card. The problem is that no single card is well matched to both bottlenecks at once, so the fleet is always carrying resource it cannot use for half the work.
Provision for prefill and you buy compute density. During the decode phase, which for a chatty or long-output workload dominates wall-clock time, that compute sits largely idle while the card strains against its memory bandwidth. Provision for decode and you buy the highest-bandwidth, highest-HBM card you can. During prefill that expensive bandwidth is underused, because prefill is limited by arithmetic, not by memory reads. Either way a fraction of every card in the fleet is the wrong resource for whichever phase is running. You have paid for the compromise, and you pay for it on every node, because the fleet is uniform.
The compromise gets worse under real traffic, where prefill and decode demand shift minute to minute with prompt lengths and output lengths. A retrieval-heavy service with long prompts and short answers is prefill-weighted. A conversational agent with short prompts and long streamed responses is decode-weighted. A uniform fleet has to be sized for the worst case of both, on the same silicon, because any given card might be doing either phase at any moment. That is the structural inefficiency disaggregation removes: it lets the two phases scale on separate hardware, each sized to its own bottleneck, instead of forcing one card to hedge both.
Two Pools, Each Fit to Its Phase
Splitting prefill and decode into separate pools only helps if you can actually put different hardware behind each one, which is where a heterogeneous catalog matters. OpenMetal ships more than one GPU profile on the same bare-metal platform, and the two map cleanly onto the two phases.
For the compute-bound prefill pool, the lever is arithmetic throughput per dollar of card, not raw memory bandwidth, because prefill reads the cache it is building rather than a large existing one. The RP6000 (NVIDIA RTX PRO 6000 Blackwell Server Edition, 96 GB GDDR7) is the value-dense option here: enough memory to hold weights and build prompt caches, with compute that keeps the parallel prefill pass fed. For the bandwidth-bound decode pool, the lever is exactly the resource decode is starved for. The H200 NVL carries 141 GB of HBM3e at 4.8 TB/s, roughly three times the RP6000’s approximately 1.6 TB/s [VERIFY: RP6000 GDDR7 bandwidth]. That bandwidth is the single biggest lever on decode tokens per second, and the larger HBM pool means more room for the KV cache the decode phase is streaming.
The point is not that one card is better than the other. It is that they are better at different phases, and a disaggregated topology lets you buy each for what it is good at instead of paying a blended premium on one card that has to do both. Our companion editorial on how H200 headroom becomes KV-cache and concurrency makes the single-card version of this argument, sizing the decode budget on one card; this piece is the topology version, where the phases live on separate pools you scale on their own curves. You add prefill capacity when prompts get longer, decode capacity when concurrency or output length grows, and neither change forces you to overbuy the other.
What Moves Between the Pools, and Over What
If prefill and decode run on different machines, the KV cache built during prefill has to reach the decode pool that will read it. This is the part of the design that has to be stated precisely, because it is where architects reasonably worry about a fabric bottleneck.
What moves is data, not memory. The prefill pool computes the prompt’s key and value tensors and transfers them to the decode pool, which loads them into its own HBM and continues generation from there. These are two sets of discrete GPUs in separate servers, not one accelerator whose memory is pooled across nodes. On OpenMetal that transfer travels over the private network, which is 20 Gbps by default (two 10 Gbps uplinks, LACP-bonded) and upgradeable to 40 Gbps, the same present-state fabric that already carries multi-node scale-out through pipeline and data parallelism. The KV cache is handed off as a tensor payload the serving framework moves, exactly the kind of cross-node communication the private network is built for.
Two honest boundaries belong here. First, this is a topology for workloads where the prefill-to-decode handoff cost is amortized against meaningful generation, typically long-output or high-concurrency serving, not single-token completions where the transfer would dominate. Sizing that trade-off against your own prompt and output distribution is the actual engineering work, and it is worth measuring rather than assuming. Second, the pools stay discrete: two H200s in one server are two 141 GB cards, not a single 282 GB accelerator, and the same is true across nodes. The design moves the cache as data; it never pretends the memory is shared. That distinction is what keeps the architecture honest, and it is the one to hold onto when a vendor pitch blurs it.
Why Fixed-Cost Delivery Makes the Split Ownable
A disaggregated topology has a commercial precondition that is easy to miss: it only pays off if both pools run hot. Two specialized pools sized to their phases are efficient precisely because each is kept busy with the phase it is good at. The moment you have to throttle either pool to control cost, the compromise you removed from the hardware comes back as a billing problem.
This is where the delivery model decides whether the architecture is practical. On a per-GPU-hour metered cloud, running two specialized pools at high utilization around the clock is the single most expensive utilization pattern the meter charges for, so there is a standing incentive to consolidate back onto one under-committed fleet, which is the inefficiency you were trying to escape. OpenMetal delivers each GPU as dedicated bare metal at a fixed monthly price, single-tenant, with no per-GPU-hour meter running whether the card is busy or idle. That inverts the incentive. Once the pools are yours at a fixed cost, saturating both is free in the only sense that matters to the bill, so you can run prefill and decode fully committed and let the topology do what it is designed to do. The pricing model and the serving strategy point the same direction: size each pool to its phase, and run both flat out.
What the Split Adds Up To
Treating inference as one uniform workload is the design decision most platforms never revisit, and it quietly taxes every card in the fleet with resource that is wrong for half the work. Prefill and decode have opposite bottlenecks, the KV cache is the boundary between them, and once you can put different hardware behind each phase and move the cache between them as data, the compromise disappears. A compute-fit prefill pool and a bandwidth-fit decode pool, each scaled on its own curve over a present-state private network, serve the same traffic with less idle silicon than one card asked to hedge both jobs. OpenMetal’s heterogeneous catalog gives you the two profiles, and its fixed-cost, single-tenant delivery is what lets you keep both pools committed instead of rationing them against a meter. The result is an inference architecture sized to what the hardware is actually doing, phase by phase.
Talk to an Architect
The right split depends on your prompt lengths, output lengths, and concurrency, and it is worth sizing against your own traffic before you commit to a topology. We can help you model a prefill and decode pool split across OpenMetal GPU hardware, size each pool to its phase, and validate the handoff over the private network with a proof of concept on your workload. Tell us how your traffic is shaped and we will help you plan the deployment.
You can also email us at sales@openmetal.io