Once a model’s weights are resident, the HBM left over is the real serving budget, and it decides how much context and concurrency you can serve at a fixed latency target.
Model-fit is usually discussed as a binary: either the weights fit on the card or they do not. But for a model that fits with room to spare, the more useful question is what the leftover memory buys. During inference the GPU holds two things in HBM: the model weights, which are fixed, and the KV cache, which grows with every token of context and every concurrent request in flight. The weights are the entry fee. The memory that remains after them is the serving budget, and it is what caps how much context length multiplied by how many concurrent sessions the card can hold at once.
This is where the H200’s capacity earns its place for an SRE running an inference service. The H200 NVL carries 141 GB of HBM3e against the H100 NVL’s 94 GB and the H100 SXM’s 80 GB. For any model that fits on both cards with headroom, that extra 47 to 61 GB is not spare capacity sitting idle; it is KV-cache budget you can spend deliberately on longer context windows, larger batches, or more concurrent requests, all at the same tail-latency target. This article is about spending that budget on purpose, and about why OpenMetal’s whole-card, fixed-cost model lets you run it fully committed without watching a meter.
Key Takeaways
- Weights are the entry fee; the rest of HBM is the serving budget. After the model is resident, remaining HBM holds the KV cache, whose size sets the envelope of context length multiplied by concurrent requests.
- The H200’s capacity edge is directly more KV headroom. For a model that fits on both, the H200’s 141 GB leaves 47 to 61 GB more than a 94 or 80 GB H100, and that surplus converts straight into context and concurrency.
- KV cache grows with context and concurrency, not model size. The cache footprint scales with sequence length, batch size, and the model’s layer and head dimensions, so headroom is what lets you raise any of those without evicting.
- Very large models at full precision leave little room; size accordingly. A 70B model at 16-bit precision nearly fills the 141 GB on its own, so the concurrency-headroom argument applies to models sized below the ceiling or served quantized.
- Whole-card, fixed-cost delivery lets you run the headroom hot. OpenMetal gives you the entire H200 at a fixed monthly price, not a time-sliced fraction on a per-GPU-hour meter, so serving at high concurrency around the clock does not change the bill.
The HBM Budget After Weights
When an inference server loads a model, the weights occupy a fixed amount of HBM that does not change with load. Everything the server does after that, every concurrent request and every token of context, draws on what is left. That remainder holds the KV cache, the stored key and value tensors for the tokens already processed, which the model reads on each new token so it does not recompute the whole sequence. The KV cache is the memory structure that actually scales with usage, and it lives in whatever HBM the weights did not claim.
So the capacity that matters for a serving deployment is not the total on the card; it is the total minus the weights. On a 94 GB H100 running a model whose weights take 70 GB, the serving budget is about 24 GB. On a 141 GB H200 running the same model, it is about 71 GB, roughly three times as much room for the KV cache. That difference is not abstract headroom. It is the concrete number of concurrent sessions and the context length you can offer before the card runs out of memory and the server starts rejecting or queueing requests.
What the KV Cache Actually Costs
The KV cache grows along three axes at once: the sequence length being served, the number of requests in flight, and the model’s own shape (its layer count, head count, and head dimension, times two for keys and values, times the precision). The model shape is fixed once you pick the model, so the levers you control at serving time are context length and concurrency, and both draw directly on the HBM budget. Serving frameworks such as vLLM manage this with paged attention, which reduces fragmentation and lets the cache grow efficiently, but no allocator can hand out memory the card does not have. Paged attention makes the budget go further; it does not enlarge it.
This is why headroom is the quantity to reason about. If you want to double the context window you offer, or hold twice as many concurrent sessions at the current context, the KV cache needs room to roughly double, and that room comes from HBM left over after the weights. On the H200 there is materially more of it, so an SRE can make those trades explicitly instead of discovering the ceiling under production load. Our companion piece on single-card inference argues the case for fitting the weights on one card; this is the other half of the same memory story, about what the remaining capacity is for.
Spending the Headroom Deliberately
Given a fixed latency target, the extra KV budget on an H200 can be spent three ways, and the right split is a workload decision. You can raise the context window, which suits document analysis, long conversations, and retrieval-heavy prompts where the model needs to attend over more tokens. You can raise the batch size, which lifts aggregate throughput for a serving endpoint handling many requests, because the memory-bound decode step amortizes better across a larger batch. Or you can raise concurrency, holding more simultaneous sessions resident so more users are served without queueing. These are not mutually exclusive; they are three claims on the same pool, and the larger the pool, the less often you have to choose.
The 4.8 TB/s of HBM bandwidth on the H200 matters here too, because throughput on the memory-bound decode step tracks bandwidth. Headroom decides how much you can hold; bandwidth decides how fast you can serve it. Together they set the real serving capacity of the card, and both favor the H200 for a service that needs to scale context or concurrency without adding GPUs.
The Honest Ceiling
The headroom argument has a boundary worth stating plainly, because it is where a spec-sheet reading goes wrong. A very large model at full precision can consume most of the card on weights alone. A 70B model in 16-bit precision needs roughly 140 GB, which fits on the 141 GB H200 but leaves almost nothing for the KV cache, so at that size the H200’s value is that the model fits at all, not that it leaves concurrency headroom. The concurrency-and-context argument in this piece applies to models sized below that ceiling, or to a 70B-class model served in a quantized format (8-bit or lower) that cuts the weight footprint roughly in half and hands the difference back as serving budget. Sizing the model, the precision, and the target load together is the actual engineering task, and it is one an OpenMetal architect will work through with you rather than leaving to a surprise under load.
Why Fixed-Cost Delivery Lets You Commit the Headroom
Serving headroom is only useful if you can keep it occupied, and that is a commercial property as much as a hardware one. On a per-GPU-hour metered platform, running a card at high concurrency around the clock is exactly the utilization pattern the meter charges most for, so there is a standing incentive to under-commit. OpenMetal gives you the whole H200 at a fixed monthly price, so the card is yours to saturate. Filling the KV budget with real concurrent load, all day, does not change what you pay. The pricing model and the serving strategy point the same direction: hold as much context and concurrency as the 141 GB allows, and run it fully committed.
What the Decisions Add Up To
For an inference service, the number that governs how much you can serve is not the card’s total memory but the memory left after the weights. The H200’s 141 GB leaves substantially more of that budget than a 94 or 80 GB H100 for any model that fits on both, and that surplus is spendable on context, batch size, or concurrency at a fixed latency target. OpenMetal then delivers the whole card at a fixed cost, so the headroom can run committed rather than rationed against a meter. The result is a serving envelope you get to size on purpose.
Request a Proof of Concept
The way to size this is against your own model, precision, and target load. We can provision an H200 so you can measure how much context and concurrency the card holds for your workload at your latency target, and compare it against what you run today. Tell us the model and the service-level objective and we will help you plan the deployment.
You can also email us at sales@openmetal.io