On latency-bound inference, the Model FLOPs Utilization your optimization stack can actually hold is capped by who else shares the box, not by the kernel that runs on it.
Sustained Model FLOPs Utilization, the share of a GPU’s theoretical throughput that turns into useful work over hours of real traffic, is decided below the kernel that runs on the card. The market has started pricing that fact directly. Reporting in August 2026 has a leading AI lab in talks to acquire an inference-optimization company for a figure in the billions (Bloomberg, Fortune, deal reportedly still in talks), and the metric that kind of software sells is MFU. The vendor claim attached to the category is high sustained utilization on real-time workloads, reportedly above 80 percent on specific optimized paths. That claim is credible, and it matters exactly where the number comes from.
The kernel cannot fix what comes next. On a steady, latency-bound serving workload, the ceiling on sustained MFU and the tail latency you can hold is set below the software layer, by tenancy. A compiler and a set of fused kernels decide how efficient a single uninterrupted run can be. Whether that efficiency survives across hours of production traffic depends on whether anything else is allowed to touch the GPU, the PCIe complex, the NIC, or the host scheduler. An OpenMetal GPU node is single-tenant bare metal with full root access, one workload holding the entire server: the GPU, 88 lanes of PCIe 5.0 off a Granite Rapids Intel Xeon 6530P, and the network card, with no hypervisor deciding when your kernels get to run.
What that buys is not an abstract cleanliness. It is the difference between a demonstrated MFU and a dependable one. A real-time inference service that has to answer inside tens of milliseconds, a continuously running generation or simulation loop, or a benchmark you need to reproduce next quarter all want the same thing: a utilization number that holds its shape under load instead of collapsing into a sawtooth when a neighbor gets busy. On a dedicated node the failure domain is also yours to reason about. When something regresses, the set of things that could have caused it is bounded by your own workload, not by tenants you cannot see.
Key Takeaways based on OpenMetal architecture
- MFU has a ceiling the kernel cannot raise. Peak utilization is won in the kernel, but sustained utilization under real traffic is set by the substrate beneath it.
- Single-tenant bare metal removes the contention path. One workload owns the GPU, PCIe complex, NIC, and host, so no cross-tenant scheduler injects jitter into a latency-bound loop.
- The tail latency you can hold is the real deliverable. A dedicated node keeps p99 close to p50 under sustained load, which is what a real-time service sells.
- Two GPUs are two discrete cards, never a pooled accelerator. Each card keeps its own memory (H200 NVL 141 GB HBM3e, or NVIDIA RTX PRO 6000 96 GB GDDR7), and capacity scales per card.
- A bounded failure domain is an operational outcome. When MFU regresses on a single-tenant node, the causes are limited to your own stack, which shortens every root-cause investigation.
OpenMetal builds to the requirement, so the catalog is not the boundary
The shipping GPU catalog is the NVIDIA RTX PRO 6000 Blackwell Server Edition and the NVIDIA H200 NVL, each delivered as a single-tenant node with one GPU as built and a second discrete card orderable. That is the configuration this article measures against, and it is not the edge of what OpenMetal will build. Requirements beyond the listed configurations, whether a different accelerator, a higher GPU density per node, or a different interconnect topology, are evaluated against the requirement rather than picked from a fixed menu. If your utilization target implies a build that is not on the page, that is a design conversation worth starting early.
MFU is a systems number, not only a kernel number
Model FLOPs Utilization is the ratio of the useful floating-point work a model performs to the theoretical peak of the silicon doing it. Two versions of that ratio matter, and they behave differently. Peak MFU is what a single clean run achieves, and it is genuinely a software property: better kernels, better fusion, better memory-access patterns, higher numbers. Sustained MFU is the average that same service holds across hours of real traffic, and it is a systems property. It degrades whenever the GPU has to wait, and the reasons a GPU waits are usually not on the GPU.
Latency-bound serving is the hard case because it has no slack to hide behind. A large batch training job can absorb a stall by keeping the accelerator saturated with queued work. A real-time inference request cannot: it arrives, it must be answered inside a deadline, and any time the kernel spends waiting on a contended PCIe transfer, a delayed host thread, or a scheduler that put another tenant first is time subtracted from both utilization and the deadline. The same event shows up twice, once as a dip in MFU and once as a spike in p99.
Where shared and oversubscribed infrastructure leaks utilization
On multi-tenant GPU infrastructure, several mechanisms quietly cap the utilization an optimization stack can reach. None of them are bugs. They are the cost of sharing.
| Mechanism | Shared or oversubscribed node | OpenMetal’s single-tenant dedicated node |
|---|---|---|
| Host CPU scheduling | Your launch threads compete with other tenants for cores; wake-up latency varies | The Granite Rapids Xeon 6530P is yours; no cross-tenant threads to wait behind |
| PCIe and host I/O | Transfers share a contended path; effective bandwidth moves with the neighbors | 88 lanes of PCIe 5.0 feed your GPU and NIC at full width, with no competing traffic |
| GPU time-slicing | vGPU or time-sliced sharing interleaves your kernels with someone else’s | The GPU runs one workload; no interleaving, no context-switch tax |
| Network path | Shared uplinks add variable queueing to request and response | Dedicated 20 Gbps private networking, upgradeable to 40 Gbps, carries only your traffic |
| Observability | You cannot see the neighbor that caused the regression | The failure domain is your stack, so the cause set is bounded |
The pattern across the rows is consistent. Every shared resource is a place where your sustained MFU depends on a decision made outside your workload, and every one of those places is where a p99 tail is born.
A single-tenant node moves the ceiling back to the kernel
The OpenMetal decision is to remove the shared layer entirely rather than tune around it. The GPU server is dedicated bare metal, single-tenant, with full root access and no oversubscription. One workload owns the whole machine, which means the only scheduler in the path is the one you run, the only PCIe traffic is yours, and the only network queueing is on packets you sent. The optimization stack is then able to reach the ceiling its kernels actually allow, because nothing below the kernel is competing for the resources it needs.
You can measure the difference rather than take it on faith. Sample GPU activity once per second across a sustained serving run and watch the variance, not just the mean:
# Watch sustained SM and tensor activity plus PCIe traffic under real load
dcgmi dmon -e 1002,1004,1005,1009,1010 -d 1000
# 1002 SM active | 1004 SM occupancy | 1005 tensor-core active
# 1009 PCIe tx bytes | 1010 PCIe rx bytes
# On a contended host these fields oscillate; on a dedicated node they hold.The utilization figure worth trusting is not the best interval, it is the p95 of MFU across the whole run:
MFU = (achieved_tokens_per_sec * FLOPs_per_token) / peak_dense_FLOPs_of_card
# Plug in your model's FLOPs per token and the card's published peak dense FLOPs.
# Report the p95 of this ratio sampled per second under sustained traffic,
# not the single best sample. The gap between mean and p95 is the tenancy tax.This is the same reasoning that puts latency-sensitive request paths on dedicated hardware in the first place, an argument this series has made for real-time bidding on high-frequency bare metal, and the same single-tenant premise behind running your own multi-tenant serving on a dedicated H200.
Two cards are two discrete GPUs
Capacity on a GPU node scales by adding a second card, and the honest description of that card matters. When two GPUs are installed they are two discrete accelerators, each with its own memory, not a single pooled device. An H200 NVL brings 141 GB of HBM3e per card; an NVIDIA RTX PRO 6000 brings 96 GB of GDDR7 at roughly 1.6 TB/s per card. A workload that needs more than one card’s memory is sharded across two independent pools by your serving runtime, with the performance depending on your model, software, and topology. Sizing to a single card, when the model fits, keeps the highest-bandwidth path on-package and avoids the cross-card hop entirely.
| Per-card capacity (discrete, not pooled) | NVIDIA H200 NVL | NVIDIA RTX PRO 6000 Blackwell SE |
|---|---|---|
| Memory per card | 141 GB HBM3e | 96 GB GDDR7 |
| Memory bandwidth per card | High-bandwidth HBM3e (per NVIDIA H200 datasheet) | ~1.6 TB/s (1597 GB/s, per NVIDIA datasheet) |
| Node tenancy | Single-tenant bare metal, one or two discrete cards, each card its own memory pool | |
The takeaway from the table is that the second card doubles capacity, not a memory space. Nothing in a two-card node fuses the two pools into one.
What this looks like on OpenMetal
| Field | Detail |
|---|---|
| Product and link | OpenMetal GPU Server (NVIDIA H200 NVL or NVIDIA RTX PRO 6000 Blackwell Server Edition) |
| Delivery model | Single-tenant bare metal, dedicated node, no oversubscription and no shared tenancy |
| Access | Root, full driver and firmware control, your own hypervisor or none |
| Base configuration | One GPU as built (H200 NVL 141 GB HBM3e, or NVIDIA RTX PRO 6000 96 GB GDDR7 at ~1.6 TB/s), Intel Xeon 6530P (32 cores, Granite Rapids), 88 lanes PCIe 5.0 |
| What you must add | Nothing is forced by this thesis; a second discrete GPU is orderable for capacity (two discrete cards, each its own memory, not pooled). Any RAM or drive add is quoted for price and lead time at the time of order |
| Local storage | Boot on 2x 960 GB M.2 (Micron 7450 PRO); data on 6.4 TB NVMe (Micron 7500 or 7600 MAX), with open front bays for expansion |
| Network | 2x 10 Gbps default (20 Gbps private), upgradeable to 4x 10 Gbps (40 Gbps private); included base egress, present-state |
| Software boundary | OpenMetal delivers the dedicated node and provisioning; you bring the CUDA and driver stack, the serving runtime (for example vLLM or TensorRT-LLM), and orchestration |
| Validation path | Prove it on your own model and traffic with a proof of concept, measuring p95 sustained MFU and p99 latency against your deadline |
| Pricing path | GPU pricing is provided on request; contact OpenMetal for a quote |
The utilization you can hold is the one you can sell
An optimization stack raises the number a GPU can hit. Tenancy decides how much of that number survives contact with production traffic. For a workload that has to answer in real time, the two are not interchangeable: a high peak MFU on a contended host still delivers a low sustained MFU and a wide p99, and a customer experiences the tail, not the mean. Removing the shared layer is the design decision that lets the kernel’s ceiling become the service’s floor.
If your workload is latency-bound and steady, size it to a single card where the model fits and validate the sustained number on a dedicated node before you commit. If it needs more memory than one card holds, plan for two discrete cards and shard deliberately. Either way, the utilization worth quoting is the one measured on hardware nobody else can touch.
Prove it on your own workload
The claim in this article is falsifiable, and the way to falsify it is to measure. Bring your model and a realistic traffic profile, run it on a single-tenant OpenMetal GPU node, and compare the p95 of sustained MFU and the p99 of latency against what the same stack holds on shared infrastructure. Request a proof of concept and validate the tail, not the headline.