Once a model’s own publisher releases a 4-bit checkpoint, GPU selection turns on whether the tensor cores execute that format natively, not on which card holds more memory.
When a model’s publisher ships its own 4-bit checkpoint, the deciding GPU specification stops being memory capacity and becomes whether the tensor cores execute 4-bit arithmetic natively. GPU selection has been a capacity question for as long as the first problem was whether the weights fit at all, and what changed is not the silicon but what the labs release. Thinking Machines released Inkling-Small, a 276B-total, 12B-active sparse Mixture-of-Experts model, under Apache 2.0 in both a BF16 reference checkpoint and a first-party NVFP4 checkpoint, with MXFP8 also supported. The 4-bit artifact is not a community quantization of uncertain quality. It is the publisher’s own release, which means the numeric format has moved from something you do to a model into something the model arrives as.
That shifts the binding constraint. When a 276B-parameter model ships at roughly 150 to 160 GB of weights, capacity stops being the interesting variable on a server carrying two of these cards. What remains is whether the tensor cores compute on those 4-bit values directly or unpack them back to a wider format first. Those two outcomes have identical memory footprints and different performance profiles, and no capacity table will distinguish them. On this model, the card with more memory per GPU is the weaker fit, because it runs the model in the one format its generation does not accelerate.
For a team standing up inference, that resolves several questions at once. A 276B multimodal model with vision and audio input becomes a single node configured with two cards rather than a cluster procurement, so there is no cross-node parallelism to design, no all-reduce riding a network link, and one failure domain to reason about instead of eight. The model’s license is Apache 2.0, so the weights are yours to fine-tune and to keep inside your own trust boundary. And on dedicated hardware at a fixed monthly price, a serving endpoint that runs continuously costs the same as one that sits idle, which is the utilization pattern inference actually produces.
Key Takeaways
- A first-party FP4 checkpoint makes numeric format a hardware requirement. Inkling-Small ships as BF16 and NVFP4 from its publisher, so the format is part of the model’s definition. Selecting a GPU whose tensor cores lack native FP4 means running the vendor’s preferred artifact through a conversion step on every forward pass.
- The RTX PRO 6000 executes the shipped format directly. NVIDIA’s datasheet for the card leads with FP4 precision on its 752 fifth-generation tensor cores, at 96 GB of GDDR7 and 1,597 GB/s per card. Two of these discrete cards hold a roughly 150 to 160 GB weight set in the exact format the lab published, so the quantization buys throughput as well as capacity.
- More VRAM does not resolve this. The H200 NVL carries 141 GB of HBM3e per card and 4.8 TB/s of bandwidth, both larger figures. Its datasheet’s precision table stops at FP8, with no FP4 row, because FP4 arrived with Blackwell. The same 4-bit checkpoint is therefore dequantized for compute, and you get the memory saving without the arithmetic saving.
- You can serve a 276B multimodal model on one node. A single 2U server with two cards, no cross-node parallelism, no network in the critical path, and one failure domain. One GPU is the standard build and the second an orderable add-on, so this is a configuration you specify rather than a default you receive, and the operational surface is materially smaller than a model of this size implies.
- Fixed monthly cost matches how inference actually runs. A serving endpoint is continuously utilized by design, which is precisely what a per-GPU-hour meter charges for. On dedicated GPU servers the bill does not track utilization, and egress is included rather than metered. Pricing on the GPU line is a quote rather than a list figure.
The checkpoint is now part of the hardware specification
Read the three formats as three different weight footprints and the arithmetic is unremarkable. At BF16, 276B parameters is roughly 552 GB. At 8-bit it halves to roughly 276 GB. At NVFP4 it lands near 150 to 160 GB rather than the naive 138 GB, because the non-expert layers stay at higher precision while the expert stacks carry the 4-bit weights.
Each format is also an instruction to the tensor cores, and a GPU generation either has hardware for that instruction or it emulates it. The two cards’ own datasheets answer the question without interpretation. The RTX PRO 6000’s specification table carries a peak FP4 row; the H200’s enumerates FP64, FP32, TF32, BFLOAT16, FP16, FP8, and INT8, and stops there. FP8 is the format Hopper’s generation contributed, and NVIDIA’s key-feature list for the H200 leads with FP8 performance for exactly that reason. FP4 is the format Blackwell’s generation added. So the same file that is a fast path on one card is a memory optimization with a conversion tax on the other. This is why the checkpoint format now belongs in a hardware evaluation alongside memory capacity and bandwidth. It is a compatibility question, and compatibility questions are answered by the instruction set, not by a capacity number.
The sparse MoE structure raises the stakes rather than changing the logic. Inkling-Small routes each token to 6 of 256 routed experts plus 2 shared experts that fire on every token across its 42 layers, so all 256 experts stay resident whatever the activation count says. Our editorial on provisioning Mixture-of-Experts models argues that case in full, and it holds here: the memory you buy follows total parameters. What the format decides is what happens to those resident weights once a token reaches them.
Blackwell: the format the silicon was built for
On the RTX PRO 6000 Blackwell Server Edition the alignment is direct, and it is the card’s own specification rather than an inference from the architecture. NVIDIA’s datasheet lists FP4 precision as the first key feature of its 752 fifth-generation tensor cores, alongside 96 GB of GDDR7 with ECC at 1,597 GB/s and a peak FP4 AI figure of 4 PFLOPS. Across two of these cards the NVFP4 checkpoint has usable room left over for KV cache and activations, and those tensor cores run the weights in the format they arrived in.
The throughput follows the same alignment. NVIDIA’s architecture documentation puts the FP4 tensor rate at twice the FP8 rate on the same silicon, and the datasheet claims up to five times the previous generation’s performance for LLM inference on the strength of FP4 support. Read against a first-party FP4 checkpoint, that ratio stops being a headline number and becomes the reason the format matters: the memory saving and the arithmetic saving arrive together, on weights the publisher already quantized.
The practical consequence is that quantization stops being a trade. In the usual framing you accept some quality risk from a community quant in exchange for fitting a model you otherwise could not run, and you accept that the arithmetic still happens at a wider precision. Neither concession applies here. The quantization is the publisher’s, so there is no third-party recipe to evaluate, and the arithmetic happens at the shipped width. That is what makes this a genuine fit rather than a technicality: the same decision satisfies capacity, avoids quantization-quality guesswork, and lands on hardware that accelerates the result.
Hopper: capacity without acceleration
The H200 NVL, also carried by OpenMetal, is the larger card on every memory line. 141 GB of HBM3e per GPU against 96 GB, and 4.8 TB/s of bandwidth against 1,597 GB/s. Two of them, when both are ordered, present 282 GB. On a capacity-only reading it wins comfortably, and on bandwidth it wins by a factor of three.
Work through the formats and the reading inverts. The 8-bit checkpoint is the format Hopper wants, since Hopper has native FP8, but 276 GB of weights inside 282 GB of card memory leaves nothing for KV cache or activations, and a configuration with no headroom is not a configuration. So the deployable option on Hopper is the 4-bit checkpoint, which fits with generous room, and which Hopper must unpack before it can compute on it. A card chosen for its memory advantage ends up running the model in the format its tensor cores do not accelerate.
None of this makes the H200 a poor GPU, and its bandwidth advantage remains real for workloads whose binding constraint is streaming weights rather than executing narrow-precision arithmetic. For a model published in FP4, the memory comparison is simply not the comparison that decides.
Two discrete cards, and what the routing actually costs
OpenMetal GPU servers ship with one GPU as the standard configuration, and a second card is available as an orderable add-on. So this deployment is a two-card order rather than a default build, and the arithmetic says why: a 150 to 160 GB weight set does not fit on one 96 GB card, and it does not fit on one 141 GB card either. Serving Inkling-Small means ordering the second GPU, and pricing for either configuration comes from a quote rather than a list.
When two cards are ordered, they are two discrete GPUs with no NVLink bridge between them. The reason differs by card. On the RTX PRO 6000 it is a property of the part: NVIDIA’s product brief lists NVLink as not supported on the PCIe card, so there is no bridge to add. The H200 NVL is a different case, because the card does support a 2-way or 4-way NVLink bridge in principle. As OpenMetal currently builds these nodes, that bridge is not present, so a two-card H200 node is also two discrete memories. Either way the design consequence is the same, and neither node should be reasoned about as a pooled accelerator.
Weights shard across the pair under tensor parallelism, which is how a 150 GB weight set occupies two 96 GB cards, but the aggregate figure describes two separate memories rather than one large one. Traffic between them crosses the host PCIe 5.0 fabric, on the order of 60 to 70 GB/s per direction, well below the on-card bandwidth on either side of it.
For a sparse MoE that boundary is where expert-parallel all-to-all traffic lands, and the model’s routing design softens the cost. Two of the eight experts a token touches are shared experts, active on every token, so they can be replicated rather than routed. Only the 6 routed experts per layer produce genuine cross-card movement. A pure top-k design with no shared experts sends a larger fraction of every token across the link. The 12B active-parameter budget keeps the per-token payload small on top of that. This is why a two-card PCIe deployment is workable here rather than merely arithmetically possible.
Both nodes are single-tenant and carry 64 cores and 128 threads across two Intel Xeon 6530P processors, with 88 PCIe 5.0 lanes per socket and 1 TB of DDR5-6400 as the base memory configuration. The host has the lanes and the threads to keep two cards fed, and the tokenization, image patching, and audio encoding paths run without competing for a hypervisor’s share of the machine.
What this model does not let you size yet
One number is missing, and it changes a real design decision. The parent Inkling model card states a context window of up to 1M tokens. The Inkling-Small card does not state one, and third-party trackers list 256K. Those differ by a factor of four, and KV cache scales with context, so the gap sits directly on top of the headroom left after weights. As of this article’s publication date, treat the context window as unresolved and size KV budgets against a figure you have confirmed for the checkpoint you are actually loading.
Two further limits are settled. Inkling-Small takes text, images, and audio as input and produces text only, so it is not a speech or image generation path. And throughput on either card is a measurement rather than a specification. The numbers that circulate for models of this shape come from other people’s hardware and other people’s serving configurations, and OpenMetal has not benchmarked this model, so no tokens-per-second figure appears here. The argument holds regardless of where those figures land, because it turns on which arithmetic the silicon performs rather than on how fast it performs it. Our editorial on KV cache, context, and concurrency headroom covers how to reason about the memory left over once the weights are placed.
Choosing by what you are serving
If you are deploying a model whose publisher ships an FP4 checkpoint, take the Blackwell card, and order the second one when the weight set is larger than a single 96 GB card. The format alignment is the whole argument, and it gives you native execution, capacity headroom, and no quantization recipe to vet.
If your serving mix is mostly models published at BF16 or FP8, where a large single-card memory space keeps weights off the PCIe link entirely, the H200’s 141 GB and 4.8 TB/s are pointed at exactly that problem, and the FP4 question never arises. If you are running both, the format distribution of your model set is the axis to sort on, ahead of the capacity comparison.
And if the model set you expect to serve in a year is genuinely unsettled, weigh the direction of travel: FP4 checkpoints are now arriving from publishers rather than from the community, and that favors silicon that executes them. Patterns for adding GPU nodes to an existing footprint are covered on the OpenMetal GPU clusters page.
Talk to an architect about format-aligned GPU selection
Bring the model set you actually intend to serve, including the checkpoint formats each publisher releases, and we will work through which card executes them natively, whether the weight set needs one card or two, and where the memory headroom lands after weights. OpenMetal GPU servers are single-tenant and available for proof-of-concept validation against your own workload before you commit to a configuration. GPU pricing is quoted rather than listed, so a conversation about configuration and a conversation about cost are the same conversation. New GPU configurations are in our roadmap, so talk to us.
Talk to an OpenMetal architect