Inkling-Small: One FP4 Checkpoint, Two Execution Modes, Two Different Cards

The checkpoint is part of the hardware specification

Take the weight footprints first, because they eliminate most of the options. The BF16 reference checkpoint is 531.9 GB on disk, which is 265.96B parameters at two bytes each, and it needs a cluster. The NVFP4 release is 170.7 GB on disk, or 159 GiB, rather than the naive 138 GB, because the non-expert layers stay at higher precision while the expert stacks carry the four-bit weights. Thinking Machines states a floor of 180 GB of aggregated VRAM for it, about 5% above the weights themselves.

The card also lists MXFP8 under numerics support. It is a format the model can run in. It is not a checkpoint you can download, so reaching it means quantizing the BF16 weights yourself. Whether that is worth doing depends on the card, and here a family name will mislead you. MXFP8 and FP8 are not the same thing. Hopper’s FP8 applies one scaling factor across a whole tensor. MX formats assign a separate scale to every block of 32 values, and on Blackwell that block scaling executes inside the tensor cores while Hopper emulates it in software at significant overhead. Thinking Machines draws the same line, noting that MXFP8 KV cache requires Blackwell and is not offered on H200.

The same distinction runs one level down. FP4 is not NVFP4, and NVFP4 is not one thing either. W4A4 keeps activations at four bits and needs tensor cores that multiply them there. W4A16 unpacks to sixteen bits for the matrix multiply and keeps only the memory saving. Both load the identical weight file.

The sparse MoE structure raises the stakes without changing the logic. Inkling-Small routes each token to 6 of 256 routed experts plus 2 shared experts that fire on every token, so all 256 experts stay resident whatever the activation count says. Our editorial on provisioning Mixture-of-Experts models argues that case in full, and it holds here: the memory you buy follows total parameters, not active ones.

Why a 276B model is cheap on KV cache

Parameter count tells you what the weights cost. It tells you nothing about what the context costs, and on this model the two diverge sharply.

Inkling-Small has 42 layers, and its published configuration marks 35 of them as local attention with a 512-token sliding window. Only 7 layers, every sixth one, attend globally. KV cache scales with sequence length on those 7 layers alone. The other 35 hold a fixed 512-token window no matter how long the sequence runs.

The arithmetic follows directly from the model’s own config: 8 key-value heads at a head dimension of 128 is 2,048 elements per layer per token, so at 16-bit KV each global layer costs 4 KiB per token. Seven of them is 28 KiB per token, plus a flat 70 MiB per sequence for the windowed layers. A dense 42-layer model with the same attention shape would cost 168 KiB per token, six times more.

Blackwell: the arithmetic the silicon was built for

On the NVIDIA RTX PRO 6000 Blackwell Server Edition the format alignment is direct. NVIDIA’s datasheet lists FP4 precision as the first key feature of its 752 fifth-generation tensor cores, alongside 96 GB of GDDR7 with ECC at 1,597 GB/s and a peak FP4 AI figure of 4 PFLOPS. NVIDIA’s architecture documentation puts the FP4 tensor rate at twice the FP8 rate on the same silicon.

Against a first-party W4A4 checkpoint, that ratio is the reason the format matters. The memory saving and the arithmetic saving arrive together, on weights the publisher already quantized. There is no third-party recipe to evaluate and no conversion step on every forward pass.

Two cards leave 33 GiB after weights. At 28 KiB per token that is roughly 1.2 million tokens of KV cache, or about 1.08 million once you allow a few GiB for CUDA context and activations. In practice that is one long session or a few hundred short ones, and Blackwell’s FP8 KV support roughly doubles it. The card with less memory is not short of room on this model.

The kernel path is where this gets less comfortable. Thinking Machines specifies W4A4 against the SM100 Blackwell family. The RTX PRO 6000 is SM120, a different compute capability with a different kernel path. Serving frameworks have been catching up unevenly: vLLM’s backend selection has fallen through to Marlin rather than native NVFP4 kernels on SM120 in recent releases, and SGLang’s published hardware list for Inkling-Small names H200, B200, B300, GB200, and GB300 without listing the RTX PRO 6000. Practitioners do get NVFP4 running on the card through FlashInfer and CUTLASS backends. That picture moves quickly, so check your framework’s release notes as of August 2026 rather than any published account, including this one.

Whether the W4A4 path is production-ready for this model on this card is a question a benchmark answers and a datasheet cannot. It is the one open risk on this recommendation, and it is the first thing we would measure in a proof of concept on the actual hardware, before you commit to a configuration.

Hopper: the same weights, four times the sessions

The H200 NVL, also carried by OpenMetal, is the larger card on every memory line. 141 GB of HBM3e per GPU against 96 GB, and 4.8 TB/s of bandwidth against 1,597 GB/s.

It also cannot execute four-bit activations. The H200 datasheet’s precision table enumerates FP64, FP32, TF32, BFLOAT16, FP16, FP8, and INT8, and stops there, because FP4 arrived with Blackwell. So an NVFP4 checkpoint on Hopper is W4A16 by necessity, and you get the memory saving without the arithmetic saving. That is a real limitation, and it is a supported path rather than a workaround: 2x H200 is the configuration Thinking Machines names for W4A16.

What the extra memory buys is not longer context. It is more of it at once. Two H200 leave 123 GiB after weights, roughly 4.6 million tokens of KV against the Blackwell pair’s 1.1 million. Both nodes clear a single long session comfortably. Only one of them clears four of them simultaneously.

So neither card wins outright. The Blackwell node executes the format the lab shipped. The Hopper node serves more concurrent sessions of it.

Two discrete cards, and what the routing costs

OpenMetal GPU servers are bare metal dedicated servers. They are not delivered as hosted private cloud on OpenStack and Ceph, and you receive the whole machine with root access.

Both GPU SKUs ship with one card as the standard build, and a second is an orderable paid add-on. Serving Inkling-Small at NVFP4 means ordering the second GPU either way, because 159 GiB of weights does not fit on one 96 GiB card and does not fit on one 141 GiB card.

When two cards are ordered they are two discrete GPUs, not a pooled accelerator. The reason differs by card. On the RTX PRO 6000 it is a property of the part: NVIDIA’s product brief lists NVLink as not supported on the PCIe card, so there is no bridge to add. The H200 NVL is a different case, because the card does support a 2-way or 4-way NVLink bridge in principle. As OpenMetal currently builds these nodes that bridge is not present, so a two-card H200 node is also two discrete memories. Either way, plan for two memories.

Weights shard across the pair under tensor parallelism, and the 8 key-value heads divide evenly by two, so the KV cache shards cleanly as well. Traffic between the cards crosses the host PCIe 5.0 fabric, on the order of tens of GB/s per direction, well below the on-card bandwidth on either side of it.

For a sparse MoE that boundary is where expert-parallel all-to-all traffic lands, and this model’s routing softens the cost. Two of the eight experts a token touches are shared and active on every token, so they can be replicated rather than routed. Only the 6 routed experts per layer produce genuine cross-card movement, and the 12B active-parameter budget keeps that payload small.

What this looks like on OpenMetal

Day one on a provisioned node is a driver install, a container runtime, and a serving process. For the four-bit checkpoint across both cards under SGLang, the shape is:

sglang serve \
  --model-path thinkingmachines/Inkling-Small-NVFP4 \
  --quantization nvfp4 \
  --tp 2 \
  --max-running-requests 48

--tp 2 is what shards the weight set and the KV cache across the two discrete memories. --quantization nvfp4 selects the format the argument has been about. On SM120 you will also be choosing a MoE runner backend explicitly rather than accepting the default, which is the practical face of the kernel-maturity caution above. Confirm flags against your serving framework’s current release before you rely on them.

What is still unmeasured

Throughput on either card is a measurement, not a specification. The figures circulating for models of this shape come from other people’s hardware and other people’s serving configurations. OpenMetal has not benchmarked Inkling-Small, so no tokens-per-second number appears here.

The KV figures above are also a ceiling rather than a budget. They assume 16-bit KV and a flat allowance for CUDA context, and they exclude the MoE all-to-all workspace, the vision and audio encoder activations, and the extra cache that speculative decoding adds if you enable the model’s draft layers. Treat a million tokens as the right order of magnitude on the Blackwell pair, not a number to provision against. Our editorial on KV cache, context, and concurrency headroom covers how to reason about what is left once the weights are placed.

One limit is settled. Inkling-Small takes text, images, and audio as input and produces text only, so it is not a speech or image generation path.

Choosing by what you are serving

If you want the lab’s four-bit arithmetic and not only its four-bit memory, take two RTX PRO 6000. The KV headroom is sufficient for long context, the format alignment is the whole argument, and the thing to validate first is the W4A4 kernel path against your own traffic.

If you are serving many long sessions at once, take two H200. Four times the KV budget is the difference between one tenant at a million tokens and several, W4A16 is a supported path rather than a compromise, and it is the configuration the publisher itself names.

If the model set you expect to run in a year is unsettled, weigh the direction of travel: first-party four-bit checkpoints are now arriving from publishers rather than from the community, and that favors silicon that executes them. Patterns for adding GPU nodes to an existing footprint are covered on the OpenMetal GPU clusters page.