Running DeepSeek-R1 on an OpenMetal 8-GPU HGX B300

1. The model at a glance

• Parameters: 671B total, about 37B active per token (mixture-of-experts). Memory follows the full 671B, not the active 37B.
• Architecture: MoE transformer built on DeepSeek-V3-Base, using Multi-head Latent Attention (MLA), which compresses the KV cache well below a dense model of similar size.
• Context window: up to 163,840 tokens.
• Modality: text in, text out.
• License: MIT.
• Weights: native FP8 (F8_E4M3) published alongside BF16, so FP8 is the intended serving format rather than a lossy down-quantization.
• Why people deploy it: R1 is an open-weight frontier reasoning model under a permissive license, the model teams reach for when they want frontier-class reasoning on their own hardware instead of a metered API. Its larger siblings in the same family, DeepSeek-V4-Pro (1.6T total / 49B active) and V4-Flash (284B / 13B), both at 1M context under MIT, scale the same sizing method up and down; verify the exact card for the checkpoint you deploy.

2. Does it fit on an HGX B300?

For a mixture-of-experts model, capacity is set by the total parameter count: every expert stays resident even though only a few activate per token. So you size against all 671B. The reference point is one node’s usable aggregate HBM, roughly 2.1 TB across the eight GPUs:

Numbers are approximate. Weights are total params x bytes-per-parameter (FP16 = 2, FP8 = 1, INT4 = 0.5 bytes); the remaining HBM holds the KV cache, activations, and per-GPU context. The 2.1 TB figure is usable aggregate (about 270 GB usable per GPU, times eight), reachable across the node over NVLink, not a single uniform pool.

The practical takeaway: serve R1 at FP8 on one HGX B300 node, with roughly 1.4 TB of usable HBM left over for KV cache and concurrency once the weights are placed. That headroom is what lets a single node hold the full 163,840-token context at meaningful concurrency, which the next section quantifies.

The weights are distributed across the eight GPUs by the serving framework using tensor, pipeline, and expert parallelism. Any GPU can read any other’s HBM over NVLink, so software treats the eight memories as one capacity budget, but the hardware does not make every read local. Where each weight shard and each slice of KV cache sits is a placement decision, and a well-placed model keeps most of a token’s work on one GPU.

A requirement beyond one node, a different GPU density, or a different accelerator entirely is a design conversation with the OpenMetal Engineering team rather than a configuration picked from a page. A proof of concept validates the build before you commit. More on the multi-node case below.

3. What you can actually serve

Only about 37B parameters activate per token, so per-token compute is closer to a mid-size model than to a 671B dense model, which is what makes a large MoE economical once the weights are placed. Two things then govern what you serve, and both favor keeping R1 on one node:

• KV cache. R1 uses Multi-head Latent Attention, which caches a compressed latent of 512 plus 64 values per layer across 61 layers rather than full per-head keys and values. That works out to roughly 68.6 KiB per token in FP16, so a single request at the full 163,840-token context needs about 10.7 GiB, and 64 concurrent requests averaging 32K tokens need on the order of 137 GiB in aggregate. Against roughly 1.4 TB of post-weights headroom at FP8, a single node holds that cache comfortably, so memory capacity is not the binding constraint it becomes when the same model spans several smaller servers. Actual concurrency and throughput depend on your serving configuration and are measured in a proof of concept. Compute the exact budget from the config of the checkpoint you deploy. 
• Interconnect. Expert parallelism routes each token to the GPUs that hold its experts. On a single B300 node that routing rides the 1.8 TB/s NVLink and NVSwitch fabric between GPUs instead of crossing a server-to-server network, which is the structural reason the same model serves more smoothly here than as a cross-server fleet. A read from a GPU’s own HBM (about 7.7 TB/s) is still faster than one across NVLink, so the framework still works to keep each token’s experts local where it can.

Measured tokens per second and concurrency for R1 on an HGX B300 node require an OpenMetal benchmark and are intentionally not asserted here. A paid proof of concept can help measure them on your prompts, context length, and concurrency target. 

4. Serving stack

vLLM and SGLang both support large MoE models with expert parallelism and native FP8 weights, and SGLang in particular is widely used for DeepSeek-class MoE serving; NVIDIA TensorRT-LLM is a third path with its own DeepSeek-V3/R1 support. On a single node you run with tensor, pipeline, and expert-parallel degrees matched to eight GPUs rather than to a multi-server topology, which simplifies the parallelism plan compared with a cross-server build. Confirm the exact flags against the serving release you deploy.

Two platform notes:

• Precision is two decisions. NVFP4 is a weight and compute format that feeds Blackwell Ultra’s FP4 tensor cores and shrinks the weight footprint; the KV cache is a separate allocation with its own precision (commonly FP16, FP8, or INT8) set in the serving framework. Native FP4 compute does not automatically store the KV cache in FP4, so confirm the cache format your stack supports rather than inferring it from the GPU.
• Weights live on local NVMe. On-box NVMe holds the full FP8 checkpoint many times over, so the node serves from local disk instead of re-pulling hundreds of gigabytes from object storage on every restart. If you want a larger on-box model library, additional NVMe is orderable; component pricing and lead time are quoted at the time of order.

5. Is the B300 node the right box for it, or a multi-server H200 build?

Both paths run DeepSeek-R1 on OpenMetal, and the honest split is about what you can order today versus what the workload actually needs.

• The shipping path, orderable now: a multi-server H200 build. On OpenMetal’s NVIDIA H200 NVL cards, R1 at FP8 is about six discrete cards across three single-tenant servers (a server holds up to two discrete cards, each with its own 141 GB, never pooled). Expert routing crosses the private network between nodes, which makes the 40 Gbps upgrade earn its place. This is a proven configuration you can deploy today, and for steady volume that fits a small fleet it is the pragmatic home. The DeepSeek-R1 on H200 guide covers it in full.
• The headroom path: the eight-GPU HGX B300 node. It holds the whole model plus a large KV cache inside one NVLink and NVSwitch domain, so the cross-GPU expert traffic stays on a 1.8 TB/s intra-node fabric rather than the server-to-server network, and the fleet collapses from three servers to one node with a single operational and failure domain. Blackwell Ultra adds FP4 compute and about 7.7 TB/s of per-GPU HBM bandwidth. It is delivered as a build-to-requirement engineered proof of concept scoped by the OpenMetal Engineering team, with no published availability date.

The decision rule: if R1 is a committed, high-concurrency production endpoint and you want its expert routing off the network and the whole working set on one tightly-coupled node, the B300 node is the better-matched substrate. If you want a path you can order today, or your volume fits a smaller fleet, the H200 build serves it now. Both keep the memory honest: the H200 cards are discrete and never pooled, and the B300 node’s aggregate HBM is reachable over NVLink, not one uniform pool.

One adjacent note: this guide sizes inference. Fine-tuning or continued pre-training of a 671B MoE stacks optimizer states, gradients, and activations on top of the weight footprint, which is a substantially larger cluster problem needs to be sized separately.

6. Running it on OpenMetal

OpenMetal delivers the HGX B300 node as single-tenant, dedicated bare metal: the GPUs are yours, with no oversubscription and no per-GPU-hour meter running whether the model is busy or idle. For an always-on reasoning endpoint, which by design keeps a node busy around the clock, that fixed monthly cost is what makes the economics work, and egress is included rather than metered per gigabyte. You get full root and out-of-band management, so you run your own OS image, GPU driver, CUDA, NCCL, and serving stack. Because a node this size is delivered as an engineered build, the OpenMetal Engineering team scopes and validates it against your reasoning workload as a proof of concept before you commit.