Running DeepSeek-R1 671B on an OpenMetal H200

1. The model at a glance

• Parameters: 671B total, ~37B active per token (mixture-of-experts). The Hugging Face card shows 671B in its evaluation table and 684B in the Safetensors metadata; size on 671B and let the rounding to six cards absorb the difference.
• Architecture: MoE transformer, built on DeepSeek-V3-Base. The R1 card points to the DeepSeek-V3 repository for layer-level architecture detail.
• Context window: 128K tokens
• Modality: text in, text out (no vision)
• License: MIT (permits commercial use, modification, and distillation)
• Weights: native FP8 (F8_E4M3) weights are published alongside BF16, so you serve FP8 without quantizing the model yourself.
• Distilled variants: DeepSeek also ships distilled versions from 1.5B to 70B, fine-tuned onto Qwen2.5 and Llama 3 backbones (see the right-sizing section below).
• Why people deploy it: R1 is an open-weight frontier reasoning model under a permissive license, which makes it the model teams reach for when they want frontier-class reasoning on their own hardware instead of a metered API. The same sizing applies across the DeepSeek-V3/R1 family (V3, R1, and the newer V3.2 line share the ~671-685B MoE shape); verify the exact card for the checkpoint you deploy.

2. Does it fit on an H200?

For a mixture-of-experts model, capacity is set by the total parameter count: every expert has to be resident in GPU memory even though only a few activate on any given token. So you size against all 671B, not the 37B active. DeepSeek publishes native FP8 weights, so FP8 is the row to plan around:

Numbers are approximate. Card count is ceil(total params x bytes-per-parameter / 141 GB) for the weights (FP16 = 2, FP8 = 1, INT4 = 0.5 bytes), plus headroom for the KV cache, activations, and per-card CUDA context, rounded up to an even count because tensor- and pipeline-parallel degrees split cleanly that way. Apply the same formula to any model and precision to size its fleet. INT4 trades a measurable amount of quality for the smaller footprint.

The practical takeaway: serve DeepSeek-R1 at FP8 across about six discrete H200 cards, which on OpenMetal is three two-card servers. INT4 roughly halves the weight footprint to ~336 GB and about three cards, at a quality cost you validate against your own evaluations; FP16 doubles it to ~1.34 TB and about ten cards. Because DeepSeek ships FP8 natively, serving at FP8 is the intended path, not a lossy conversion from BF16, which is why it is the recommended row rather than a compromise.

The cards that hold R1 are discrete GPUs, each with its own 141 GB. You shard the model across them with tensor, pipeline, and expert parallelism; you never address their memory as one pooled 671 GB space. The figure above shows each card holding its own weight shard rather than contributing to a single pool.

A build at this scale, with higher GPU density per node or a larger multi-node topology, is exactly the kind of requirement the OpenMetal catalog is actively expanding to meet. We describe the shipping configurations here (single-card as built, a second discrete card orderable), and a requirement beyond them is scoped to the workload with the OpenMetal Engineering team rather than picked from a fixed menu. A proof of concept is available to validate the deployment before you commit. Talk to us about the shape your R1 build needs.

3. What you can actually serve

Only ~37B parameters activate per token, so per-token compute is closer to a mid-size model than to a 671B dense model, which is what makes a large MoE economical to run once the weights are placed. Two things then govern what you serve:

• KV cache. The DeepSeek-V3/R1 family uses Multi-head Latent Attention (MLA), which compresses the key/value cache well below what a dense model of similar size would need. The practical effect for an operator is that once the weights are placed, a given card holds more concurrent or longer-context sessions than the raw model size suggests. The R1 card defers the exact attention parameters to the V3 repo, so compute the precise per-token KV budget from the config of the checkpoint you deploy rather than from this guide. 
• Interconnect. Expert parallelism routes each token to the cards that hold its experts, so serving R1 across nodes shuffles activations between discrete cards and across servers. That traffic rides the private network between nodes, which makes the network a design variable rather than an afterthought. Where it fits, keep expert-parallel groups within a single server so the heaviest routing stays on-box, and use the 40 Gbps private-network upgrade for the cross-node expert and pipeline traffic.

4. Serving stack on the H200

vLLM and SGLang both support large MoE models with expert parallelism and native FP8 weights, and SGLang in particular is widely used for DeepSeek-class MoE serving. The deployment is multi-GPU and multi-node, so you run with tensor, pipeline, and expert-parallel settings matched to your card and server count (confirm the exact flags against the serving release you deploy).

An illustrative shape below, not a copy-paste config:

vllm serve deepseek-ai/DeepSeek-R1 \
  --quantization fp8 \
  --tensor-parallel-size 2 \
  --pipeline-parallel-size 3 \
  --max-model-len 131072

Other notes:

• NVIDIA TensorRT-LLM is a third option with its own DeepSeek-V3/R1 serving path.
• NVIDIA AI Enterprise (NVAIE) could be made available on OpenMetal H200 builds; contact OpenMetal for for availability and licensing specifics. (CUDA itself is unlicensed and included.)
• Weights live on local NVMe. The included 6.4 TB on-box NVMe holds the full ~671 GB FP8 checkpoint many times over, so each node serves from local disk instead of re-pulling hundreds of gigabytes from object storage on every restart or scale-out. If you want a larger on-box model library, additional NVMe is orderable; component pricing and lead time are quoted at the time of order.

5. Is the H200 the right box for it?

DeepSeek-R1 runs as a multi-server build on OpenMetal, and for a model this size the H200 is the right GPU. It spans several servers on any card we offer, so the real choice is which card makes that fleet smaller and faster.

H200 vs. the RTX PRO 6000 for this model. Both are available on OpenMetal, and R1 is multi-server on either, but two things favor the H200 at this scale:

• Density. At 141 GB per card, the H200 reaches 671 GB of FP8 weights in about six cards; the RTX PRO 6000’s 96 GB would take about eight. Fewer discrete cards means fewer parallelism hops and a simpler fleet for the same model.
• Memory bandwidth. The H200’s 4.8 TB/s HBM3e is roughly 3x the RTX PRO 6000’s ~1.6 TB/s GDDR7. For memory-bound token generation, that bandwidth gap is the single biggest lever on tokens-per-second per card.

Plan the fleet in even card counts, since tensor- and pipeline-parallel degrees split cleanly that way. The RTX PRO 6000 remains the value pick for models that fit on one or two cards; for a 671B MoE that is already a multi-server build, the H200 is the denser, faster foundation. Scaling R1 to a higher GPU density per node or a larger multi-node topology is a configuration we build to the requirement with Engineering and validate with a proof of concept. For a code-focused DeepSeek MoE, the DeepSeek-Coder-V2 guide covers the same multi-server sizing for that model.

One adjacent question engineers ask: training vs inference. This guide sizes inference. Fine-tuning or continued pre-training of a 671B MoE stacks optimizer states, gradients, and activations on top of the already multi-server weight footprint, which makes it a substantially larger cluster problem; size it separately.

If a full R1 fleet is more than you need: use the distilled variants

DeepSeek ships distilled versions of R1 from 1.5B to 70B parameters, fine-tuned onto Qwen2.5 and Llama 3 backbones. They carry much of R1’s reasoning behavior at a fraction of the footprint, and they change the hardware answer entirely. DeepSeek-R1-Distill-Llama-70B, for example, is a 70B dense model that fits a single H200 at FP8 (about 70 GB of weights), the same single-card profile as Llama 3.3 70B on the H200; the smaller 32B and 14B distills fit comfortably on one card, including the RTX PRO 6000. If your workload wants R1-style reasoning but not the full 671B fleet, start with a distilled variant on one card and move to the full model only when the quality gap justifies the capacity.

6. Running it on OpenMetal

OpenMetal delivers each H200 as dedicated bare metal: the cards are yours on single-tenant servers with no oversubscription and no per-GPU-hour meter running whether the model is busy or idle. For a self-hosted frontier model you keep warm for production traffic, that fixed monthly cost is what makes the economics work: you size the fleet once and run it flat out, and egress is included rather than metered per gigabyte. You get root on every box, so you run your own OS image, GPU driver, CUDA, and serving stack across the fleet, and the full H200 box specification lives on its hardware details page.

Multi-server R1 runs as discrete cards across single-tenant servers, connected by a private network (20 Gbps standard, upgradeable to 40 Gbps) that carries the tensor, pipeline, and expert-parallel traffic between nodes. Because a build this size spans several servers and leans on that network, OpenMetal delivers it as an engineered build validated with the Engineering team, with proof-of-concept access so you can confirm it against your own workload before you commit.