Running Llama 3.3 70B on an OpenMetal H200

1. The model at a glance

• Parameters: 70.0 B (dense transformer, not a mixture-of-experts)
• Context window: 128K tokens
• Modality: text in, text out
• Languages: English, German, French, Italian, Portuguese, Hindi, Spanish, Thai
• License: Llama 3.3 Community License (permits commercial use; review the acceptable-use terms)
• Why people deploy it: it lands near the quality of much larger frontier models on reasoning and coding while staying inside a single-GPU memory budget, which makes it the default self-hosted workhorse for chat, RAG, and code assistance.

2. Does it fit on one H200?

Everything turns on how you quantize the weights, because 141 GB is a fixed ceiling:

Numbers are approximate. Weight footprint is parameters x bytes-per-parameter; real usage adds ~1–2 GB CUDA context plus activation and framework overhead, and INT4 trades a measurable amount of quality for the smaller footprint. Treat the table as a sizing guide, not a guarantee.

The practical takeaway: serve Llama 3.3 70B at FP8 on one H200. FP8 weights use roughly half the card, leaving the rest for the KV cache. In vLLM’s published measurements the FP8 KV-cache path retains roughly 97 to 98 percent of baseline quality, so you keep the full 128K context without a meaningful accuracy tax. Drop to INT4 (AWQ or GPTQ) only when you need the extra headroom for very high concurrency and can absorb a measurable, task-dependent quality cost; validate it against your own evaluations rather than a public leaderboard.

FP16 is the case that does not fit on one card. If you have a hard requirement for BF16 weights, that is a two-card job (see below), and this is where we will be honest: in OpenMetal’s build two H200s in one server are two discrete 141 GB GPUs, not one pooled 282 GB accelerator. You serve FP16 70B across them with tensor parallelism (TP=2), not by treating their memory as a single 282 GB pool.

3. What you can actually serve

At FP8 on a single card, the ~70 GB of free memory after weights is your KV-cache budget, and that budget is what sets how much context and concurrency you get. Llama 3.3 70B uses grouped-query attention with 8 key/value heads across 80 layers, which keeps the per-token cache small. The worked math:

KV cache per token = 2 (K and V) x layers x kv_heads x head_dim x bytes
                   = 2 x 80 x 8 x 128 x 1 byte (FP8)
                   ~= 0.16 MB per token

Full 128K-token sequence  ~= 128K x 0.16 MB  ~= 20 GB (FP8)
                          (~40 GB if you run the KV cache at FP16)

So one maximum-length 128K session costs ~20 GB of your ~70 GB free budget at FP8, leaving room for the full context window plus meaningful concurrent traffic. In practice you either serve a few very-long-context sessions or many shorter ones from the same card. Exact requests-in-flight at a given latency target depends on your prompt lengths, output lengths, and serving stack, so size concurrency by benchmarking against your own traffic.

4. Serving stack on the H200

vLLM is the straightforward path: single-GPU, FP8 weights, FP8 KV-cache, full 128K context, no tensor parallelism needed. An illustrative launch (confirm the exact flags against the vLLM release you deploy):

vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --quantization fp8 \
  --kv-cache-dtype fp8 \
  --max-model-len 131072 \
  --gpu-memory-utilization 0.92

--gpu-memory-utilization 0.92 tells vLLM to claim most of the 141 GB card for weights plus KV cache while leaving headroom for CUDA context and activations. Because the model fits on one GPU at FP8, you skip --tensor-parallel-size entirely and avoid all cross-GPU communication overhead.

Other options:

• SGLang and NVIDIA TensorRT-LLM are strong alternatives; TensorRT-LLM with speculative decoding has shown substantial throughput gains for this model in NVIDIA’s published H200-class results.
• NVIDIA AI Enterprise (NVAIE) is available on OpenMetal H200 builds; contact OpenMetal for licensing specifics. (CUDA itself is unlicensed and included.)

A few operational notes:

• Model load is fast. Weights stream off the included local NVMe (Micron 7500 MAX, about 7 GB/s sequential reads). The raw read floor for ~70 GB of FP8 weights is roughly ten seconds; real wall-clock adds host-to-device transfer and format parsing, so expect tens of seconds, not minutes. That matters for restarts and autoscaling, not just first boot.
• Two-card FP16. If you require BF16 weights, add --tensor-parallel-size 2 to shard across the two discrete cards. Expect some cross-GPU communication overhead, which is exactly why a single-card FP8 deployment is faster whenever the model fits.
• Know when you are bandwidth-bound. Decode at this size is memory-bandwidth-bound, not compute-bound: as you raise batch size, watch for tokens per second plateauing while SM utilization stays below saturation. That signature is why the H200’s 4.8 TB/s HBM3e drives throughput more than its raw FLOPS, and it is the number to size against.

5. Is the H200 the right box for Llama 3.3 70B?

For Llama 3.3 70B specifically, the H200 is arguably the sweet-spot card: the model fits FP8 on one GPU, so you avoid multi-GPU communication overhead entirely, and the 4.8 TB/s HBM3e bandwidth keeps token generation fast (inference at this size is memory-bandwidth-bound, not compute-bound).

H200 vs. the RP6000 (RTX PRO 6000 Blackwell) for this model. Both are available on OpenMetal, and 70B at FP8 (~70 GB weights) technically fits on the RP6000’s 96 GB too. But two things separate them for a 128K-context 70B:

• KV-cache headroom. After FP8 weights, the H200 leaves ~70 GB for the KV cache; the RP6000 leaves ~26 GB. A single 128K session already wants ~20 GB, so the H200 has far more room for long context and concurrency.
• Memory bandwidth. The H200’s 4.8 TB/s HBM3e is roughly 3x the RP6000’s ~1.6 TB/s GDDR7. For memory-bound token generation, that bandwidth gap is the single biggest lever on tokens-per-second.

So the RP6000 is the value pick for smaller models and lighter concurrency; the H200 is the right call when you want a 70B served at full context with production throughput. You reach for two discrete H200s only if you need FP16 weights or want to co-locate a second large model, and multi-server over the private network only for frontier models beyond a two-card budget (Llama 405B, large MoE reasoning models), not for 70B.

Two adjacent questions engineers ask:

• Training vs inference. This guide sizes inference. Full fine-tuning of 70B is a different budget, because optimizer states, gradients, and activations stack on top of the weights (several times the inference footprint), which makes it a multi-card or multi-node job. Parameter-efficient methods (LoRA, QLoRA) fit comfortably on one H200.
• Partitioning. The H200 supports MIG if you want to carve one card into isolated instances for smaller co-located models. A single 70B fills too much of the card to bother, but it is worth knowing the option exists.

6. Running it on OpenMetal

OpenMetal delivers the H200 as dedicated bare metal: the whole card is yours on a single-tenant server with no oversubscription and no per-GPU-hour meter running whether the model is busy or idle. You get root-level control of the box, so you run your own OS image, GPU driver, CUDA, and serving stack rather than a provider’s fixed runtime. For a self-hosted 70B that you keep warm for production traffic, that fixed-cost model is the point: you size the box once and run it flat out.

The build ships as a single- or dual-GPU server with 141 GB HBM3e per card and local NVMe sized for a real model library. The included 6.4 TB NVMe holds dozens of 70B checkpoints on-box (a 70B FP8 model is about 70 GB), so you serve from local disk instead of re-pulling weights from object storage on every scale-out, and open drive bays let you expand that library. When one box is not enough, a private network (20 Gbps standard, upgradeable to 40 Gbps) carries multi-node scale-out through pipeline and data parallelism.

Base H200 build vs maxed-out build

It is the same server whether you run one model or push it to its limits. What changes as you scale the box:

For a single Llama 3.3 70B the base build already serves it at FP8 with full context, so maxing out is not about making one 70B faster (that is fixed by one card’s 4.8 TB/s bandwidth). It buys room for more: the second discrete card runs FP16 70B with tensor parallelism or hosts a second large model, 2 TB of RAM (which runs at DDR5-5200 when both slots per channel are populated) widens host-side staging and page cache, the extra NVMe bays turn the box into a larger on-disk model library, and 40 Gbps speeds multi-node scale-out. The two cards stay discrete GPUs, not a pooled 282 GB accelerator.