Which models fit which OpenMetal box, and how fast they run
A reference for matching open-weight models to the H200, RP6000, and XL v5 CPU. GPU fit is bound by a model’s total size in card memory; CPU throughput is bound by its active size and memory bandwidth. If those terms need unpacking, start with the explainer; otherwise the tables below are the answer.
The hardware
Three boxes, two constraints
The two GPU boxes are governed by capacity (total size against fixed card memory); the CPU box is governed by bandwidth (active size and memory speed). Read each matrix through its own binding column, tinted to match. New to those terms? See the explainer. These tables are for inference; to fine-tune a model, see the training fit guide.
H200
The larger-memory GPU. Fits more weights at higher precision and leaves more room for KV cache, so longer context and more users.
Full hardware specs →RP6000
The smaller-memory GPU. The same capacity logic with a tighter ceiling, so several models drop to INT4 or stop fitting.
Full hardware specs →XL v5 CPU
Capacity is a non-issue, everything fits. The story is speed: the combined memory bandwidth of both sockets, plus Intel AMX, decides tokens per second.
Full hardware specs →
Both matrices are for a single GPU card, the configuration these guides cover. A model that does not fit one card is marked accordingly, with the CPU box (or a smaller or MoE model) as the alternative shown in the tables.
Reading the Type column: Dense and MoE are explained in the explainer; ASR is automatic speech recognition (audio to text, such as Whisper); Diffusion is image generation (such as FLUX.1); VL is vision-language, a text model that also accepts images (such as Qwen2-VL).
GPU fit · H200 and RP6000
Capacity binds, so read the Total column
The two right-hand columns are the single-card GPU boxes, the H200 (141 GB) and the RP6000 (96 GB). Each cell names the lowest precision the model must drop to in order to fit that card, plus any KV-cache pressure. “Any precision” means even full FP16 fits with room to spare.
| Model | Total (store) | Active / token | Type | H200 · 141 GB | RP6000 · 96 GB |
|---|---|---|---|---|---|
| Small language models · under 8B (fit trivially at any precision) | |||||
| Gemma 2 2B | 2.6B | 2.6B | Dense | Trivial | Trivial |
| Llama 3.2 3B | 3.2B | 3.2B | Dense | Trivial | Trivial |
| Phi-4-mini | 3.8B | 3.8B | Dense | Trivial | Trivial |
| Qwen2.5 7B | 7.6B | 7.6B | Dense | Trivial | Trivial |
| Speech, image, and mid-to-large models | |||||
| Whisper large-v3 | 1.55B | 1.55B | ASR | Trivial | Fits |
| FLUX.1 [dev]NC | 12B | 12B | Diffusion | Fits easily | Fits |
| Phi-4 | 14.7B | 14.7B | Dense | Any precision | Fits |
| Qwen3.6-27B | 27B | 27B | Dense · VL | FP16 (images use KV) | FP16 |
| Gemma 2 27B | 27.2B | 27.2B | Dense | FP16 | FP16 |
| Qwen2.5-Coder 32B | 32.5B | 32.5B | Dense | FP16 | FP16 |
| Qwen3.6-35B-A3B | 35B | ~3B | MoE · VL | FP16, roomy | FP8, roomy |
| Mixtral 8x7B | 46.7B | ~13B | MoE | FP8, roomy | FP8 |
| Llama 3.3 70B | 70B | 70B | Dense | FP8, full context | FP8, limited KV |
| Qwen2.5 72B | 72.7B | 72.7B | Dense | FP8 | FP8, tight |
| Qwen2-VL 72B | 73B | 73B | Dense · VL | FP8 (images use KV) | FP8, tight |
| Command R+NC | 104B | 104B | Dense | FP8, moderate KV | INT4 only |
| Mistral Large 2NC | 123B | 123B | Dense | FP8, tight KV | INT4 only |
| Mixtral 8x22B | 141B | ~39B | MoE | INT4 only | INT4 only |
| DeepSeek-Coder-V2 | 236B | ~21B | MoE | INT4 only, tight | No fit |
| Llama 3.1 405B | 405B | 405B | Dense | No fit, use CPU | No fit |
| DeepSeek-R1 | 671B | ~37B | MoE | No fit, use CPU | No fit |
On a GPU the active column does not help you: Mixtral 8x22B is forced to INT4 by its 141B total, even though only about 39B are active, because every expert must still be resident in memory.
NC marks a research or non-commercial license: self-hosting for evaluation is fine, but commercial use needs a separate license from the model owner. Owner and license for every model are in the models reference on the hub.
CPU usability · XL v5
Bandwidth binds, so read the Active column
This is the XL v5 CPU box (dual-socket, Intel AMX). Everything fits in its 1 to 2 TB of RAM, so “fits” is not the question. The question is whether it is worth running, tiered by practical usability and ordered best first. CPU serving typically runs 4-bit (Q4) quantization.
| Model | Total (store) | Active / token | Type | On the XL v5 | Notes |
|---|---|---|---|---|---|
| Small language models · under 8B (fast, interactive) | |||||
| Gemma 2 2B | 2.6B | 2.6B | Dense | Interactive | Tiny and fast; high-volume classification and routing |
| Llama 3.2 3B | 3.2B | 3.2B | Dense | Interactive | Edge-class model; snappy on CPU at high throughput |
| Phi-4-mini | 3.8B | 3.8B | Dense | Interactive | Strong small reasoner; cheap to run many instances |
| Qwen2.5 7B | 7.6B | 7.6B | Dense | Interactive | Capable 7B; comfortable CPU workhorse |
| Speech, image, and mid-to-large models | |||||
| Whisper large-v3 | 1.55B | 1.55B | ASR | CPU-native | Batch and near-real-time transcription; no GPU needed |
| Phi-4 | 14.7B | 14.7B | Dense | Interactive | The “just serve it on CPU” model |
| Gemma 2 27B | 27.2B | 27.2B | Dense | Interactive | Strong quality for its size |
| Qwen3.6-27B | 27B | 27B | Dense · VL | Interactive | Current-gen dense; vision adds cost per image |
| Qwen2.5-Coder 32B | 32.5B | 32.5B | Dense | Interactive | Good CPU coding-assistant candidate |
| Mixtral 8x7B | 46.7B | ~13B | MoE | Interactive | Sweet spot: runs like a ~13B, fits like a 47B |
| Qwen3.6-35B-A3B | 35B | ~3B | MoE · VL | Interactive | Only ~3B active: runs like a 3B, fits like a 35B, the standout CPU MoE |
| FLUX.1 [dev]NC | 12B | 12B | Diffusion | Batch | Seconds per image; offline image generation |
| Mixtral 8x22B | 141B | ~39B | MoE | Batch-viable | Behaves like a ~39B, not a 141B |
| DeepSeek-Coder-V2 | 236B | ~21B | MoE | Batch-viable | Low active count makes it viable despite 236B total |
| DeepSeek-R1 | 671B | ~37B | MoE | Batch-viable | More usable than the 405B despite 2.7x the total |
| Llama 3.3 70B | 70B | 70B | Dense | Batch-only | High quality, slow single-stream; background jobs |
| Qwen2.5 72B | 72.7B | 72.7B | Dense | Batch-only | As above |
| Qwen2-VL 72B | 73B | 73B | Dense · VL | Batch-only | Vision adds cost; batch document and image work |
| Command R+NC | 104B | 104B | Dense | Batch, slower | Use if this model’s quality is required |
| Mistral Large 2NC | 123B | 123B | Dense | Batch, slower | Offline only |
| Llama 3.1 405B | 405B | 405B | Dense | Not recommended | Fits in memory but too slow here; use a GPU box or a smaller or MoE model |
Two positive tiers, not just a warning. The dense 405B is the one model to steer away from on this box, because every one of its 405B weights is touched per token. Every MoE sits above it despite larger headline numbers.
A note on the dual-socket design: the XL v5 has two processor sockets, each with eight memory channels, so the usable memory bandwidth that sets CPU throughput is the combined total of both (the shipped 1 TB config populates both sockets at full DDR5-6400). The tiers above assume serving that uses both sockets (NUMA-aware: memory interleaved, or one instance pinned per socket); a naive single-socket process sees roughly half the bandwidth and effectively drops a tier. As with every throughput call here, OpenMetal can benchmark your model on the box.
The short version
What each box is for
Interactive serving of dense models up to roughly 70B to 120B (the H200 has the headroom; the RP6000 follows the same logic with a tighter ceiling). Capacity is the wall, so more memory means higher precision and longer context. Large MoE models are less efficient here because you pay for every expert in limited memory.
Best on dense models up to about 32B, any low-active MoE, and non-chat inference (transcription, embeddings, batch image generation) where the token-by-token cost does not apply. The 1 to 2 TB of RAM makes the large MoE models unusually viable. It is not the box for interactive dense 70B and up. That boundary, small dense and MoE yes, large dense no, is the whole story.
Which of these maps to a real workload? See the companion use-case guide for chat, RAG, agents, search, transcription, and image generation mapped to these exact models and boxes.
Let us size the right box for your models
Tell us which models you want to run and how (live chat, background jobs, how many users) and we will map them to a configuration across the H200, RP6000, and XL v5 CPU. Proof-of-concept environments are available for testing before you commit.
Pricing for GPU configurations is provided on request.
Talk to an engineer See the use casesReference
Models reference and glossary
The full table of every model with its owner and license, plus a glossary of every term used across these guides, lives on the AI Deployment Guides hub.
Companion guides: How AI models work (the concepts) and the use-case guide (workloads mapped to models and boxes).
Fit calls and usability tiers are based on model architecture, precision, and memory bandwidth. Actual throughput depends on your model, precision, context length, and concurrency; we can benchmark your target models on request. Model availability and specifications should be confirmed with OpenMetal.