What can you actually build, and what does it take to run it?
You do not start with a model. You start with a job to be done: answer questions from your own documents, transcribe calls, power a coding assistant, sort a flood of tickets. This guide maps the real workloads to the models that serve them, and to the OpenMetal box each one belongs on.
Why run these on OpenMetal
The case for owning the box
Every use case in this guide can be rented from a public AI API. Self-hosting on dedicated OpenMetal hardware changes the economics and the control in ways that matter most exactly when a workload becomes real and sustained.
Your data stays on your hardware
Prompts, documents, source code, and call recordings never leave infrastructure you control. For sensitive or regulated data, that is often the deciding factor between “yes” and “not allowed.”
Fixed monthly cost, no per-token meter
A dedicated box is a predictable monthly cost with no per-token or per-GPU-hour billing. For sustained, high-volume workloads, that removes the runaway-bill risk of usage-metered APIs.
No idle-silicon penalty
You are not paying a premium meter while the box sits between jobs. Batch and off-hours workloads run on capacity you already have, which is what makes offline use cases so economical here.
Confidential computing option
For the most sensitive workloads, Intel TDX confidential computing is available on bare metal, so data can stay encrypted even while in use. Ask us about matching it to your compliance needs.
At a glance
Use case to box, on one screen
| Use case | Live / batch | Modality | Example models | Box |
|---|---|---|---|---|
| Conversational assistant | Interactive | Text | Gemma 27B · Llama 70B | H200 / RP6000 |
| RAG (docs Q&A) | Interactive | Text + embedding | embedding + 70B chat | XL v5 + GPU |
| Code assistant | Interactive | Code | Qwen2.5-Coder 32B | GPU or XL v5 |
| Agents / tool use | Interactive | Text | Llama 70B · Mixtral 8x7B | H200 |
| Summarization / extraction | Batch | Text | Llama 70B · Mixtral 8x22B | XL v5 CPU |
| Classification / routing | Single-pass | Text | Phi-4 · Gemma 27B | XL v5 CPU |
| Semantic search | Single-pass | Embedding | embedding models | XL v5 CPU |
| Speech-to-text | Batch / near-live | Speech | Whisper large-v3 | XL v5 CPU |
| Document / image understanding | Either | Vision + text | Qwen2-VL 72B | H200 |
| Image generation | Either | Image | FLUX.1 | RP6000 / H200 |
| Translation / localization | Either | Text | Qwen 72B · Gemma 27B | GPU or XL v5 |
| Data labeling / synthetic data | Batch | Text | DeepSeek-R1 · Mixtral 8x22B | XL v5 CPU |
Box column is a starting recommendation for a typical deployment of each workload. Your model choice, volume, concurrency, and latency targets can move it, which is what a configuration conversation resolves. The exact fit of each model on each box is in the fit guide; each model’s owner and license are in the models reference.
The framing
Every workload sorts on three axes
Before any model or box enters the picture, a use case is defined by three simple properties. Get these right and the hardware choice mostly falls out on its own. Each use case below is tagged with them. These are inference workloads (running finished models); for projects that change a model on your own data, see the training use-case guide.
1. Live or offline?
Does a person wait on the answer in real time, or does the work run in the background where a few minutes or hours is fine?
2. What goes in and out?
Text, code, speech, images, or a numeric “meaning” vector. This is the model’s modality, and it rules out most of the field immediately.
3. One pass or many steps?
Does the model read once and emit a result (fast), or generate an answer one token at a time (slower, and what “tokens per second” measures)?
The explainer covers why these three decide the hardware. The short version: interactive token-by-token work leans toward the GPU boxes (the H200 and RP6000), while batch work and single-pass jobs (search, transcription, classification) often run beautifully on the CPU box (the XL v5). Throughout this guide, a blue tag means GPU-served, a teal tag means CPU-served, and slate means it pairs models or runs comfortably on either.
The workloads
Twelve use cases, mapped end to end
Each card gives the job in plain terms, a concrete example, what it demands, the models that fit, and the box we would put it on. The “fit note” ties back to the reasoning in the fit guide.
Conversational assistant / chatbot
textinteractiveGPUA general assistant your staff or customers talk to: internal knowledge copilot, support agent, HR or IT helpdesk. It holds a conversation, so responses must stream back at reading speed.
Fit note: a 70B dense model is interactive on the H200 at FP8 with room for long context. Drop to a 27B and the XL v5 CPU can serve it interactively too.
RAG: answers from your own documents
text + embeddinginteractivePairs modelsRetrieval-Augmented Generation is the most common enterprise pattern. Instead of hoping the model memorized your data, you store your documents, retrieve the few relevant passages at question time, and hand them to a chat model to answer from. It grounds answers in your real content and cuts made-up facts.
Fit note: RAG naturally splits across boxes. The embedding and vector-search half is single-pass and CPU-native; the answering half is interactive and GPU-served. Long retrieved context grows the KV cache, so plan headroom.
Code assistant / developer tooling
codeinteractiveGPU or CPUAutocomplete, code review, refactoring help, and “explain this repo” tools for your engineers, kept in-house so proprietary source never leaves your network.
Fit note: a 32B coder model fits at FP16 on either GPU, and is interactive on the CPU box for smaller teams. DeepSeek-Coder-V2 is a large MoE, so it is batch-friendly on CPU for bulk jobs.
AI agents & tool use / workflow automation
textinteractiveGPUA model that does not just answer but acts: it calls tools and APIs, reads results, and loops until a task is done (fill a form, reconcile records, triage and route a request). Each step is another model call, so many calls happen per task.
Fit note: agents are where the latency-vs-throughput distinction bites. Many concurrent steps favor a GPU that batches well, and long tool-result context favors the H200’s KV headroom.
Summarization & information extraction
textbatchCPUCondense or pull structured fields out of large volumes of text: contracts, research, meeting transcripts, invoices. Usually run as a background job over a queue, not live.
Fit note: this is the CPU box’s home turf. Because latency does not matter, even a 70B dense or a big MoE runs cost-effectively through an offline queue on abundant system memory.
Classification, routing & moderation
textsingle-passCPUSort, tag, or gate large streams of text: route tickets to the right team, flag policy violations, detect spam, label sentiment. Each item is one quick decision, at high volume.
Fit note: classification rarely needs a large model or a GPU. Small models on the CPU box clear high volume, and the work is single-pass so it dodges the slow token-by-token phase entirely.
Semantic search & recommendations
embeddingsingle-passCPUSearch by meaning rather than keywords, and power “more like this” recommendations. An embedding model turns every document and query into a numeric vector; matches are the nearest vectors.
Fit note: embeddings are a single forward pass with no decode phase, so the token-by-token penalty never applies. This is one of the clearest cases where a GPU is unnecessary.
Speech-to-text & call analytics
speechbatch or near-liveCPUTurn audio into text: transcribe meetings, calls, and voicemails, then feed the text to search, summarization, or compliance review.
Fit note: Whisper is small and CPU-native. Pairing it with CPU-side summarization keeps an entire call-analytics pipeline on one box without touching a GPU.
Document & image understanding
vision + textinteractive or batchGPURead images and documents: extract fields from scanned forms and invoices, describe images, answer questions about a diagram or a screenshot. Uses a vision-language model that accepts pictures alongside text.
Fit note: images inflate the KV cache fast, so vision work wants the H200’s memory headroom when interactive. For bulk extraction where latency is relaxed, the CPU box handles it as a batch job.
Image & media generation
imageinteractive or batchGPUGenerate images from text prompts for marketing, product mockups, or design exploration.
Fit note: image generation is comfortable on either GPU. The CPU box can produce images too, but at seconds-to-minutes each, so reserve it for offline batches. Note on licensing: FLUX.1 [dev] is non-commercial; FLUX.1 [schnell] is Apache-licensed for commercial use.
Translation & localization
textinteractive or batchEither boxTranslate content across languages: product copy, documentation, support replies. Live for chat, batch for a whole content library.
Fit note: a strong multilingual mid-size model covers most needs. Live translation wants a GPU; bulk library localization is a classic CPU batch job.
Data labeling & synthetic data generation
textbatch / offlineCPUUse a large, high-quality model offline to label datasets, generate training examples, or produce evaluation sets. Quality matters more than speed, and it runs unattended.
Fit note: this is exactly where the big Mixture-of-Experts models shine on the CPU box. They store like giants but only compute a fraction per token, so offline they deliver top-tier quality on the big-memory machine without a GPU.
Before you buy
Three decisions that shape your setup
Teaching a model your knowledge: RAG vs long context vs fine-tuning
A frequent question is how to make a model “know” your business. There are three routes, and they have very different cost and hardware profiles:
- RAG (retrieve at question time) is the default for most companies. Your data stays in a searchable store, the model never has to memorize it, and you can update content instantly. It is pure inference, so it fits everything in this guide. Start here.
- Long context (put it all in the prompt) works when the relevant material is small enough to paste in every time. It is simple but expensive: long prompts inflate the KV cache and eat memory, which pushes you toward the H200’s headroom.
- Fine-tuning (bake knowledge into the weights) changes the model itself. This is a training job, not inference, and it is a heavier, separate hardware conversation. Most teams do not need it: RAG covers the majority of “use our data” needs at a fraction of the effort. When you do need it, see the training guides.
Reach for RAG first. Add long context only for the specific queries that need a lot of material at once. Consider fine-tuning last, and only for style, format, or a narrow domain that retrieval cannot capture.
Interactive or batch is the single biggest lever on cost
The same job can be cheap or expensive depending on whether a human waits on it. If you can run work offline (overnight summarization, bulk translation, dataset labeling), the CPU box turns models that would be “too slow to chat with” into economical workhorses. Ask of every workload: does anyone actually wait on this response? If not, batch it, and consider the CPU box.
One box can host several use cases
Most of these workloads do not each need their own server. A single XL v5 CPU box can run transcription, embeddings, search, and classification together. A GPU box can host an interactive assistant while batching an agent workflow. Sizing is about the mix of workloads and their peak concurrency, not one model in isolation, which is the conversation to have when you configure a box.
Let us size the right box for your workload
Tell us the jobs you want to run (the three axes at the top are a good start) and we will map them to models and a configuration across the H200, RP6000, and XL v5 CPU. Proof-of-concept environments are available for testing before you commit.
Pricing for GPU configurations is provided on request.
Talk to an engineer See the fit guideReference
Glossary and models reference
Every term used across these guides, plus the full table of models with owner and license, lives on the AI Deployment Guides hub.
Companion guides: How AI models work (the concepts in plain terms) and the fit guide (which models fit which box, at what precision and speed).
Model and box recommendations here are starting points for a configuration conversation, not guarantees; final sizing depends on your models, volume, concurrency, and latency targets. Model availability and specifications should be confirmed with OpenMetal.