AI Deployment Guides

The guides

Six guides, two tracks

The inference track is about running finished models; the training track is about fine-tuning a model on your own data. Each track has an explainer (the concepts), a fit guide (the reference tables), and a use-case guide (real workloads).

Reference

Models in these guides

Every model named across the guides, with its owner and license. All are open-weight (downloadable and self-hostable), and OpenMetal runs whichever you choose. Honoring the license is yours: most permit commercial use (many under permissive Apache or MIT terms), a few carry conditions (for example a large monthly-active-user threshold), and three are research/non-commercial only (^NC), meaning commercial use needs a separate license from the owner. Confirm the current license for your own use before deploying.

“Yes, conditions” means commercial use is permitted but the license adds terms (for example the Llama community license requires a separate agreement above 700 million monthly active users, and the Qwen license adds an attribution and large-MAU clause). FLUX.1 [schnell] is available under Apache 2.0 for commercial use; the [dev] weights referenced in these guides are non-commercial. Licenses change: verify the current terms on the model’s official page before deploying.

Reference

Glossary

Every term used across the six guides, in one place.

Inference

Using a finished model to produce output, as opposed to training it.

Training

Changing a model’s weights so it behaves differently. The opposite of inference, which uses a frozen model.

Self-hosting

Running a model on your own hardware instead of a third-party API, so data and cost stay under your control.

Open-weight vs open source

Open-weight means the weights are publicly downloadable (the only kind you can self-host). Open source is narrower: a permissive license (Apache or MIT) that also allows commercial use.

Token

The unit models read and write in, roughly a word or word-piece. Context and speed are both counted in tokens.

Weight / parameter

One of the billions of learned numbers that make up a model. “70B” means 70 billion of them.

Context

How long the prompt plus output is, in tokens. Longer context uses more memory.

Dense

Every weight is used for every token. Total and active size are the same number.

MoE (Mixture-of-Experts)

Many experts stored, only a few used per token, so active size is far below total. Runs faster than its size suggests; especially strong on the CPU box.

Total vs active size

Total is what you store; it binds the GPU (fills card memory). Active is what you touch per token; it binds the CPU (drives tokens/sec).

SLM (small language model)

A model roughly under 8B weights: fast, cheap, and often CPU-native. Strong for classification, routing, search, and high-volume work.

FP16 / FP8 / INT4 (Q4)

Bytes used per weight: 2 / 1 / 0.5. Lower means smaller and faster with a slight quality loss.

Quantization

Deliberately storing weights at lower precision to shrink a model so it fits a smaller box.

KV cache

The model’s running memory of the conversation so it need not recompute history. Grows with context and users; shares memory with the weights.

Memory bandwidth

How fast weights move from memory into the cores. The limit for CPU inference and for the decode phase in general.

AMX

Intel’s on-CPU matrix-math accelerator. Makes CPU inference fast enough that memory bandwidth, not math, is the limit.

Prefill / decode

Prefill reads the whole prompt in parallel (fast). Decode writes the answer one token at a time (slow); this is what tokens per second measures.

Latency vs throughput

Latency is one response for one user. Throughput is total tokens per second across all users. Sizing usually turns on throughput.

Batching

Serving many requests at once. Raises throughput cheaply, but each user needs their own KV cache.

Interactive vs batch

Interactive means a person waits on the answer live. Batch means the work runs in the background where minutes or hours are fine.

Single-pass vs token-by-token

Some jobs (search, classification) read once and emit a result. Others (chat, writing) generate one token at a time, which is slower.

GPU

A chip with thousands of parallel cores, ideal for the math models run on. Fast, but with limited on-board memory.

CPU

The general-purpose processor in every computer. Fewer, more flexible cores, paired with large, inexpensive system memory.

GPU / CPU box

The GPU boxes (H200, RP6000) excel at interactive generation. The CPU box (XL v5) excels at batch and single-pass work with abundant memory.

RAG

Retrieval-Augmented Generation: retrieve relevant passages from your own data at question time and let the model answer from them. Grounds answers and reduces made-up facts; no training.

Embedding

A numeric vector that captures the meaning of a piece of text, used for semantic search and recommendations. Produced in a single fast pass.

Agent / tool use

A model that calls tools or APIs in a loop to complete a multi-step task, rather than just answering once.

Vision-language (VL)

A model that accepts images alongside text, for reading documents, describing pictures, or answering questions about them.

ASR

Automatic Speech Recognition: converting audio into text (transcription).

Diffusion

The model type behind image generators. Produces a picture rather than text.

Fine-tuning

Continuing to train an existing open-weight model on your own data to change how it behaves. The realistic form of training for most companies.

Full fine-tuning

Updating every weight. Highest quality ceiling, highest memory cost; single-box only for smaller models.

LoRA

Freezing the model and training small add-on adapter weights. Fits much larger models because you only store and update the adapters.

QLoRA

LoRA with the frozen base compressed to 4-bit. Lets very large models fine-tune on a single card. The common default.

Pre-training

Building a model from scratch on huge data. Needs a large cluster; out of scope for a single box.

Continued pre-training

Further training a base model on a large proprietary corpus. The heaviest project; a larger deployment scoped with OpenMetal.

Distillation

Training a small model to imitate a larger one, so everyday serving runs on cheaper, faster hardware.

Guardrail tuning

Teaching a model your refusal and escalation rules, often paired with a small classifier in front.

Gradient

For each weight being trained, the direction to nudge it. Costs one extra number per updated weight.

Optimizer state

The training process’s running memory of each weight, for stable updates. The main reason full fine-tuning needs so much memory.

Activations

Intermediate results kept during a step so the update can be computed. Grow with batch size and sequence length.

Epoch / step

An epoch is one pass over your dataset; a step is one batch. Runs use several epochs.

Checkpoint

A saved snapshot of the model mid-run, so you can resume or compare.

BF16 / mixed precision

16-bit math used for training, with a 32-bit master copy for stability. Training rarely uses 4-bit except a QLoRA frozen base.

Offload

Spilling part of the training state to system memory to fit a slightly oversized job on one card, at the cost of speed.

Gradient checkpointing

Recomputing some activations instead of storing them, trading a little compute to cut memory.

RAG vs fine-tuning

RAG adds knowledge at question time without training. Fine-tuning changes behavior by training. Many projects use both.