The answer is set by three things: the model’s weight footprint, the KV cache your context and concurrency produce, and how your serving framework spreads both across GPUs. An eight-GPU
Tag: NVIDIA HGX B300
The NVIDIA HGX B300 is NVIDIA’s Blackwell Ultra platform in its densest single-node form: eight B300 SXM GPUs wired together on one baseboard by fifth-generation NVLink and NVSwitch into a