Software
Pre-loading a model
How serving templates pull a model before first boot: Hugging Face and Ollama names, precision, the memory-fit rule the order form enforces, changing the model later.
With vLLM, SGLang, TGI or Ollama, the order form asks for a model and a precision. The weights are pulled while the server is provisioned, so the API answers at first boot instead of an hour later.
Naming the model
- Hugging Face repositories, as
owner/name:meta-llama/Llama-3.3-70B-Instruct,Qwen/Qwen2.5-72B-Instruct. Gated repositories need your own token: give it after boot and restart the service. - Ollama tags for the Ollama template:
llama3.1:70b,qwen2.5:72b-instruct-q4_K_M.
Precision
| Precision | When | Memory |
|---|---|---|
| BF16 / FP16 | full quality · 2 bytes per parameter | 2 B/param |
| FP8 | near-lossless · 1 byte · Ada, Hopper, Blackwell, MI300X | 1 B/param |
| INT8 | 1 byte per parameter · any card | 1 B/param |
| INT4 (AWQ / GPTQ) | 0.5 byte per parameter · smallest footprint | 0.5 B/param |
FP8 needs Ada, Hopper, Blackwell or MI300X silicon; the form refuses it on Ampere cards. INT8 and INT4 use quantised checkpoints (AWQ, GPTQ, GGUF); name one that exists for the model you want.
The fit rule
The form reads the parameter count from the name (70B, 8x22B, :72b) and checks that parameters × bytes per parameter × 1.2 is at most the server's total GPU memory. The 20 % covers the KV cache and runtime buffers at a normal context length. Example: 70 B parameters at BF16 / FP16 need about 168 GB; this server has 160 GB. Pick a smaller precision, more cards or a bigger card. At FP8: 70 B parameters at FP8 need about 84 GB with a working KV cache; this server has 160 GB.
A name without a parameter count is accepted without a check; make sure it fits. The VRAM guide has the full table.
Changing the model later
nano /opt/dedigpu/vllm/env # MODEL=..., DTYPE=..., MAX_MODEL_LEN=...
systemctl restart vllm
journalctl -u vllm -f # watch the new weights loadWeights are cached under /opt/dedigpu/models on the local NVMe; a 70B model in BF16 is about 140 GB, so keep an eye on df -h on the smaller cards.