Inference · 10 min read
Serve Llama 3.3 70B with vLLM on a dedicated GPU server, step by step
Pick the server that fits, pre-load the model at order time, check the OpenAI-compatible API, tune context and parallelism, and put a token in front of it.
In short
- Llama 3.3 70B at FP8 needs 84 GB: one H200, one MI300X, or two H100 SXM with tensor parallelism.
- Order with the vLLM template and the model name; the weights are pulled before first boot and the API answers on port 8000.
- Put an API key and a firewall in front before you share the address: the port is public and vLLM starts without authentication.
A 70B instruction model behind an OpenAI-compatible endpoint is the most common thing people build on a rented GPU server. With a serving template the model is pulled before first boot, so the work left is choosing the server, checking the API and hardening it. Forty minutes, most of it waiting for weights.
1. Pick a server that fits
Llama 3.3 70B needs, with the fit rule (parameters × bytes × 1.2):
- BF16, 168 GB
- 1× MI300X $782/mo
- FP8, 84 GB
- 1× RTX PRO 6000 $493/mo
- INT4 (AWQ), 42 GB
- 1× RTX A6000 $208/mo
For production, FP8 on a single H200 (141 GB, $1,252) or a single MI300X (192 GB, $782) leaves the most room for the KV cache, which is what throughput is made of. Two H100 SXM ($2,268) give the same memory with NVLink tensor parallelism and higher aggregate bandwidth.
2. Order with the model pre-loaded
On the order form: the card and count, a region near your users, Ubuntu 24.04, the vLLM template, model meta-llama/Llama-3.3-70B-Instruct, precision FP8. The form checks the fit and shows the monthly price with the term discount. Llama weights are gated on Hugging Face: order with the model anyway, then give the service your token at first login (below); the download resumes.
3. Check the API
ssh [email protected]
systemctl status vllm
journalctl -u vllm -f # weights loading, then "Uvicorn running on 0.0.0.0:8000"
curl localhost:8000/v1/modelscurl localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
"model": "meta-llama/Llama-3.3-70B-Instruct",
"messages": [{"role": "user", "content": "Two sentences on why NVLink matters."}],
"max_tokens": 120
}'4. Gated weights and the Hugging Face token
nano /opt/dedigpu/vllm/env
# HF_TOKEN=hf_...
systemctl restart vllm5. Tune it
The service reads its arguments from /opt/dedigpu/vllm/env. The ones that matter:
MAX_MODEL_LEN=32768- The longest context you will accept. Lower it to leave more cache for concurrency; 8192 is plenty for chat.
GPU_MEMORY_UTILIZATION=0.92- How much of the card vLLM may take. 0.90 to 0.95 on a dedicated server; there is nothing else running.
TENSOR_PARALLEL_SIZE=2- Set to the number of cards. On multi-card servers vLLM splits every layer across NVLink.
QUANTIZATION=fp8- Already set by the precision you chose. On H100/H200 the weights are quantised at load; on B200 use an FP8 or NVFP4 checkpoint.
MAX_NUM_SEQS=256- Concurrent sequences. Raise it on the H200; watch the cache usage in the logs.
6. Put a token in front of it
vLLM starts with no authentication and the port is public. Two lines fix it:
# in /opt/dedigpu/vllm/env
API_KEY=$(openssl rand -hex 24)
systemctl restart vllm
# clients send: Authorization: Bearer <that key>For TLS, put Caddy in front (apt install caddy, one line of config per hostname) or keep the port closed with ufw and reach it through a VPN or an SSH tunnel.
7. What to expect
On one H200 at FP8 with 8k context, a 70B model serves on the order of 1,500 to 2,500 output tokens per second aggregate at 64 concurrent requests, and 30 to 40 tokens per second for a single stream. Two H100 with tensor parallelism land in the same range with a little more single-stream speed. Benchmark your own prompts with vllm bench serve, shipped with the template.
8. The bill
One H200 at $1,252 a month, $1,126.80 on a 12-month term, egress free. At 1.5 billion output tokens a month that is about $0.9 per million tokens, before you count the cache hits.
Keep reading.
