Templates
Your stack, running before you log in.
Pick an operating system and a template on the order form. Drivers, toolkits and the service are installed before first boot; serving templates pull the model you name. Root over SSH from the first minute, and nothing you cannot replace.
- 14 templates, 5 operating systems
- Model pre-loaded on serving templates
- systemd services, root everywhere
Operating systems
Five systems, driver included.
| System | Notes | Cards | Licence | Supported until |
|---|---|---|---|---|
| Ubuntu 24.04 LTS | Kernel 6.8 · the default for every template | NVIDIA, AMD · sxm, pcie, consumer | included | April 2029 |
| Ubuntu 22.04 LTS | Kernel 5.15 HWE · for stacks pinned to CUDA 11.8 / 12.1 | NVIDIA, AMD · sxm, pcie, consumer | included | April 2027 |
| Debian 12 | Bookworm · minimal, driver from the vendor repository | NVIDIA, AMD · sxm, pcie, consumer | included | June 2028 |
| Rocky Linux 9 | RHEL-compatible · Slurm, enterprise and HPC stacks | NVIDIA, AMD · sxm, pcie, consumer | included | May 2032 |
| Windows Server 2022 | Standard · RDP on :3389 · licence billed at our rate | NVIDIA · pcie, consumer | $28/mo | October 2031 |
NVIDIA driver 570 with CUDA 12.8 on Linux, ROCm 6.4 on MI300X, Studio or data-centre driver on Windows. Drivers and pinned toolkits.
Base system
The driver and nothing else.
-
Bare OS + driver
NVIDIA driver 570 with the CUDA 12.8 toolkit, or ROCm 6.4 on AMD. Nothing else running.
- Listens on
- nothing
- Runs on
- NVIDIA · AMD · Linux · Windows
- First check
nvidia-smi
Frameworks
Frameworks with the toolchain in place.
-
PyTorch 2.7
CUDA 12.8 or ROCm 6.4 build, cuDNN, torchvision, torchaudio, Jupyter Lab on :8888.
- Listens on
- Jupyter Lab :8888
- Runs on
- NVIDIA · AMD · Linux
- First check
python3 -c 'import torch; print(torch.cuda.device_count())'
-
TensorFlow 2.18 + JAX 0.5
CUDA 12.8, XLA, Keras 3, Jupyter Lab on :8888.
- Listens on
- Jupyter Lab :8888
- Runs on
- NVIDIA · Linux
- First check
python3 -c 'import jax; print(jax.devices())'
-
Jupyter Lab
Lab on :8888 with PyTorch, JAX and TensorFlow kernels side by side.
- Listens on
- Jupyter Lab :8888
- Runs on
- NVIDIA · Linux
- First check
jupyter server list
LLM serving
An API on a port, model already pulled.
-
vLLM 0.9 + model
OpenAI-compatible server on :8000, tensor-parallel across every card, model pulled before first boot.
- Listens on
- OpenAI-compatible API :8000
- Runs on
- NVIDIA · AMD · Linux
- First check
curl localhost:8000/v1/models
-
SGLang 0.4 + model
Structured generation with RadixAttention on :30000, model pulled before first boot.
- Listens on
- SGLang API :30000
- Runs on
- NVIDIA · Linux
- First check
curl localhost:30000/health
-
Text Generation Inference 3 + model
Hugging Face TGI on :8080 with the Messages API, model pulled before first boot.
- Listens on
- TGI API :8080
- Runs on
- NVIDIA · AMD · Linux
- First check
curl localhost:8080/info
-
Ollama + Open WebUI + model
Ollama on :11434 and a chat UI on :3000. Name an Ollama tag such as llama3.1:70b and it is pulled first.
- Listens on
- Ollama API :11434, Open WebUI :3000
- Runs on
- NVIDIA · AMD · Linux
- First check
ollama list
Diffusion and rendering
Diffusion, video and rendering.
-
ComfyUI
ComfyUI on :8188 with ComfyUI-Manager, Flux and SDXL loaders. No checkpoints bundled — bring yours.
- Listens on
- ComfyUI :8188
- Runs on
- NVIDIA · Linux · Windows
- First check
systemctl status comfyui
-
Stable Diffusion WebUI Forge
Forge on :7860 with ControlNet and the usual extensions.
- Listens on
- Forge :7860
- Runs on
- NVIDIA · Linux · Windows
- First check
systemctl status forge
-
Render node
Blender 4.4 headless with Cycles and OptiX, ready for your Octane, Redshift or V-Ray licence.
- Listens on
- nothing
- Runs on
- NVIDIA · Linux · Windows
- First check
blender -b --version
Orchestration
Containers, Kubernetes, Slurm.
-
Docker + GPU toolkit
Docker 27 with the compose plugin and the NVIDIA Container Toolkit, or the ROCm runtime on AMD.
- Listens on
- nothing
- Runs on
- NVIDIA · AMD · Linux
- First check
docker run --rm --gpus all nvidia/cuda:12.8.0-base-ubuntu24.04 nvidia-smi
-
Kubernetes node (k3s)
Single-node k3s with the NVIDIA GPU Operator. Join it to your cluster or run it alone.
- Listens on
- Kubernetes API :6443
- Runs on
- NVIDIA · Linux
- First check
kubectl get nodes -o wide
-
Slurm worker
slurmd 24.11 with Pyxis and Enroot; give us your controller address after boot and it joins.
- Listens on
- nothing
- Runs on
- NVIDIA · Linux
- First check
sinfo
Pre-loaded model
Name it on the order form. It is there at first boot.
vLLM, SGLang, TGI and Ollama take a model and a precision when you order. The weights are pulled while the server is provisioned, and the order form checks the fit — parameters × bytes per parameter × 1.2 must be at most the server's memory — so a 70B model at BF16 is refused on a single 80 GB card rather than failing at boot.
| Precision | When | Memory |
|---|---|---|
| BF16 / FP16 | full quality · 2 bytes per parameter | 2 B/param |
| FP8 | near-lossless · 1 byte · Ada, Hopper, Blackwell, MI300X | 1 B/param |
| INT8 | 1 byte per parameter · any card | 1 B/param |
| INT4 (AWQ / GPTQ) | 0.5 byte per parameter · smallest footprint | 0.5 B/param |
Under the hood
A service you own.
systemctl status vllm # or comfyui, jupyter, ollama, tgi, sglang, forge
journalctl -u vllm -f # logs
nano /opt/dedigpu/vllm/env # MODEL, DTYPE, MAX_MODEL_LEN, API_KEY…
systemctl restart vllmWeights cache under /opt/dedigpu/models on the local NVMe. Change the model in the env file and restart; the docs cover gated repositories and quantised checkpoints.
Questions
About templates.
Can I change or remove a template after boot?
Yes. Each template is a systemd service under /opt/dedigpu/<template> with its arguments in an env file. Stop it, edit it, replace it, or re-image the server with another template from the console.
Which templates accept a pre-loaded model?
vLLM, SGLang, TGI and Ollama. You name a Hugging Face repository or an Ollama tag and a precision on the order form; the weights are pulled before first boot and the order form refuses a model that would not fit in memory.
Are the ports open to the Internet?
Yes, there is no firewall in front of the server. Jupyter and ComfyUI start with a token or password shown on the server page; vLLM, SGLang, TGI and Ollama start without authentication. Reach them through an SSH tunnel or put a key in front before you share the address.
Which templates run on AMD MI300X?
Bare ROCm, PyTorch, vLLM, TGI, Ollama and Docker with the ROCm runtime. TensorFlow, SGLang, the diffusion templates, k3s and Slurm are NVIDIA only for now.
Which templates run on Windows Server?
Bare driver, ComfyUI, Stable Diffusion WebUI Forge and the render node, on PCIe and workstation cards. Frameworks and serving templates are Linux only.
Pick a card, pick a template, name the model.
The server comes up with the API answering. Billed at cost, one term at a time.