Fine-tune an 8B model with QLoRA on a single RTX 4090

Everything that fits in 24 GB: 4-bit base weights, LoRA adapters, a real dataset, the training script, memory numbers, hours, and what it costs on a server billed at cost.

DediGPU engineering Published 30 July 2026 Updated 2 September 2026 Prices checked 2 September 2026

QLoRA made fine-tuning an 8B model a one-card job: the base weights sit in 4-bit, the adapters train in BF16, and the whole thing fits in 24 GB with a useful batch. Here is the recipe on a rented RTX 4090, from order to a merged model you can serve.

Why it fits

Llama 3.1 8B in 4-bit NF4 is about 5.5 GB. LoRA adapters on every linear layer at rank 16 add under 200 MB, and their optimiser state a few hundred more. What remains, roughly 16 GB, is activations: with gradient checkpointing, a sequence length of 2048 and a micro-batch of 4 you stay under 20 GB. Full fine-tuning of the same model needs 16 bytes per parameter for weights, gradients and Adam state, 128 GB, a very different server.

1. Order

An RTX 4090 with the PyTorch 2.7 template, Ubuntu 24.04, $139 for the month. The template has CUDA 12.8, torch, and Jupyter on port 8888 if you prefer a notebook.

2. Install the training stack

ssh [email protected]
pip install -U transformers peft trl bitsandbytes datasets accelerate
huggingface-cli login      # for gated Llama weights

3. A dataset

Any instruction dataset in chat format works; the example uses a public one. Replace it with your own JSONL of {\"messages\": [...]} rows. Quality beats size: two thousand clean examples do more than fifty thousand scraped ones.

4. The script

# train.py
from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig
from trl import SFTTrainer, SFTConfig
import torch

base = "meta-llama/Llama-3.1-8B-Instruct"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(
    base, device_map="auto",
    quantization_config=BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_use_double_quant=True))

ds = load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft[:20000]")

cfg = SFTConfig(output_dir="out", num_train_epochs=1, per_device_train_batch_size=4,
    gradient_accumulation_steps=4, learning_rate=2e-4, bf16=True, logging_steps=20,
    max_length=2048, gradient_checkpointing=True, save_steps=500, lr_scheduler_type="cosine")
lora = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, target_modules="all-linear", task_type="CAUSAL_LM")

SFTTrainer(model=model, args=cfg, train_dataset=ds, peft_config=lora, processing_class=tok).train()

python train.py 2>&1 | tee train.log
nvidia-smi --query-gpu=memory.used --format=csv -l 30   # in another shell

5. What to expect

Memory: 18 to 21 GB at the settings above. If you see an out-of-memory error, drop the micro-batch to 2 and double the accumulation.
Speed: About 2,500 to 3,000 tokens per second on a 4090 with these settings: 20,000 examples of ~1,000 tokens is roughly two hours per epoch.
Cost: At $139 a month the server costs $4.63 a day. A two-hour run is $0.39 of rent. The term is what you pay for; the runs are free after that.

6. Merge and serve

python - <<'EOF'
from peft import AutoPeftModelForCausalLM
m = AutoPeftModelForCausalLM.from_pretrained("out/checkpoint-1250", torch_dtype="bfloat16")
m.merge_and_unload().save_pretrained("merged", safe_serialization=True)
EOF
pip install vllm && vllm serve ./merged --dtype bfloat16 --max-model-len 8192

The merged 8B model in BF16 is 16 GB and serves on the same card at FP8 or BF16. If the goal is an Ollama model, convert with llama.cpp's convert_hf_to_gguf.py and quantise to Q4_K_M.

Going bigger

A 5090 (32 GB, $163) trains the same recipe with sequence length 4096 and faster. For 13B to 32B models in QLoRA, an RTX A6000 (48 GB, $208) is the cheapest 48 GB card on the catalogue; for 70B in QLoRA you need 48 GB minimum and 80 GB comfortably, which is an A100 80 GB at $654.