*[DediGPU](https://dedigpu.com/) — Markdown mirror of [https://dedigpu.com/guides/qlora-fine-tune-on-one-rtx-4090](https://dedigpu.com/guides/qlora-fine-tune-on-one-rtx-4090) · updated 2026-09-02 · index for LLMs: [llms.txt](https://dedigpu.com/llms.txt) · everything: [llms-full.txt](https://dedigpu.com/llms-full.txt)*

# Fine-tune an 8B model with QLoRA on a single RTX 4090

> Everything that fits in 24 GB: 4-bit base weights, LoRA adapters, a real dataset, the training script, memory numbers, hours, and what it costs on a server billed at cost.

*Fine-tuning · published 30 July 2026 · updated 2 September 2026 · by DediGPU engineering*

In short:

- An 8B model in 4-bit plus LoRA adapters trains in 18 to 21 GB: one RTX 4090 at $139 a month.
- About two hours per epoch on 20,000 examples of a thousand tokens; a run costs $0.39 of rent.
- Merge the adapter and serve the result with vLLM on the same card, or convert it to GGUF for Ollama.

QLoRA made fine-tuning an 8B model a one-card job: the base weights sit in 4-bit, the adapters train in BF16, and the whole thing fits in 24 GB with a useful batch. Here is the recipe on a rented RTX 4090, from order to a merged model you can serve.

## Why it fits

Llama 3.1 8B in 4-bit NF4 is about 5.5 GB. LoRA adapters on every linear layer at rank 16 add under 200 MB, and their optimiser state a few hundred more. What remains, roughly 16 GB, is activations: with gradient checkpointing, a sequence length of 2048 and a micro-batch of 4 you stay under 20 GB. Full fine-tuning of the same model needs 16 bytes per parameter for weights, gradients and Adam state, 128 GB, a very different server.

## 1. Order

An [RTX 4090](https://dedigpu.com/gpu/rtx-4090) with the **PyTorch 2.7** template, Ubuntu 24.04, $139 for the month. The template has CUDA 12.8, torch, and Jupyter on port 8888 if you prefer a notebook.

## 2. Install the training stack

```
ssh root@203.0.113.10
pip install -U transformers peft trl bitsandbytes datasets accelerate
huggingface-cli login      # for gated Llama weights
```

## 3. A dataset

Any instruction dataset in chat format works; the example uses a public one. Replace it with your own JSONL of `{\"messages\": [...]}` rows. Quality beats size: two thousand clean examples do more than fifty thousand scraped ones.

## 4. The script

```
# train.py
from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig
from trl import SFTTrainer, SFTConfig
import torch

base = "meta-llama/Llama-3.1-8B-Instruct"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(
    base, device_map="auto",
    quantization_config=BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_use_double_quant=True))

ds = load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft[:20000]")

cfg = SFTConfig(output_dir="out", num_train_epochs=1, per_device_train_batch_size=4,
    gradient_accumulation_steps=4, learning_rate=2e-4, bf16=True, logging_steps=20,
    max_length=2048, gradient_checkpointing=True, save_steps=500, lr_scheduler_type="cosine")
lora = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, target_modules="all-linear", task_type="CAUSAL_LM")

SFTTrainer(model=model, args=cfg, train_dataset=ds, peft_config=lora, processing_class=tok).train()
```

```
python train.py 2>&1 | tee train.log
nvidia-smi --query-gpu=memory.used --format=csv -l 30   # in another shell
```

## 5. What to expect

- **Memory:** 18 to 21 GB at the settings above. If you see an out-of-memory error, drop the micro-batch to 2 and double the accumulation.
- **Speed:** About 2,500 to 3,000 tokens per second on a 4090 with these settings: 20,000 examples of ~1,000 tokens is roughly two hours per epoch.
- **Cost:** At $139 a month the server costs $4.63 a day. A two-hour run is $0.39 of rent. The term is what you pay for; the runs are free after that.

## 6. Merge and serve

```
python - <<'EOF'
from peft import AutoPeftModelForCausalLM
m = AutoPeftModelForCausalLM.from_pretrained("out/checkpoint-1250", torch_dtype="bfloat16")
m.merge_and_unload().save_pretrained("merged", safe_serialization=True)
EOF
pip install vllm && vllm serve ./merged --dtype bfloat16 --max-model-len 8192
```

The merged 8B model in BF16 is 16 GB and serves on the same card at FP8 or BF16. If the goal is an Ollama model, convert with `llama.cpp`'s `convert_hf_to_gguf.py` and quantise to Q4_K_M.

## Going bigger

A 5090 (32 GB, $163) trains the same recipe with sequence length 4096 and faster. For 13B to 32B models in QLoRA, an RTX A6000 (48 GB, $208) is the cheapest 48 GB card on the catalogue; for 70B in QLoRA you need 48 GB minimum and 80 GB comfortably, which is an [A100 80 GB](https://dedigpu.com/gpu/a100-80) at $654.