# Train Your Own SDXL LoRA on a Single GPU with kohya-ss

> Source: <https://sourcefeed.dev/a/train-your-own-sdxl-lora-on-a-single-gpu-with-kohya-ss>
> Published: 2026-08-24 17:43:19+00:00

# Train Your Own SDXL LoRA on a Single GPU with kohya-ss

Fine-tune a product LoRA from 20 photos with sd-scripts, then load it in diffusers for consistent on-brand images.

[Priya Nair](https://sourcefeed.dev/u/priya_nair)

## What you'll build

You'll train a ~45 MB SDXL LoRA from 15–30 photos of a product using [kohya-ss/sd-scripts](https://github.com/kohya-ss/sd-scripts) (the trainer behind the [kohya_ss](https://github.com/bmaltais/kohya_ss) GUI), then load it into a [diffusers](https://huggingface.co/docs/diffusers) pipeline and generate on-brand images with a trigger word.

## Prerequisites

- Linux (Ubuntu 22.04/24.04) or WSL2, NVIDIA GPU with 12 GB+ VRAM. The docs say 8 GB works with
`--network_dim`

4–8; this guide assumes 12 GB. - Python 3.10.x and git. sd-scripts lists 3.11/3.12 as "will work but not tested."
- Verified against: sd-scripts
`main`

(v0.11.1, July 2026), PyTorch 2.6.0+cu124 (RTX 50-series needs 2.8.0+cu128), diffusers 0.32.1 and accelerate 1.6.0 (both pinned by sd-scripts'`requirements.txt`

), huggingface_hub 0.34.3. - About 15 GB free disk: 7 GB for the SDXL base checkpoint plus latent caches.
- 15–30 JPG/PNG photos of one subject, ideally 1024 px or larger on the short side, with varied angles, backgrounds and lighting.

## 1. Install sd-scripts

```
git clone https://github.com/kohya-ss/sd-scripts.git
cd sd-scripts
python3.10 -m venv venv
source venv/bin/activate
pip install torch==2.6.0 torchvision==0.21.0 --index-url https://download.pytorch.org/whl/cu124
pip install --upgrade -r requirements.txt
accelerate config default --mixed_precision bf16
```

On an RTX 50-series card replace the torch line with `pip install torch==2.8.0 torchvision==0.23.0 --index-url https://download.pytorch.org/whl/cu128`

. `accelerate config default`

writes a single-GPU config without the interactive questionnaire.

Download the single-file SDXL base checkpoint (6.94 GB) rather than the diffusers repo, which pulls fp32 weights twice that size:

```
mkdir -p ~/lora/models ~/lora/data/product ~/lora/output
hf download stabilityai/stable-diffusion-xl-base-1.0 sd_xl_base_1.0.safetensors --local-dir ~/lora/models
```

## 2. Prepare the dataset

Copy your photos into `~/lora/data/product`

, then create one caption `.txt`

per image with the same basename. Start every caption with a rare trigger token (here `ohwx`

) followed by the class, then describe what *varies* between shots — background, angle, lighting — so the LoRA learns the product and not the kitchen counter:

```
cd ~/lora/data/product
for f in *.jpg; do echo "ohwx bottle, product photo" > "${f%.jpg}.txt"; done
```

Now edit each file, e.g. `IMG_0412.txt`

:

```
ohwx bottle, product photo, on a wooden desk, window light from the left, slight top-down angle
```

Images without a caption file fall back to `class_tokens`

from the config below, so a missing file won't crash training — it just trains on a weaker caption.

## 3. Write the dataset config

Save as `~/lora/dataset.toml`

(replace `/home/you`

; TOML doesn't expand `~`

):

```
[general]
caption_extension = ".txt"
keep_tokens = 1

[[datasets]]
resolution = 1024
batch_size = 1
enable_bucket = true
min_bucket_reso = 640
max_bucket_reso = 1536
bucket_reso_steps = 64

  [[datasets.subsets]]
  image_dir = "/home/you/lora/data/product"
  class_tokens = "ohwx bottle"
  num_repeats = 10
```

`enable_bucket`

groups images by aspect ratio so nothing gets center-cropped; the min/max values must be divisible by `bucket_reso_steps`

. With 20 images × 10 repeats you get 200 steps per epoch.

Add a sample prompt file so you can watch the LoRA converge. Save as `~/lora/sample_prompts.txt`

:

```
ohwx bottle on a marble kitchen counter, soft morning light --n blurry, lowres, watermark --w 1024 --h 1024 --d 1 --s 28 --l 7
```

`--n`

is the negative prompt, `--d`

the seed, `--s`

steps, `--l`

CFG scale.

## 4. Train

```
cd ~/sd-scripts && source venv/bin/activate
accelerate launch --num_cpu_threads_per_process 1 sdxl_train_network.py \
  --pretrained_model_name_or_path ~/lora/models/sd_xl_base_1.0.safetensors \
  --dataset_config ~/lora/dataset.toml \
  --output_dir ~/lora/output --output_name product-lora \
  --save_model_as safetensors --save_precision bf16 \
  --network_module networks.lora --network_dim 16 --network_alpha 8 \
  --network_train_unet_only \
  --learning_rate 1e-4 --optimizer_type AdamW8bit \
  --lr_scheduler cosine --lr_warmup_steps 80 \
  --max_train_epochs 8 --save_every_n_epochs 2 \
  --mixed_precision bf16 --gradient_checkpointing --sdpa \
  --cache_latents --cache_latents_to_disk \
  --cache_text_encoder_outputs --cache_text_encoder_outputs_to_disk \
  --no_half_vae --noise_offset 0.0357 --min_snr_gamma 5 \
  --sample_prompts ~/lora/sample_prompts.txt --sample_every_n_epochs 2 --sample_sampler euler_a \
  --seed 42 --logging_dir ~/lora/logs
```

Why these flags:

`--network_train_unet_only`

is mandatory once you cache text-encoder outputs (the script asserts it), and the SDXL docs recommend it anyway because training both text encoders gives unpredictable results.`--cache_latents`

+`--cache_text_encoder_outputs`

skip the VAE and CLIP passes every step; combined with`--gradient_checkpointing`

,`--sdpa`

and`AdamW8bit`

this is what fits 1024 px training in 12 GB.`--no_half_vae`

keeps the VAE in fp32 during latent caching; it's insurance against the SDXL fp16 VAE producing NaNs.`--noise_offset 0.0357`

matches what SDXL base was trained with;`--min_snr_gamma 5`

stabilizes early loss.- 1600 total steps at 1e-4 with cosine decay is a reasonable starting point for a single object; a 4090 finishes in roughly 20–25 minutes, a 3060 in about an hour.

The first run encodes latents and text embeddings to disk (`.npz`

files next to your images) before the step counter starts.

## 5. Load the LoRA in diffusers

diffusers needs [PEFT](https://huggingface.co/docs/peft) to load LoRA weights, which sd-scripts doesn't install:

```
pip install peft
```

Save as `~/lora/generate.py`

:

``` python
import os
import torch
from diffusers import StableDiffusionXLPipeline

home = os.path.expanduser("~/lora")

pipe = StableDiffusionXLPipeline.from_single_file(
    f"{home}/models/sd_xl_base_1.0.safetensors", torch_dtype=torch.float16
).to("cuda")

pipe.load_lora_weights(f"{home}/output", weight_name="product-lora.safetensors")

image = pipe(
    "ohwx bottle on a matte black coffee table, studio lighting, editorial product shot",
    negative_prompt="blurry, lowres, watermark, text",
    num_inference_steps=30,
    guidance_scale=7.0,
    cross_attention_kwargs={"scale": 0.8},
    generator=torch.Generator("cuda").manual_seed(1),
).images[0]

image.save(f"{home}/product-table.png")
print("saved", f"{home}/product-table.png")
```

`from_single_file`

reuses the checkpoint you already downloaded. `cross_attention_kwargs={"scale": 0.8}`

dials LoRA strength between 0 (base model) and 1 (full LoRA); 0.7–0.9 usually keeps the subject faithful without cooking the composition.

## Verify it works

Training should end with the step bar at 100% and two log lines:

```
steps: 100%|██████████| 1600/1600 [23:41<00:00,  1.13it/s, avr_loss=0.0862]

saving checkpoint: /home/you/lora/output/product-lora.safetensors
model saved.
```

`avr_loss`

for SDXL LoRA typically settles somewhere around 0.08–0.12; a value that keeps climbing or hits `nan`

means the learning rate is too high.

Check the outputs:

```
ls -la ~/lora/output ~/lora/output/sample
```

You should see `product-lora.safetensors`

(about 45 MB for dim 16, U-Net only), epoch checkpoints named `product-lora-000002.safetensors`

, `-000004`

, `-000006`

, and PNGs in `sample/`

for epochs 2, 4, 6 and 8. Flip through the samples: by epoch 4 the bottle's shape and label should be recognizable, and by epoch 8 it should be consistent across seeds. If epoch 6 looks better than 8 (over-baked, blown-out contrast), use that checkpoint instead.

Then generate:

```
python ~/lora/generate.py
saved /home/you/lora/product-table.png
```

Run it again with `cross_attention_kwargs={"scale": 0.0}`

— the bottle should disappear into a generic one. That difference is your LoRA working.

## Troubleshooting

** AssertionError: network for Text Encoder cannot be trained with caching Text Encoder outputs** — you passed

`--cache_text_encoder_outputs`

without `--network_train_unet_only`

. Add the flag, or drop the caching flags and add `--text_encoder_lr1 1e-5 --text_encoder_lr2 1e-5`

if you actually want to train the text encoders.** No data found. Please verify arguments (train_data_dir must be the parent of folders with images)** then the script exits —

`image_dir`

in `dataset.toml`

is wrong or contains no images. For `--dataset_config`

the path must point directly at the folder holding the images (no `10_bottle`

subfolder convention), and it must be absolute.** RuntimeError: NaN detected in latents: /home/you/lora/data/product/IMG_0412.jpg** — the VAE overflowed in half precision while caching. Make sure

`--no_half_vae`

is on the command line; if you're on a GPU without bf16 (GTX 16xx/RTX 20xx) and used `--mixed_precision fp16`

, that flag is the fix. If a single file is still named, the image itself is corrupt — re-export it.** torch.OutOfMemoryError: CUDA out of memory. Tried to allocate ...** — first drop

`--sample_prompts`

(sampling loads a full inference pipeline mid-training), then set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`

in front of the launch command to reduce fragmentation, then lower `--network_dim`

to 8 and `resolution`

to 768 in the TOML.** ValueError: PEFT backend is required for this method.** from

`generate.py`

— `peft`

isn't installed in the venv you're running from. `pip install peft`

and rerun.## Next steps

- Train the text encoders too for a stronger trigger word: remove both
`--cache_text_encoder_outputs*`

flags and`--network_train_unet_only`

, add`--text_encoder_lr1 1e-5 --text_encoder_lr2 1e-5`

, and expect ~2 GB more VRAM. - Auto-caption larger sets with
`finetune/tag_images_by_wd14_tagger.py --onnx --repo_id SmilingWolf/wd-eva02-large-tagger-v3 --remove_underscore`

(needs`pip install onnx onnxruntime-gpu`

). - Try
`--optimizer_type Prodigy --learning_rate 1.0`

to skip learning-rate tuning, or LoHa/LoKr via`docs/loha_lokr.md`

for style LoRAs. - Prefer clicking?
`git clone --recursive https://github.com/bmaltais/kohya_ss && ./setup.sh && ./gui.sh`

gives you the same trainer behind a Gradio UI, and it prints the equivalent CLI command it runs. - The
`.safetensors`

file drops straight into ComfyUI's`models/loras`

or A1111's`models/Lora`

with`<lora:product-lora:0.8>`

in the prompt.

## Sources & further reading

-
[kohya-ss/sd-scripts README (installation, SDXL training notes)](https://github.com/kohya-ss/sd-scripts)— github.com -
[How to Use the SDXL LoRA Training Script sdxl_train_network.py](https://github.com/kohya-ss/sd-scripts/blob/main/docs/sdxl_train_network.md)— github.com -
[sd-scripts Dataset Configuration Guide](https://github.com/kohya-ss/sd-scripts/blob/main/docs/config_README-en.md)— github.com -
[sd-scripts SDXL training recommendations](https://github.com/kohya-ss/sd-scripts/blob/main/docs/train_SDXL-en.md)— github.com -
[Diffusers v0.32.1 - Load adapters (LoRA)](https://huggingface.co/docs/diffusers/v0.32.1/en/using-diffusers/loading_adapters)— huggingface.co -
[Accelerate CLI reference (config default, launch)](https://huggingface.co/docs/accelerate/package_reference/cli)— huggingface.co

[Priya Nair](https://sourcefeed.dev/u/priya_nair)· AI & Developer Experience Writer

Priya covers AI frameworks, developer productivity tooling, and the startup ecosystem across South and Southeast Asia, bringing a researcher's rigour and a practitioner's empathy to every story. She is deeply sceptical of benchmarks and asks hard questions so her readers don't have to.

## Discussion 1

curious how this handles versioning drift - have you pinned everything or does it still work with latest diffusers