Train Your Own SDXL LoRA on a Single GPU with kohya-ss Kohya-ss/sd-scripts enables training a ~45 MB SDXL LoRA from 15–30 product photos on a single 12 GB GPU, with a step-by-step guide covering installation, dataset preparation, and training. The workflow uses a trigger word and diffusers for consistent on-brand image generation, verified with sd-scripts v0.11.1, PyTorch 2.6.0+cu124, diffusers 0.32.1, and accelerate 1.6.0. Train Your Own SDXL LoRA on a Single GPU with kohya-ss Fine-tune a product LoRA from 20 photos with sd-scripts, then load it in diffusers for consistent on-brand images. Priya Nair https://sourcefeed.dev/u/priya nair What you'll build You'll train a ~45 MB SDXL LoRA from 15–30 photos of a product using kohya-ss/sd-scripts https://github.com/kohya-ss/sd-scripts the trainer behind the kohya ss https://github.com/bmaltais/kohya ss GUI , then load it into a diffusers https://huggingface.co/docs/diffusers pipeline and generate on-brand images with a trigger word. Prerequisites - Linux Ubuntu 22.04/24.04 or WSL2, NVIDIA GPU with 12 GB+ VRAM. The docs say 8 GB works with --network dim 4–8; this guide assumes 12 GB. - Python 3.10.x and git. sd-scripts lists 3.11/3.12 as "will work but not tested." - Verified against: sd-scripts main v0.11.1, July 2026 , PyTorch 2.6.0+cu124 RTX 50-series needs 2.8.0+cu128 , diffusers 0.32.1 and accelerate 1.6.0 both pinned by sd-scripts' requirements.txt , huggingface hub 0.34.3. - About 15 GB free disk: 7 GB for the SDXL base checkpoint plus latent caches. - 15–30 JPG/PNG photos of one subject, ideally 1024 px or larger on the short side, with varied angles, backgrounds and lighting. 1. Install sd-scripts git clone https://github.com/kohya-ss/sd-scripts.git cd sd-scripts python3.10 -m venv venv source venv/bin/activate pip install torch==2.6.0 torchvision==0.21.0 --index-url https://download.pytorch.org/whl/cu124 pip install --upgrade -r requirements.txt accelerate config default --mixed precision bf16 On an RTX 50-series card replace the torch line with pip install torch==2.8.0 torchvision==0.23.0 --index-url https://download.pytorch.org/whl/cu128 . accelerate config default writes a single-GPU config without the interactive questionnaire. Download the single-file SDXL base checkpoint 6.94 GB rather than the diffusers repo, which pulls fp32 weights twice that size: mkdir -p ~/lora/models ~/lora/data/product ~/lora/output hf download stabilityai/stable-diffusion-xl-base-1.0 sd xl base 1.0.safetensors --local-dir ~/lora/models 2. Prepare the dataset Copy your photos into ~/lora/data/product , then create one caption .txt per image with the same basename. Start every caption with a rare trigger token here ohwx followed by the class, then describe what varies between shots — background, angle, lighting — so the LoRA learns the product and not the kitchen counter: cd ~/lora/data/product for f in .jpg; do echo "ohwx bottle, product photo" "${f%.jpg}.txt"; done Now edit each file, e.g. IMG 0412.txt : ohwx bottle, product photo, on a wooden desk, window light from the left, slight top-down angle Images without a caption file fall back to class tokens from the config below, so a missing file won't crash training — it just trains on a weaker caption. 3. Write the dataset config Save as ~/lora/dataset.toml replace /home/you ; TOML doesn't expand ~ : general caption extension = ".txt" keep tokens = 1 datasets resolution = 1024 batch size = 1 enable bucket = true min bucket reso = 640 max bucket reso = 1536 bucket reso steps = 64 datasets.subsets image dir = "/home/you/lora/data/product" class tokens = "ohwx bottle" num repeats = 10 enable bucket groups images by aspect ratio so nothing gets center-cropped; the min/max values must be divisible by bucket reso steps . With 20 images × 10 repeats you get 200 steps per epoch. Add a sample prompt file so you can watch the LoRA converge. Save as ~/lora/sample prompts.txt : ohwx bottle on a marble kitchen counter, soft morning light --n blurry, lowres, watermark --w 1024 --h 1024 --d 1 --s 28 --l 7 --n is the negative prompt, --d the seed, --s steps, --l CFG scale. 4. Train cd ~/sd-scripts && source venv/bin/activate accelerate launch --num cpu threads per process 1 sdxl train network.py \ --pretrained model name or path ~/lora/models/sd xl base 1.0.safetensors \ --dataset config ~/lora/dataset.toml \ --output dir ~/lora/output --output name product-lora \ --save model as safetensors --save precision bf16 \ --network module networks.lora --network dim 16 --network alpha 8 \ --network train unet only \ --learning rate 1e-4 --optimizer type AdamW8bit \ --lr scheduler cosine --lr warmup steps 80 \ --max train epochs 8 --save every n epochs 2 \ --mixed precision bf16 --gradient checkpointing --sdpa \ --cache latents --cache latents to disk \ --cache text encoder outputs --cache text encoder outputs to disk \ --no half vae --noise offset 0.0357 --min snr gamma 5 \ --sample prompts ~/lora/sample prompts.txt --sample every n epochs 2 --sample sampler euler a \ --seed 42 --logging dir ~/lora/logs Why these flags: --network train unet only is mandatory once you cache text-encoder outputs the script asserts it , and the SDXL docs recommend it anyway because training both text encoders gives unpredictable results. --cache latents + --cache text encoder outputs skip the VAE and CLIP passes every step; combined with --gradient checkpointing , --sdpa and AdamW8bit this is what fits 1024 px training in 12 GB. --no half vae keeps the VAE in fp32 during latent caching; it's insurance against the SDXL fp16 VAE producing NaNs. --noise offset 0.0357 matches what SDXL base was trained with; --min snr gamma 5 stabilizes early loss.- 1600 total steps at 1e-4 with cosine decay is a reasonable starting point for a single object; a 4090 finishes in roughly 20–25 minutes, a 3060 in about an hour. The first run encodes latents and text embeddings to disk .npz files next to your images before the step counter starts. 5. Load the LoRA in diffusers diffusers needs PEFT https://huggingface.co/docs/peft to load LoRA weights, which sd-scripts doesn't install: pip install peft Save as ~/lora/generate.py : python import os import torch from diffusers import StableDiffusionXLPipeline home = os.path.expanduser "~/lora" pipe = StableDiffusionXLPipeline.from single file f"{home}/models/sd xl base 1.0.safetensors", torch dtype=torch.float16 .to "cuda" pipe.load lora weights f"{home}/output", weight name="product-lora.safetensors" image = pipe "ohwx bottle on a matte black coffee table, studio lighting, editorial product shot", negative prompt="blurry, lowres, watermark, text", num inference steps=30, guidance scale=7.0, cross attention kwargs={"scale": 0.8}, generator=torch.Generator "cuda" .manual seed 1 , .images 0 image.save f"{home}/product-table.png" print "saved", f"{home}/product-table.png" from single file reuses the checkpoint you already downloaded. cross attention kwargs={"scale": 0.8} dials LoRA strength between 0 base model and 1 full LoRA ; 0.7–0.9 usually keeps the subject faithful without cooking the composition. Verify it works Training should end with the step bar at 100% and two log lines: steps: 100%|██████████| 1600/1600 23:41<00:00, 1.13it/s, avr loss=0.0862 saving checkpoint: /home/you/lora/output/product-lora.safetensors model saved. avr loss for SDXL LoRA typically settles somewhere around 0.08–0.12; a value that keeps climbing or hits nan means the learning rate is too high. Check the outputs: ls -la ~/lora/output ~/lora/output/sample You should see product-lora.safetensors about 45 MB for dim 16, U-Net only , epoch checkpoints named product-lora-000002.safetensors , -000004 , -000006 , and PNGs in sample/ for epochs 2, 4, 6 and 8. Flip through the samples: by epoch 4 the bottle's shape and label should be recognizable, and by epoch 8 it should be consistent across seeds. If epoch 6 looks better than 8 over-baked, blown-out contrast , use that checkpoint instead. Then generate: python ~/lora/generate.py saved /home/you/lora/product-table.png Run it again with cross attention kwargs={"scale": 0.0} — the bottle should disappear into a generic one. That difference is your LoRA working. Troubleshooting AssertionError: network for Text Encoder cannot be trained with caching Text Encoder outputs — you passed --cache text encoder outputs without --network train unet only . Add the flag, or drop the caching flags and add --text encoder lr1 1e-5 --text encoder lr2 1e-5 if you actually want to train the text encoders. No data found. Please verify arguments train data dir must be the parent of folders with images then the script exits — image dir in dataset.toml is wrong or contains no images. For --dataset config the path must point directly at the folder holding the images no 10 bottle subfolder convention , and it must be absolute. RuntimeError: NaN detected in latents: /home/you/lora/data/product/IMG 0412.jpg — the VAE overflowed in half precision while caching. Make sure --no half vae is on the command line; if you're on a GPU without bf16 GTX 16xx/RTX 20xx and used --mixed precision fp16 , that flag is the fix. If a single file is still named, the image itself is corrupt — re-export it. torch.OutOfMemoryError: CUDA out of memory. Tried to allocate ... — first drop --sample prompts sampling loads a full inference pipeline mid-training , then set PYTORCH CUDA ALLOC CONF=expandable segments:True in front of the launch command to reduce fragmentation, then lower --network dim to 8 and resolution to 768 in the TOML. ValueError: PEFT backend is required for this method. from generate.py — peft isn't installed in the venv you're running from. pip install peft and rerun. Next steps - Train the text encoders too for a stronger trigger word: remove both --cache text encoder outputs flags and --network train unet only , add --text encoder lr1 1e-5 --text encoder lr2 1e-5 , and expect ~2 GB more VRAM. - Auto-caption larger sets with finetune/tag images by wd14 tagger.py --onnx --repo id SmilingWolf/wd-eva02-large-tagger-v3 --remove underscore needs pip install onnx onnxruntime-gpu . - Try --optimizer type Prodigy --learning rate 1.0 to skip learning-rate tuning, or LoHa/LoKr via docs/loha lokr.md for style LoRAs. - Prefer clicking? git clone --recursive https://github.com/bmaltais/kohya ss && ./setup.sh && ./gui.sh gives you the same trainer behind a Gradio UI, and it prints the equivalent CLI command it runs. - The .safetensors file drops straight into ComfyUI's models/loras or A1111's models/Lora with