{"slug": "train-your-own-sdxl-lora-on-a-single-gpu-with-kohya-ss", "title": "Train Your Own SDXL LoRA on a Single GPU with kohya-ss", "summary": "Kohya-ss/sd-scripts enables training a ~45 MB SDXL LoRA from 15–30 product photos on a single 12 GB GPU, with a step-by-step guide covering installation, dataset preparation, and training. The workflow uses a trigger word and diffusers for consistent on-brand image generation, verified with sd-scripts v0.11.1, PyTorch 2.6.0+cu124, diffusers 0.32.1, and accelerate 1.6.0.", "body_md": "# Train Your Own SDXL LoRA on a Single GPU with kohya-ss\n\nFine-tune a product LoRA from 20 photos with sd-scripts, then load it in diffusers for consistent on-brand images.\n\n[Priya Nair](https://sourcefeed.dev/u/priya_nair)\n\n## What you'll build\n\nYou'll train a ~45 MB SDXL LoRA from 15–30 photos of a product using [kohya-ss/sd-scripts](https://github.com/kohya-ss/sd-scripts) (the trainer behind the [kohya_ss](https://github.com/bmaltais/kohya_ss) GUI), then load it into a [diffusers](https://huggingface.co/docs/diffusers) pipeline and generate on-brand images with a trigger word.\n\n## Prerequisites\n\n- Linux (Ubuntu 22.04/24.04) or WSL2, NVIDIA GPU with 12 GB+ VRAM. The docs say 8 GB works with\n`--network_dim`\n\n4–8; this guide assumes 12 GB. - Python 3.10.x and git. sd-scripts lists 3.11/3.12 as \"will work but not tested.\"\n- Verified against: sd-scripts\n`main`\n\n(v0.11.1, July 2026), PyTorch 2.6.0+cu124 (RTX 50-series needs 2.8.0+cu128), diffusers 0.32.1 and accelerate 1.6.0 (both pinned by sd-scripts'`requirements.txt`\n\n), huggingface_hub 0.34.3. - About 15 GB free disk: 7 GB for the SDXL base checkpoint plus latent caches.\n- 15–30 JPG/PNG photos of one subject, ideally 1024 px or larger on the short side, with varied angles, backgrounds and lighting.\n\n## 1. Install sd-scripts\n\n```\ngit clone https://github.com/kohya-ss/sd-scripts.git\ncd sd-scripts\npython3.10 -m venv venv\nsource venv/bin/activate\npip install torch==2.6.0 torchvision==0.21.0 --index-url https://download.pytorch.org/whl/cu124\npip install --upgrade -r requirements.txt\naccelerate config default --mixed_precision bf16\n```\n\nOn an RTX 50-series card replace the torch line with `pip install torch==2.8.0 torchvision==0.23.0 --index-url https://download.pytorch.org/whl/cu128`\n\n. `accelerate config default`\n\nwrites a single-GPU config without the interactive questionnaire.\n\nDownload the single-file SDXL base checkpoint (6.94 GB) rather than the diffusers repo, which pulls fp32 weights twice that size:\n\n```\nmkdir -p ~/lora/models ~/lora/data/product ~/lora/output\nhf download stabilityai/stable-diffusion-xl-base-1.0 sd_xl_base_1.0.safetensors --local-dir ~/lora/models\n```\n\n## 2. Prepare the dataset\n\nCopy your photos into `~/lora/data/product`\n\n, then create one caption `.txt`\n\nper image with the same basename. Start every caption with a rare trigger token (here `ohwx`\n\n) followed by the class, then describe what *varies* between shots — background, angle, lighting — so the LoRA learns the product and not the kitchen counter:\n\n```\ncd ~/lora/data/product\nfor f in *.jpg; do echo \"ohwx bottle, product photo\" > \"${f%.jpg}.txt\"; done\n```\n\nNow edit each file, e.g. `IMG_0412.txt`\n\n:\n\n```\nohwx bottle, product photo, on a wooden desk, window light from the left, slight top-down angle\n```\n\nImages without a caption file fall back to `class_tokens`\n\nfrom the config below, so a missing file won't crash training — it just trains on a weaker caption.\n\n## 3. Write the dataset config\n\nSave as `~/lora/dataset.toml`\n\n(replace `/home/you`\n\n; TOML doesn't expand `~`\n\n):\n\n```\n[general]\ncaption_extension = \".txt\"\nkeep_tokens = 1\n\n[[datasets]]\nresolution = 1024\nbatch_size = 1\nenable_bucket = true\nmin_bucket_reso = 640\nmax_bucket_reso = 1536\nbucket_reso_steps = 64\n\n  [[datasets.subsets]]\n  image_dir = \"/home/you/lora/data/product\"\n  class_tokens = \"ohwx bottle\"\n  num_repeats = 10\n```\n\n`enable_bucket`\n\ngroups images by aspect ratio so nothing gets center-cropped; the min/max values must be divisible by `bucket_reso_steps`\n\n. With 20 images × 10 repeats you get 200 steps per epoch.\n\nAdd a sample prompt file so you can watch the LoRA converge. Save as `~/lora/sample_prompts.txt`\n\n:\n\n```\nohwx bottle on a marble kitchen counter, soft morning light --n blurry, lowres, watermark --w 1024 --h 1024 --d 1 --s 28 --l 7\n```\n\n`--n`\n\nis the negative prompt, `--d`\n\nthe seed, `--s`\n\nsteps, `--l`\n\nCFG scale.\n\n## 4. Train\n\n```\ncd ~/sd-scripts && source venv/bin/activate\naccelerate launch --num_cpu_threads_per_process 1 sdxl_train_network.py \\\n  --pretrained_model_name_or_path ~/lora/models/sd_xl_base_1.0.safetensors \\\n  --dataset_config ~/lora/dataset.toml \\\n  --output_dir ~/lora/output --output_name product-lora \\\n  --save_model_as safetensors --save_precision bf16 \\\n  --network_module networks.lora --network_dim 16 --network_alpha 8 \\\n  --network_train_unet_only \\\n  --learning_rate 1e-4 --optimizer_type AdamW8bit \\\n  --lr_scheduler cosine --lr_warmup_steps 80 \\\n  --max_train_epochs 8 --save_every_n_epochs 2 \\\n  --mixed_precision bf16 --gradient_checkpointing --sdpa \\\n  --cache_latents --cache_latents_to_disk \\\n  --cache_text_encoder_outputs --cache_text_encoder_outputs_to_disk \\\n  --no_half_vae --noise_offset 0.0357 --min_snr_gamma 5 \\\n  --sample_prompts ~/lora/sample_prompts.txt --sample_every_n_epochs 2 --sample_sampler euler_a \\\n  --seed 42 --logging_dir ~/lora/logs\n```\n\nWhy these flags:\n\n`--network_train_unet_only`\n\nis mandatory once you cache text-encoder outputs (the script asserts it), and the SDXL docs recommend it anyway because training both text encoders gives unpredictable results.`--cache_latents`\n\n+`--cache_text_encoder_outputs`\n\nskip the VAE and CLIP passes every step; combined with`--gradient_checkpointing`\n\n,`--sdpa`\n\nand`AdamW8bit`\n\nthis is what fits 1024 px training in 12 GB.`--no_half_vae`\n\nkeeps the VAE in fp32 during latent caching; it's insurance against the SDXL fp16 VAE producing NaNs.`--noise_offset 0.0357`\n\nmatches what SDXL base was trained with;`--min_snr_gamma 5`\n\nstabilizes early loss.- 1600 total steps at 1e-4 with cosine decay is a reasonable starting point for a single object; a 4090 finishes in roughly 20–25 minutes, a 3060 in about an hour.\n\nThe first run encodes latents and text embeddings to disk (`.npz`\n\nfiles next to your images) before the step counter starts.\n\n## 5. Load the LoRA in diffusers\n\ndiffusers needs [PEFT](https://huggingface.co/docs/peft) to load LoRA weights, which sd-scripts doesn't install:\n\n```\npip install peft\n```\n\nSave as `~/lora/generate.py`\n\n:\n\n``` python\nimport os\nimport torch\nfrom diffusers import StableDiffusionXLPipeline\n\nhome = os.path.expanduser(\"~/lora\")\n\npipe = StableDiffusionXLPipeline.from_single_file(\n    f\"{home}/models/sd_xl_base_1.0.safetensors\", torch_dtype=torch.float16\n).to(\"cuda\")\n\npipe.load_lora_weights(f\"{home}/output\", weight_name=\"product-lora.safetensors\")\n\nimage = pipe(\n    \"ohwx bottle on a matte black coffee table, studio lighting, editorial product shot\",\n    negative_prompt=\"blurry, lowres, watermark, text\",\n    num_inference_steps=30,\n    guidance_scale=7.0,\n    cross_attention_kwargs={\"scale\": 0.8},\n    generator=torch.Generator(\"cuda\").manual_seed(1),\n).images[0]\n\nimage.save(f\"{home}/product-table.png\")\nprint(\"saved\", f\"{home}/product-table.png\")\n```\n\n`from_single_file`\n\nreuses the checkpoint you already downloaded. `cross_attention_kwargs={\"scale\": 0.8}`\n\ndials LoRA strength between 0 (base model) and 1 (full LoRA); 0.7–0.9 usually keeps the subject faithful without cooking the composition.\n\n## Verify it works\n\nTraining should end with the step bar at 100% and two log lines:\n\n```\nsteps: 100%|██████████| 1600/1600 [23:41<00:00,  1.13it/s, avr_loss=0.0862]\n\nsaving checkpoint: /home/you/lora/output/product-lora.safetensors\nmodel saved.\n```\n\n`avr_loss`\n\nfor SDXL LoRA typically settles somewhere around 0.08–0.12; a value that keeps climbing or hits `nan`\n\nmeans the learning rate is too high.\n\nCheck the outputs:\n\n```\nls -la ~/lora/output ~/lora/output/sample\n```\n\nYou should see `product-lora.safetensors`\n\n(about 45 MB for dim 16, U-Net only), epoch checkpoints named `product-lora-000002.safetensors`\n\n, `-000004`\n\n, `-000006`\n\n, and PNGs in `sample/`\n\nfor epochs 2, 4, 6 and 8. Flip through the samples: by epoch 4 the bottle's shape and label should be recognizable, and by epoch 8 it should be consistent across seeds. If epoch 6 looks better than 8 (over-baked, blown-out contrast), use that checkpoint instead.\n\nThen generate:\n\n```\npython ~/lora/generate.py\nsaved /home/you/lora/product-table.png\n```\n\nRun it again with `cross_attention_kwargs={\"scale\": 0.0}`\n\n— the bottle should disappear into a generic one. That difference is your LoRA working.\n\n## Troubleshooting\n\n** AssertionError: network for Text Encoder cannot be trained with caching Text Encoder outputs** — you passed\n\n`--cache_text_encoder_outputs`\n\nwithout `--network_train_unet_only`\n\n. Add the flag, or drop the caching flags and add `--text_encoder_lr1 1e-5 --text_encoder_lr2 1e-5`\n\nif you actually want to train the text encoders.** No data found. Please verify arguments (train_data_dir must be the parent of folders with images)** then the script exits —\n\n`image_dir`\n\nin `dataset.toml`\n\nis wrong or contains no images. For `--dataset_config`\n\nthe path must point directly at the folder holding the images (no `10_bottle`\n\nsubfolder convention), and it must be absolute.** RuntimeError: NaN detected in latents: /home/you/lora/data/product/IMG_0412.jpg** — the VAE overflowed in half precision while caching. Make sure\n\n`--no_half_vae`\n\nis on the command line; if you're on a GPU without bf16 (GTX 16xx/RTX 20xx) and used `--mixed_precision fp16`\n\n, that flag is the fix. If a single file is still named, the image itself is corrupt — re-export it.** torch.OutOfMemoryError: CUDA out of memory. Tried to allocate ...** — first drop\n\n`--sample_prompts`\n\n(sampling loads a full inference pipeline mid-training), then set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`\n\nin front of the launch command to reduce fragmentation, then lower `--network_dim`\n\nto 8 and `resolution`\n\nto 768 in the TOML.** ValueError: PEFT backend is required for this method.** from\n\n`generate.py`\n\n— `peft`\n\nisn't installed in the venv you're running from. `pip install peft`\n\nand rerun.## Next steps\n\n- Train the text encoders too for a stronger trigger word: remove both\n`--cache_text_encoder_outputs*`\n\nflags and`--network_train_unet_only`\n\n, add`--text_encoder_lr1 1e-5 --text_encoder_lr2 1e-5`\n\n, and expect ~2 GB more VRAM. - Auto-caption larger sets with\n`finetune/tag_images_by_wd14_tagger.py --onnx --repo_id SmilingWolf/wd-eva02-large-tagger-v3 --remove_underscore`\n\n(needs`pip install onnx onnxruntime-gpu`\n\n). - Try\n`--optimizer_type Prodigy --learning_rate 1.0`\n\nto skip learning-rate tuning, or LoHa/LoKr via`docs/loha_lokr.md`\n\nfor style LoRAs. - Prefer clicking?\n`git clone --recursive https://github.com/bmaltais/kohya_ss && ./setup.sh && ./gui.sh`\n\ngives you the same trainer behind a Gradio UI, and it prints the equivalent CLI command it runs. - The\n`.safetensors`\n\nfile drops straight into ComfyUI's`models/loras`\n\nor A1111's`models/Lora`\n\nwith`<lora:product-lora:0.8>`\n\nin the prompt.\n\n## Sources & further reading\n\n-\n[kohya-ss/sd-scripts README (installation, SDXL training notes)](https://github.com/kohya-ss/sd-scripts)— github.com -\n[How to Use the SDXL LoRA Training Script sdxl_train_network.py](https://github.com/kohya-ss/sd-scripts/blob/main/docs/sdxl_train_network.md)— github.com -\n[sd-scripts Dataset Configuration Guide](https://github.com/kohya-ss/sd-scripts/blob/main/docs/config_README-en.md)— github.com -\n[sd-scripts SDXL training recommendations](https://github.com/kohya-ss/sd-scripts/blob/main/docs/train_SDXL-en.md)— github.com -\n[Diffusers v0.32.1 - Load adapters (LoRA)](https://huggingface.co/docs/diffusers/v0.32.1/en/using-diffusers/loading_adapters)— huggingface.co -\n[Accelerate CLI reference (config default, launch)](https://huggingface.co/docs/accelerate/package_reference/cli)— huggingface.co\n\n[Priya Nair](https://sourcefeed.dev/u/priya_nair)· AI & Developer Experience Writer\n\nPriya covers AI frameworks, developer productivity tooling, and the startup ecosystem across South and Southeast Asia, bringing a researcher's rigour and a practitioner's empathy to every story. She is deeply sceptical of benchmarks and asks hard questions so her readers don't have to.\n\n## Discussion 1\n\ncurious how this handles versioning drift - have you pinned everything or does it still work with latest diffusers", "url": "https://wpnews.pro/news/train-your-own-sdxl-lora-on-a-single-gpu-with-kohya-ss", "canonical_source": "https://sourcefeed.dev/a/train-your-own-sdxl-lora-on-a-single-gpu-with-kohya-ss", "published_at": "2026-08-24 17:43:19+00:00", "updated_at": "2026-08-24 18:12:48.369462+00:00", "lang": "en", "topics": ["machine-learning", "generative-ai", "ai-tools", "ai-infrastructure"], "entities": ["kohya-ss/sd-scripts", "kohya_ss", "diffusers", "stabilityai/stable-diffusion-xl-base-1.0", "PyTorch", "accelerate", "huggingface_hub", "Priya Nair"], "alternates": {"html": "https://wpnews.pro/news/train-your-own-sdxl-lora-on-a-single-gpu-with-kohya-ss", "markdown": "https://wpnews.pro/news/train-your-own-sdxl-lora-on-a-single-gpu-with-kohya-ss.md", "text": "https://wpnews.pro/news/train-your-own-sdxl-lora-on-a-single-gpu-with-kohya-ss.txt", "jsonld": "https://wpnews.pro/news/train-your-own-sdxl-lora-on-a-single-gpu-with-kohya-ss.jsonld"}}