Most of us just guess the rank when setting up a LoRA, usually defaulting to 8 or 16 because that is what the community scripts use. However, looking at the data from arXiv:2609.10656v1, cranking up the rank often yields diminishing returns. In a controlled test using a DDPM U-Net on CIFAR-10, rank 4 actually hit the best FID score at 124.1380, slightly beating out rank 8 (124.2136). If you are fighting for VRAM or trying to shave off training time, there is very little reason to go above 8 unless your dataset is incredibly complex.
How rank affects your training budget #
The trade-off here isn't just about file size; it is about the relationship between trainable parameters and actual image quality. When testing ranks across 2, 4, 8, 16, and 32, the trend showed that higher ranks increase the adaptation cost without a proportional jump in quality.
- FID Performance: Rank 4 outperformed rank 8 and 16 in the DDPM tests.
- Memory Usage: Higher ranks naturally consume more GPU memory, which can be the difference between fitting a larger batch size or hitting an Out of Memory (OOM) error.
- Runtime: Training time increases as you scale the rank, yet the "extended-budget" runs (20 epochs) confirmed that the gains from higher ranks remain limited.
Testing across different architectures #
It is easy to dismiss these results as "just a U-Net thing," but the study also looked at a Tiny DiT (Diffusion Transformer) backbone over 10 epochs. Even with a different architecture, the results for ranks 4, 8, and 16 remained consistent: moderate ranks are the most efficient.
If you are currently using a rank of 32 or 64 for a simple style or character LoRA, you are likely wasting compute. I have found that in my own experiments, jumping from rank 4 to 16 often just adds noise or overfits the small dataset faster without adding any discernible detail to the output.
Practical implementation for your next run #
If you are setting up a training script in PyTorch, keep your rank low to start. You can verify your results using a local-folder pytorch-fid protocol to ensure your FID scores are reproducible and not just a fluke of the seed.
-
Start with
rank=4. -
If the model isn't capturing the specific details of your subject, bump it to
rank=8. -
If you still see a lack of detail, the problem is likely your learning rate or dataset quality, not the rank.
For those using a budget GPU, staying at rank 4 allows for larger batches, which often stabilizes training more than a high rank ever would. Stop over-allocating parameters to your adapters; the data shows that the most efficient path to a low FID is actually a lean LoRA. Next Streaming WARDOGS and Valheim 1.0 on GeForce NOW avoids the storage headache →
All Replies （4） #
I want to try this tonight. Does this hold up for Flux or only SDXL? I'm seeing a 404 on that paper...
So stoked you're trying this! I think it works for Flux, but have you checked the Kohya_ss logs?
finally some proof. i used rank 128 for a month and it just fried my vram, but Kohya feels weird...
Excited to test this. I've noticed rank 4 works best when using AdamW with a specific learning rate... maybe 1e-4?