{"slug": "sdxl-performance-on-low-vram", "title": "SDXL Performance on Low VRAM", "summary": "Experiments on a Google Colab T4 with a 16 GB GPU show that SDXL's low-VRAM performance depends more on the granularity of runtime offloading than on the nominal memory budget, with finer-grained policies (group, leaf, sequential) passing a 4 GiB allocator limit while model-level offload fails. A simple Base-to-Refiner stage-lifecycle control completed under a 6 GiB budget, suggesting the residency transition is a key boundary, and all successful 4 GiB runs produced identical output hashes, indicating no correctness difference from residency policy changes.", "body_md": "For now, here’s what I got from a few experiments on a Colab T4:\n\nI tried to treat this as a **standard-Diffusers control**, rather than as an attempt to reproduce Celestium itself.\n\nThe main thing that stood out to me is that, at least in this control, the low-VRAM boundary moved quite a lot depending on **what was resident at the same time and at what granularity the runtime offloaded it**. The nominal “4 / 6 / 8 GB” number by itself was much less informative.\n\nOne important caveat up front: I used a 16 GB-class T4 and limited the **PyTorch caching allocator** to 4 / 6 / 8 GiB with [ torch.cuda.memory.set_per_process_memory_fraction()](https://docs.pytorch.org/docs/main/generated/torch.cuda.memory.set_per_process_memory_fraction.html). So these are\n\nWith that limitation, the most useful part of the result looked like this:\n\n| Allocator budget | Diffusers policy / control | Result | Peak PyTorch GPU memory |\n|---|---|---|---|\n| 4 GiB | model CPU offload | OOM |\nfailed around 3.82 GiB allocated |\n| 4 GiB | group offload, block-level | PASS |\n~3.30 GiB allocated / 3.92 GiB reserved |\n| 4 GiB | group offload, leaf-level | PASS |\n~3.01 GiB / 3.69 GiB |\n| 4 GiB | sequential CPU offload | PASS |\n~3.01 GiB / 3.02 GiB |\n| 6 GiB | plain full CUDA residency | OOM |\nfailed while placing the pipeline |\n| 6 GiB | model CPU offload | PASS |\n~5.21 GiB / 5.77 GiB |\n| 6 GiB | Base → explicit unload → Refiner | PASS |\n~5.77 GiB max reserved |\n| 8 GiB | model CPU offload | PASS |\n~5.21 GiB / 5.77 GiB |\n\nSo, very roughly, my control behaved more like:\n\n```\n\"Does SDXL fit in N GiB?\"\n        ↓\n\"Which working set has to fit in N GiB at the same time?\"\n        ↓\n\"At what granularity can the runtime evict / reload it?\"\n```\n\nThat seems potentially relevant to your 4 / 6 / 8 GB results, especially the **6 GB Base + Refiner = partially stable** boundary.\n\nDiffusers’ own offloading terminology maps fairly cleanly onto this distinction. Its [group-offloading documentation](https://huggingface.co/docs/diffusers/v0.36.0/en/api/utilities) describes:\n\nThat was almost exactly what I saw at the artificial 4 GiB boundary: model-level offload did not fit, while finer-grained group/leaf/sequential policies did.\n\nThe successful block-group, leaf-group, and sequential 4 GiB runs also produced the **same output file hash** with the same seed/prompt in this sanity check, so I did not see an obvious output-correctness difference from changing only the residency policy.\n\nI also tried a deliberately simple stage-lifecycle control:\n\n```\nSDXL Base\n→ generate latent\n→ move latent to CPU\n→ release Base hooks\n→ delete Base\n→ gc.collect()\n→ torch.cuda.empty_cache()\n→ load Refiner\n→ run Refiner\n```\n\nThat completed under a **6 GiB allocator budget**.\n\nI would not interpret that as “6 GB should always work” — a physical 6 GB RTX 3050 is a very different environment — but it suggests that **the exact Base → Refiner residency transition may be a high-information boundary**.\n\nIn other words, your “partially stable” result may become easier to compare with other runtimes if a log only identifies which broad stage the unstable runs reach:\n\n```\nBase load\n→ Base denoise\n→ Base/Refiner transition\n→ Refiner\n→ VAE decode\n→ later post-processing\n```\n\nEven just the stage label would separate several possibilities without requiring a large new benchmark.\n\nI also followed up on the fragmentation side because the first OOM snapshots did show the classic pattern where there was nontrivial reserved-but-unused space, but no sufficiently large usable block.\n\nI reran three boundary cases with:\n\n```\nPYTORCH_CUDA_ALLOC_CONF=expandable_segments:True\n```\n\nPyTorch’s recent [CUDA caching-allocator write-up](https://docs.pytorch.org/devlogs/eager/2026-06-01-cuda-caching-allocator/) is useful here: allocations in separate normal CUDA segments cannot simply be merged, while expandable segments remove some of those cross-segment barriers. It also explicitly notes that expandable segments do **not** eliminate every kind of fragmentation.\n\nIn my runs, expandable segments substantially reduced the inactive/free-block fragmentation:\n\n| Control | Inactive blocks, default | With expandable segments | Pass/fail changed? |\n|---|---|---|---|\n| plain 6 GiB | ~0.253 GiB | ~0.0006 GiB | No |\n| model-offload 4 GiB | ~0.169 GiB | ~0.0056 GiB | No |\n| plain 8 GiB | ~0.315 GiB | ~0.054 GiB | No |\n\nSo for these particular controls, I would separate two claims:\n\nAt 6 GiB with expandable segments, for example, the pipeline had essentially filled the allocator budget with active allocations before failing. At 4 GiB, model-level offload similarly ended up essentially filling the budget, while finer-grained offloading succeeded.\n\nThat makes me think it could be useful to treat:\n\n```\nworking-set / residency limit\noffload granularity\nallocator fragmentation\nlong-session state stability\n```\n\nas related but separate axes when comparing Celestium with other low-VRAM runtimes.\n\nThat is not a diagnosis of Celestium; it is just the separation that gave the clearest results in the T4 control.\n\nFor model CPU offload at a 6 GiB allocator budget, repeated inference in the same process looked approximately like:\n\n```\nrun 1: 33.1 s\nrun 2: 15.9 s\nrun 3: 15.5 s\nrun 4: 15.6 s\nrun 5: 15.2 s\n```\n\nPeak GPU allocation stayed around **5.21 GiB**, with ~** 5.77 GiB reserved**.\n\nSo if performance numbers are compared between runtimes, I think it would help to distinguish **cold first generation** from **warm/steady-state generation**. Otherwise the initialization / paging / transfer cost can dominate the apparent result.\n\nI also saw process RSS rise substantially during the first couple of offloaded runs and then roughly plateau around 9.4–9.5 GiB in this small Colab environment. PyTorch’s pinned-host allocator counters stayed essentially near zero, so I would **not** call this a pinned-memory leak; RSS includes much more than the pinned allocator. But it did reinforce that peak/system RAM is useful to record next to peak VRAM when evaluating an offload-heavy strategy.\n\nA plain 8 GiB run eventually OOMed while executing `self.vae.decode(...)`\n\n, specifically inside an upsampling convolution in the VAE decoder, on a 512 MiB allocation.\n\nI also tried `enable_vae_tiling()`\n\n, and at first glance it appeared to make no difference: same OOM, same memory state.\n\nThere is a subtle reason not to over-interpret that result. Diffusers’ [ AutoencoderKL implementation](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/autoencoders/autoencoder_kl.py) only switches to tiled decode when the latent dimension is\n\n`sample_size: 1024`\n\n; with the four VAE resolution blocks, that corresponds to a 128-latent threshold. A 1024×1024 SDXL latent is also 128×128, so the strict `>`\n\ntest means my exact 1024×1024 control followed the normal decode path even though tiling had been enabled.So I would treat that A/B as a useful little warning about benchmark setup, **not** as evidence that VAE tiling is ineffective. A deliberate tiling test should use a resolution above the default threshold or explicitly inspect/configure the tiling threshold.\n\nSo my current read from the control is:\n\n**the interesting comparison is probably not just “can SDXL generate on 4/6/8 GB?”, but “what residency/eviction policy gets it there, what working headroom does it preserve, and does that policy stay stable over stage transitions and repeated runs?”**\n\nYour 6 GB Base+Refiner boundary in particular looks like a good place to learn a lot with very little extra instrumentation. A stage label plus peak GPU memory — and ideally system RAM / cold-vs-warm timing — would already make it much easier to compare against Diffusers-style offloading without assuming that the two runtimes are doing the same thing internally.", "url": "https://wpnews.pro/news/sdxl-performance-on-low-vram", "canonical_source": "https://discuss.huggingface.co/t/sdxl-performance-on-low-vram/178826#post_4", "published_at": "2026-08-21 00:19:00+00:00", "updated_at": "2026-08-21 00:43:29.989020+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "generative-ai", "ai-infrastructure"], "entities": ["SDXL", "Google Colab T4", "PyTorch", "Diffusers", "Hugging Face", "Celestium", "RTX 3050"], "alternates": {"html": "https://wpnews.pro/news/sdxl-performance-on-low-vram", "markdown": "https://wpnews.pro/news/sdxl-performance-on-low-vram.md", "text": "https://wpnews.pro/news/sdxl-performance-on-low-vram.txt", "jsonld": "https://wpnews.pro/news/sdxl-performance-on-low-vram.jsonld"}}