SDXL Performance on Low VRAM

Experiments on a Google Colab T4 with a 16 GB GPU show that SDXL's low-VRAM performance depends more on the granularity of runtime offloading than on the nominal memory budget, with finer-grained policies (group, leaf, sequential) passing a 4 GiB allocator limit while model-level offload fails. A simple Base-to-Refiner stage-lifecycle control completed under a 6 GiB budget, suggesting the residency transition is a key boundary, and all successful 4 GiB runs produced identical output hashes, indicating no correctness difference from residency policy changes.

For now, here’s what I got from a few experiments on a Colab T4: I tried to treat this as a standard-Diffusers control , rather than as an attempt to reproduce Celestium itself. The main thing that stood out to me is that, at least in this control, the low-VRAM boundary moved quite a lot depending on what was resident at the same time and at what granularity the runtime offloaded it . The nominal “4 / 6 / 8 GB” number by itself was much less informative. One important caveat up front: I used a 16 GB-class T4 and limited the PyTorch caching allocator to 4 / 6 / 8 GiB with torch.cuda.memory.set per process memory fraction https://docs.pytorch.org/docs/main/generated/torch.cuda.memory.set per process memory fraction.html . So these are With that limitation, the most useful part of the result looked like this: | Allocator budget | Diffusers policy / control | Result | Peak PyTorch GPU memory | |---|---|---|---| | 4 GiB | model CPU offload | OOM | failed around 3.82 GiB allocated | | 4 GiB | group offload, block-level | PASS | ~3.30 GiB allocated / 3.92 GiB reserved | | 4 GiB | group offload, leaf-level | PASS | ~3.01 GiB / 3.69 GiB | | 4 GiB | sequential CPU offload | PASS | ~3.01 GiB / 3.02 GiB | | 6 GiB | plain full CUDA residency | OOM | failed while placing the pipeline | | 6 GiB | model CPU offload | PASS | ~5.21 GiB / 5.77 GiB | | 6 GiB | Base → explicit unload → Refiner | PASS | ~5.77 GiB max reserved | | 8 GiB | model CPU offload | PASS | ~5.21 GiB / 5.77 GiB | So, very roughly, my control behaved more like: "Does SDXL fit in N GiB?" ↓ "Which working set has to fit in N GiB at the same time?" ↓ "At what granularity can the runtime evict / reload it?" That seems potentially relevant to your 4 / 6 / 8 GB results, especially the 6 GB Base + Refiner = partially stable boundary. Diffusers’ own offloading terminology maps fairly cleanly onto this distinction. Its group-offloading documentation https://huggingface.co/docs/diffusers/v0.36.0/en/api/utilities describes: That was almost exactly what I saw at the artificial 4 GiB boundary: model-level offload did not fit, while finer-grained group/leaf/sequential policies did. The successful block-group, leaf-group, and sequential 4 GiB runs also produced the same output file hash with the same seed/prompt in this sanity check, so I did not see an obvious output-correctness difference from changing only the residency policy. I also tried a deliberately simple stage-lifecycle control: SDXL Base → generate latent → move latent to CPU → release Base hooks → delete Base → gc.collect → torch.cuda.empty cache → load Refiner → run Refiner That completed under a 6 GiB allocator budget . I would not interpret that as “6 GB should always work” — a physical 6 GB RTX 3050 is a very different environment — but it suggests that the exact Base → Refiner residency transition may be a high-information boundary . In other words, your “partially stable” result may become easier to compare with other runtimes if a log only identifies which broad stage the unstable runs reach: Base load → Base denoise → Base/Refiner transition → Refiner → VAE decode → later post-processing Even just the stage label would separate several possibilities without requiring a large new benchmark. I also followed up on the fragmentation side because the first OOM snapshots did show the classic pattern where there was nontrivial reserved-but-unused space, but no sufficiently large usable block. I reran three boundary cases with: PYTORCH CUDA ALLOC CONF=expandable segments:True PyTorch’s recent CUDA caching-allocator write-up https://docs.pytorch.org/devlogs/eager/2026-06-01-cuda-caching-allocator/ is useful here: allocations in separate normal CUDA segments cannot simply be merged, while expandable segments remove some of those cross-segment barriers. It also explicitly notes that expandable segments do not eliminate every kind of fragmentation. In my runs, expandable segments substantially reduced the inactive/free-block fragmentation: | Control | Inactive blocks, default | With expandable segments | Pass/fail changed? | |---|---|---|---| | plain 6 GiB | ~0.253 GiB | ~0.0006 GiB | No | | model-offload 4 GiB | ~0.169 GiB | ~0.0056 GiB | No | | plain 8 GiB | ~0.315 GiB | ~0.054 GiB | No | So for these particular controls, I would separate two claims: At 6 GiB with expandable segments, for example, the pipeline had essentially filled the allocator budget with active allocations before failing. At 4 GiB, model-level offload similarly ended up essentially filling the budget, while finer-grained offloading succeeded. That makes me think it could be useful to treat: working-set / residency limit offload granularity allocator fragmentation long-session state stability as related but separate axes when comparing Celestium with other low-VRAM runtimes. That is not a diagnosis of Celestium; it is just the separation that gave the clearest results in the T4 control. For model CPU offload at a 6 GiB allocator budget, repeated inference in the same process looked approximately like: run 1: 33.1 s run 2: 15.9 s run 3: 15.5 s run 4: 15.6 s run 5: 15.2 s Peak GPU allocation stayed around 5.21 GiB , with ~ 5.77 GiB reserved . So if performance numbers are compared between runtimes, I think it would help to distinguish cold first generation from warm/steady-state generation . Otherwise the initialization / paging / transfer cost can dominate the apparent result. I also saw process RSS rise substantially during the first couple of offloaded runs and then roughly plateau around 9.4–9.5 GiB in this small Colab environment. PyTorch’s pinned-host allocator counters stayed essentially near zero, so I would not call this a pinned-memory leak; RSS includes much more than the pinned allocator. But it did reinforce that peak/system RAM is useful to record next to peak VRAM when evaluating an offload-heavy strategy. A plain 8 GiB run eventually OOMed while executing self.vae.decode ... , specifically inside an upsampling convolution in the VAE decoder, on a 512 MiB allocation. I also tried enable vae tiling , and at first glance it appeared to make no difference: same OOM, same memory state. There is a subtle reason not to over-interpret that result. Diffusers’ AutoencoderKL implementation https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/autoencoders/autoencoder kl.py only switches to tiled decode when the latent dimension is sample size: 1024 ; with the four VAE resolution blocks, that corresponds to a 128-latent threshold. A 1024×1024 SDXL latent is also 128×128, so the strict test means my exact 1024×1024 control followed the normal decode path even though tiling had been enabled.So I would treat that A/B as a useful little warning about benchmark setup, not as evidence that VAE tiling is ineffective. A deliberate tiling test should use a resolution above the default threshold or explicitly inspect/configure the tiling threshold. So my current read from the control is: the interesting comparison is probably not just “can SDXL generate on 4/6/8 GB?”, but “what residency/eviction policy gets it there, what working headroom does it preserve, and does that policy stay stable over stage transitions and repeated runs?” Your 6 GB Base+Refiner boundary in particular looks like a good place to learn a lot with very little extra instrumentation. A stage label plus peak GPU memory — and ideally system RAM / cold-vs-warm timing — would already make it much easier to compare against Diffusers-style offloading without assuming that the two runtimes are doing the same thing internally.