SDXL Performance on Low VRAM

For now, here’s what I got from a few experiments on a Colab T4:

I tried to treat this as a standard-Diffusers control, rather than as an attempt to reproduce Celestium itself.

The main thing that stood out to me is that, at least in this control, the low-VRAM boundary moved quite a lot depending on what was resident at the same time and at what granularity the runtime offloaded it. The nominal “4 / 6 / 8 GB” number by itself was much less informative.

One important caveat up front: I used a 16 GB-class T4 and limited the PyTorch caching allocator to 4 / 6 / 8 GiB with torch.cuda.memory.set_per_process_memory_fraction(). So these are

With that limitation, the most useful part of the result looked like this:

So, very roughly, my control behaved more like:

"Does SDXL fit in N GiB?"
        ↓
"Which working set has to fit in N GiB at the same time?"
        ↓
"At what granularity can the runtime evict / reload it?"

That seems potentially relevant to your 4 / 6 / 8 GB results, especially the 6 GB Base + Refiner = partially stable boundary.

Diffusers’ own off terminology maps fairly cleanly onto this distinction. Its group-off documentation describes:

That was almost exactly what I saw at the artificial 4 GiB boundary: model-level offload did not fit, while finer-grained group/leaf/sequential policies did.

The successful block-group, leaf-group, and sequential 4 GiB runs also produced the same output file hash with the same seed/prompt in this sanity check, so I did not see an obvious output-correctness difference from changing only the residency policy.

I also tried a deliberately simple stage-lifecycle control:

SDXL Base
→ generate latent
→ move latent to CPU
→ release Base hooks
→ delete Base
→ gc.collect()
→ torch.cuda.empty_cache()
→ load Refiner
→ run Refiner

That completed under a 6 GiB allocator budget.

I would not interpret that as “6 GB should always work” — a physical 6 GB RTX 3050 is a very different environment — but it suggests that the exact Base → Refiner residency transition may be a high-information boundary.

In other words, your “partially stable” result may become easier to compare with other runtimes if a log only identifies which broad stage the unstable runs reach:

Base load
→ Base denoise
→ Base/Refiner transition
→ Refiner
→ VAE decode
→ later post-processing

Even just the stage label would separate several possibilities without requiring a large new benchmark.

I also followed up on the fragmentation side because the first OOM snapshots did show the classic pattern where there was nontrivial reserved-but-unused space, but no sufficiently large usable block.

I reran three boundary cases with:

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

PyTorch’s recent CUDA caching-allocator write-up is useful here: allocations in separate normal CUDA segments cannot simply be merged, while expandable segments remove some of those cross-segment barriers. It also explicitly notes that expandable segments do not eliminate every kind of fragmentation.

In my runs, expandable segments substantially reduced the inactive/free-block fragmentation:

So for these particular controls, I would separate two claims:

At 6 GiB with expandable segments, for example, the pipeline had essentially filled the allocator budget with active allocations before failing. At 4 GiB, model-level offload similarly ended up essentially filling the budget, while finer-grained off succeeded.

That makes me think it could be useful to treat:

working-set / residency limit
offload granularity
allocator fragmentation
long-session state stability

as related but separate axes when comparing Celestium with other low-VRAM runtimes.

That is not a diagnosis of Celestium; it is just the separation that gave the clearest results in the T4 control.

For model CPU offload at a 6 GiB allocator budget, repeated inference in the same process looked approximately like:

run 1: 33.1 s
run 2: 15.9 s
run 3: 15.5 s
run 4: 15.6 s
run 5: 15.2 s

Peak GPU allocation stayed around 5.21 GiB, with ~** 5.77 GiB reserved**.

So if performance numbers are compared between runtimes, I think it would help to distinguish cold first generation from warm/steady-state generation. Otherwise the initialization / paging / transfer cost can dominate the apparent result.

I also saw process RSS rise substantially during the first couple of offloaded runs and then roughly plateau around 9.4–9.5 GiB in this small Colab environment. PyTorch’s pinned-host allocator counters stayed essentially near zero, so I would not call this a pinned-memory leak; RSS includes much more than the pinned allocator. But it did reinforce that peak/system RAM is useful to record next to peak VRAM when evaluating an offload-heavy strategy.

A plain 8 GiB run eventually OOMed while executing self.vae.decode(...)

, specifically inside an upsampling convolution in the VAE decoder, on a 512 MiB allocation.

I also tried enable_vae_tiling()

, and at first glance it appeared to make no difference: same OOM, same memory state.

There is a subtle reason not to over-interpret that result. Diffusers’ AutoencoderKL implementation only switches to tiled decode when the latent dimension is

sample_size: 1024

; with the four VAE resolution blocks, that corresponds to a 128-latent threshold. A 1024×1024 SDXL latent is also 128×128, so the strict >

test means my exact 1024×1024 control followed the normal decode path even though tiling had been enabled.So I would treat that A/B as a useful little warning about benchmark setup, not as evidence that VAE tiling is ineffective. A deliberate tiling test should use a resolution above the default threshold or explicitly inspect/configure the tiling threshold.

So my current read from the control is:

the interesting comparison is probably not just “can SDXL generate on 4/6/8 GB?”, but “what residency/eviction policy gets it there, what working headroom does it preserve, and does that policy stay stable over stage transitions and repeated runs?”

Your 6 GB Base+Refiner boundary in particular looks like a good place to learn a lot with very little extra instrumentation. A stage label plus peak GPU memory — and ideally system RAM / cold-vs-warm timing — would already make it much easier to compare against Diffusers-style off without assuming that the two runtimes are doing the same thing internally.