Fitting Gemma 4 E2B and an SDXL model on one iPhone: notes from building Anvil

Congrats on 4.1, and thanks for writing up the memory side. I work on an iPhone app that also runs Stable Diffusion through Core ML, and these are the parts of Apple’s ml-stable-diffusion README that made the most difference:

• 6-bit palettization is the floor Apple recommends for SD weights. Going lower without visible damage needs their Mixed-Bit Palettization recipes, which mix 1/2/4/6/8-bit layers to land around 4 bits on average; they publish precomputed recipes for SDXL base on Hugging Face (apple/coreml-stable-diffusion-mixed-bit-palettization).
• On iOS 17 and later, palettized weights can be decompressed just in time instead of at load. Their FAQ puts the peak-RAM saving from 16-bit to 4-bit at up to 75%.
• For the Neural Engine on iPhone they recommend SPLIT_EINSUM attention. Their SD 1.x/2.x iPhone benchmarks use SPLIT_EINSUM_V2, but they advise against V2 for SDXL because compilation takes far too long.
• Ship .mlmodelc rather than .mlpackage: the compiled model gets cached, while an .mlpackage re-runs model preparation on every load.
• reduceMemory on StableDiffusionPipeline loads and unloads the text encoder, UNet and VAE just in time, which is the same idea as your chat/image model swap, one level down.

Have you noticed quality differences between Neural Engine and GPU output at 1024? Apple mentions small differences across compute units and I’ve wondered how visible they get with SDXL.