Setup Diffusers for Stable Diffusion XL inference using Nvidia Pascal GPU in Linux Mint

A hands-on guide details how to run Stable Diffusion XL inference through Hugging Face's Diffusers library on an Nvidia GTX 1070 Pascal GPU under Linux Mint, installing PyTorch from the CUDA 12.4 wheel index (https://download.pytorch.org/whl/cu124) and pinning transformers to version 5.5.0, which the author found was the last release that worked with Diffusers before API changes broke the setup. The accompanying Python script uses StableDiffusionXLPipeline with the DPMSolverMultistepScheduler, torch.bfloat16 precision, 26 sampling steps, a 6.5 guidance scale and 2 images per prompt by default. The author notes Diffusers may have since fixed the transformers incompatibility but has not tested it on the GTX 1070 setup.

You need to first ensure that all the necessary drivers CUDA most important for the Nvidia GPU are installed. This post will NOT cover how to do that. What i am working with here is a Nvidia GTX 1070, just so you know. Preparing the project directory and virtual environment Create a new directory/folder whichever terminology your familiar with and cd to it from a terminal instance. Now pay attention here. Diffusers uses PyTorch https://pytorch.org/ as its backend. PyTorch has several wheel repositories for different CUDA versions , and you need to choose for Pascal GPUs the one compatible with CUDA 12.1 or 12.4. To get the right working libraries, we need to fetch them from the repository https://download.pytorch.org/whl/cu124 https://download.pytorch.org/whl/cu124 using the --index-url pip CLI parameter/argument. Now, to the Diffusers Framework https://huggingface.co/docs/diffusers/index . Once the PyTorch related things are done, the only thing to watch out for is ensuring that transformers https://github.com/huggingface/transformers is fixed at version 5.5.0 . Why is that? My hunch and days of trying to figure out why my setup broke when i updated my libraries in my virtual environment , parts of the transformers API was changed in later version, and diffusers didn't take that into account in their newer versions. From my findings, version 5.5.0 was the last working one. Since we want this to just work, make sure that you only get THAT version until diffusers decides to fix this NOTE: i think they did fix it, but i haven't tested it yet on my GTX 1070 setup . Now, lets get to what to run inside our project directory. python3 -m venv ./venv/ ./venv/bin/pip install torch torchvision --index-url 'https://download.pytorch.org/whl/cu124' ./venv/bin/pip install diffusers transformers==5.5.0 accelerate peft This should be all that is needed to get a working virtual environment setup. Using Diffusers for running Stable Diffusion XL models This is now the part in which we need to interact with the Diffusers API. In this case, a regular Python script will do the job, and so, i build one, and will share it with you. python ./venv/bin/python3 import argparse from pathlib import Path import torch from diffusers import DPMSolverMultistepScheduler, StableDiffusionXLPipeline, Constants. ========== IMAGES BATCH COUNT = 2 TORCH DTYPE = torch.bfloat16 Prepare and parse the terminal arguments. ========================================= args parser = argparse.ArgumentParser description="Custom program for text-to-image using Stable Diffusion XL models." args parser.add argument "--model", type=str, required=True, help="Huggingface repository path to the SDXL model weights.", args parser.add argument "--prompt-path", type=Path, default=Path "./prompt.txt" , help="Filepath to the prompt on what to generate.", args parser.add argument "--negative-prompt-path", type=Path, default=None, help="Filepath to the prompt on what NOT to generate. Ignored if not set.", args parser.add argument "--sampling-steps", type=int, default=26, help="How many steps are performed in the denoising procedure. More usually result in better image quality.", args parser.add argument "--guidance-scale", type=float, default=6.5, help="How close it should follow the prompt. Lower values result in more freedom, while higher ones result in more consistency.", args parser.add argument "--images-count", type=int, default=IMAGES BATCH COUNT, help="How many images should be generated per prompt.", args parser.add argument "--seed", type=int, default=None, help="The seed used to set the state of the PRNG. If not set, chooses a random one.", args parser.add argument "--output-dirpath", type=Path, default=Path "./txt2img batch" , help="Where to store the generated images.", parsed args = args parser.parse args Prepare the text-to-image pipeline to generate images. ====================================================== prompt = parsed args.prompt path.read text negative prompt = None if parsed args.negative prompt path == None else parsed args.negative prompt path.read text pipeline = StableDiffusionXLPipeline.from single file parsed args.model, custom pipeline="lpw stable diffusion xl", torch dtype=TORCH DTYPE, variant="bfp16", safety checker=None, pipeline.scheduler = DPMSolverMultistepScheduler.from config pipeline.scheduler.config, use karras sigmas=True, algorithm type="dpmsolver++", pipeline.enable model cpu offload pipeline.enable attention slicing slice size="auto" pipeline.vae.enable slicing Generate and save the images. ============================= batch count, leftover count = divmod parsed args.images count, IMAGES BATCH COUNT images batch sequence = if batch count 0: images batch sequence += IMAGES BATCH COUNT batch count if leftover count 0: images batch sequence += leftover count generator = None if parsed args.seed == None else torch.manual seed parsed args.seed parsed args.output dirpath.mkdir parents=True, exist ok=True, for num images per prompt in images batch sequence: inference result = pipeline prompt=prompt, negative prompt=negative prompt, num inference steps=parsed args.sampling steps, guidance scale=parsed args.guidance scale, num images per prompt=num images per prompt, generator=generator, file counter = 0 saved image paths = for generated image in inference result.images: output filepath = parsed args.output dirpath.joinpath f"image{file counter}.png" while output filepath.exists : file counter += 1 output filepath = parsed args.output dirpath.joinpath f"image{file counter}.png" generated image.save output filepath saved image paths.append output filepath.name print f"Saved the generated {num images per prompt} image/s {saved image paths} into the directory '{parsed args.output dirpath}'." This is what i use to generate AI images with Stable Diffusion XL models. It has a solid good enough CLI interface, and the script can be adapted for ones individual needs if something is insufficient. One important thing to note. Most Stable Diffusion XL models are around 7 GB in size . Add the pipeline components, and we are talking 8+ GB of VRAM taken. So reducing the VRAM usage is a must, and this is usually done with offloading https://huggingface.co/docs/diffusers/v0.40.0/en/optimization/memory offloading , attention slicing https://huggingface.co/docs/diffusers/v0.40.0/en/optimization/memory memory-efficient-attention and VAE slicing https://huggingface.co/docs/diffusers/v0.40.0/en/optimization/memory vae-slicing . For optimal speed, i recommend using model offloading https://huggingface.co/docs/diffusers/v0.40.0/en/optimization/memory model-offloading , but keep in mind that the whole model is loaded, with the pipeline components being swapped out as needed. This means, it will use more VRAM , and if you are short on it, get ready for out of memory exceptions . In the case of my GTX 1070, i need to ensure that almost nothing is filling the GPUs VRAM other then Diffusers doing its thing, otherwise the script will likely exit with a out of memory exception when saving the generated images , to give you an idea. Now, to save on VRAM as much as possible and sacrificing some extra speed , use cpu offloading https://huggingface.co/docs/diffusers/v0.40.0/en/optimization/memory cpu-offloading . This will make it run even on 6 GB of VRAM , but as warned, will take longer. Best i have found speed and reliability wise is using group offloading https://huggingface.co/docs/diffusers/v0.40.0/en/optimization/memory group-offloading with CUDA streaming enabled, but this required having more RAM. In my experience, 16 GB is not enough for Stable Diffusion XL models. You need around 24 GB minimum. Regarding attention slicing, due to PyTorch being kinda a PITA in dealing with Pascal based Nvidia GPU's, you must enable it using the enable attention slicing https://huggingface.co/docs/diffusers/v0.40.0/en/api/pipelines/overview diffusers.DiffusionPipeline.enable attention slicing method from the base class DiffusionPipeline https://huggingface.co/docs/diffusers/v0.40.0/en/api/pipelines/overview diffusers.DiffusionPipeline . Why is that? Due to FlashAttention https://huggingface.co/docs/diffusers/v0.40.0/en/optimization/fp16 scaled-dot-product-attention the enabled default on newer PyTorch builds simply not working atleast on my GPU , xFormers not working properly atleast on my GPU, again , so...was left with only this. VAE slicing saves memory by splitting large batches of inputs into a single batch of data and separately processing them. This method works best when generating more than one image at a time, which will be the case usually, but keep in mind that inference will take longer the more images are generated in a run. Generating images faster using Ays I was researching how to speed up the inference of Stable Diffusion XL models on my GPU, and found this gem https://research.nvidia.com/labs/toronto-ai/AlignYourSteps/ by Nvidia. And to my luck, its supported by Diffusers. Now, how useful is using the AYS schedule for inference? In my experience, so-so. Yes, it is much faster and does generate good enough images. But...the image quality in regards to prompt adherence is worse , and this depends alot on the model you are using . If it was fine tuned very well, it will perform mostly fine . If not, the images will most likely look weird . Also from my findings , it is best to use the sde-dpmsolver++ scheduler https://huggingface.co/docs/diffusers/v0.40.0/en/api/schedulers/multistep dpm solver dpmsolvermultistepscheduler for this, due to it converging faster and giving good quality results most of the time in 10 steps of inference. Anything else will be less usable, but maybe i missed something. I personally recommend using this to experiment with new ideas. To get you started on using AYS with Diffusers, i give you another script. python ./venv/bin/python3 import argparse from pathlib import Path import torch from diffusers import DPMSolverMultistepScheduler, StableDiffusionXLPipeline, from diffusers.schedulers import AysSchedules Constants. ========== IMAGES BATCH COUNT = 2 TORCH DTYPE = torch.bfloat16 Prepare and parse the terminal arguments. ========================================= args parser = argparse.ArgumentParser description="Custom program for text-to-image using normal Stable Diffusion models." args parser.add argument "--model", type=str, required=True, help="Huggingface repository path to the SDXL model weights.", args parser.add argument "--prompt-path", type=Path, default=Path "./prompt.txt" , help="Filepath to the prompt on what to generate.", args parser.add argument "--negative-prompt-path", type=Path, default=None, help="Filepath to the prompt on what NOT to generate. Ignored if not set.", args parser.add argument "--guidance-scale", type=float, default=3.0, help="How close it should follow the prompt. Lower values result in more freedom, while higher ones result in more consistency.", args parser.add argument "--images-count", type=int, default=IMAGES BATCH COUNT, help="How many images should be generated per prompt.", args parser.add argument "--seed", type=int, default=None, help="The seed used to set the state of the PRNG. If not set, chooses a random one.", args parser.add argument "--output-dirpath", type=Path, default=Path "./txt2img batch" , help="Where to store the generated images.", parsed args = args parser.parse args Prepare the text-to-image pipeline to generate images. ====================================================== prompt = parsed args.prompt path.read text negative prompt = None if parsed args.negative prompt path == None else parsed args.negative prompt path.read text pipeline = StableDiffusionXLPipeline.from single file parsed args.model, custom pipeline="lpw stable diffusion xl", torch dtype=TORCH DTYPE, safety checker=None, pipeline.scheduler = DPMSolverMultistepScheduler.from config pipeline.scheduler.config, algorithm type="sde-dpmsolver++", solver order=2, pipeline.enable model cpu offload pipeline.enable attention slicing slice size="auto" pipeline.vae.enable slicing Generate and save the images. ============================= batch count, leftover count = divmod parsed args.images count, IMAGES BATCH COUNT images batch sequence = if batch count 0: images batch sequence += IMAGES BATCH COUNT batch count if leftover count 0: images batch sequence += leftover count generator = None if parsed args.seed == None else torch.manual seed parsed args.seed parsed args.output dirpath.mkdir parents=True, exist ok=True, for num images per prompt in images batch sequence: inference result = pipeline prompt=prompt, negative prompt=negative prompt, guidance scale=parsed args.guidance scale, num images per prompt=num images per prompt, generator=generator, timesteps=AysSchedules "StableDiffusionXLTimesteps" , file counter = 0 for generated image in inference result.images: output filepath = parsed args.output dirpath.joinpath f"image{file counter}.png" while output filepath.exists : file counter += 1 output filepath = parsed args.output dirpath.joinpath f"image{file counter}.png" generated image.save output filepath print f"Saved the generated {num images per prompt} image/s into the directory '{parsed args.output dirpath}'." Enjoy. Conclusion At the time of writing this article, it is doable and works OK-ish. Software support unless obtained from other sources will get harder with time, sometimes forcing one to compile their own stuff to get it working. Now, why am i this opinion? cuML https://github.com/NVIDIA/cuml and cuDF https://github.com/NVIDIA/cudf not supporting Nvidia Pascal GPU's, but cupy https://github.com/cupy/cupy/ for whatever reason working fine. On to the topic of inference speed. It's slow, atleast on my GTX 1070. 2 image take normally around 3:40 minutes to generate with model offloading, and around 1:30 with AYS and model offloading. This is the fastest possible at the moment, atleast on a GPU as mine. I think it will be much faster on a GTX 1080ti, but these are not exacly budget cards. If you ask me, buy a used Nvidia 2000 or 3000 series GPU. Best i can recommend is the RTX 3060 12 GB model, price to performance for model inference.