{"slug": "setup-diffusers-for-stable-diffusion-xl-inference-using-nvidia-pascal-gpu-in", "title": "Setup Diffusers for Stable Diffusion XL inference using Nvidia Pascal GPU in Linux Mint", "summary": "A hands-on guide details how to run Stable Diffusion XL inference through Hugging Face's Diffusers library on an Nvidia GTX 1070 Pascal GPU under Linux Mint, installing PyTorch from the CUDA 12.4 wheel index (https://download.pytorch.org/whl/cu124) and pinning transformers to version 5.5.0, which the author found was the last release that worked with Diffusers before API changes broke the setup. The accompanying Python script uses StableDiffusionXLPipeline with the DPMSolverMultistepScheduler, torch.bfloat16 precision, 26 sampling steps, a 6.5 guidance scale and 2 images per prompt by default. The author notes Diffusers may have since fixed the transformers incompatibility but has not tested it on the GTX 1070 setup.", "body_md": "You need to first ensure that all the necessary drivers (CUDA most important) for the Nvidia GPU are installed. This post will NOT cover how to do that.\n\nWhat i am working with here is a Nvidia GTX 1070, just so you know.\n\n## Preparing the project directory and virtual environment\n\nCreate a new directory/folder (whichever terminology your familiar with) and `cd` to it from a terminal instance.\n\n**Now pay attention here.**\n\nDiffusers uses [PyTorch](https://pytorch.org/) as its backend.\nPyTorch has *several wheel repositories for different CUDA versions*, and you need to choose for Pascal GPUs the one compatible with CUDA 12.1 or 12.4.\n\nTo get the right working libraries, we need to fetch them from the repository [https://download.pytorch.org/whl/cu124](https://download.pytorch.org/whl/cu124) using the `--index-url` pip CLI parameter/argument.\n\nNow, to the [Diffusers Framework](https://huggingface.co/docs/diffusers/index).\n\nOnce the PyTorch related things are done,\nthe only thing to watch out for is ensuring that [transformers](https://github.com/huggingface/transformers) is fixed at version `5.5.0` .\nWhy is that? My hunch (and days of trying to figure out why my setup broke when i updated my libraries in my virtual environment), \nparts of the `transformers` API was changed in later version, and `diffusers` didn't take that into account in their newer versions.\nFrom my findings, version `5.5.0` was the last working one. Since we want this to just work, \nmake sure that you only get THAT version until `diffusers` decides to fix this (NOTE: i think they did fix it, but i haven't tested it yet on my GTX 1070 setup).\n\nNow, lets get to what to run inside our project directory.\n\n```\npython3 -m venv ./venv/\n./venv/bin/pip install torch torchvision --index-url 'https://download.pytorch.org/whl/cu124'\n./venv/bin/pip install diffusers transformers==5.5.0 accelerate peft\n```\n\nThis should be all that is needed to get a working virtual environment setup.\n\n## Using Diffusers for running Stable Diffusion XL models\n\nThis is now the part in which we need to interact with the Diffusers API. In this case, a regular Python script will do the job, and so, i build one, and will share it with you.\n\n``` python\n#!./venv/bin/python3\nimport argparse\nfrom pathlib import Path\n\nimport torch\nfrom diffusers import (\n    DPMSolverMultistepScheduler,\n    StableDiffusionXLPipeline,\n)\n\n# Constants.\n# ==========\nIMAGES_BATCH_COUNT = 2\nTORCH_DTYPE = torch.bfloat16\n\n# Prepare and parse the terminal arguments.\n# =========================================\nargs_parser = argparse.ArgumentParser(\n    description=\"Custom program for text-to-image using Stable Diffusion XL models.\"\n)\nargs_parser.add_argument(\n    \"--model\",\n    type=str,\n    required=True,\n    help=\"Huggingface repository path to the SDXL model weights.\",\n)\nargs_parser.add_argument(\n    \"--prompt-path\",\n    type=Path,\n    default=Path(\"./prompt.txt\"),\n    help=\"Filepath to the prompt on what to generate.\",\n)\nargs_parser.add_argument(\n    \"--negative-prompt-path\",\n    type=Path,\n    default=None,\n    help=\"Filepath to the prompt on what NOT to generate. Ignored if not set.\",\n)\nargs_parser.add_argument(\n    \"--sampling-steps\",\n    type=int,\n    default=26,\n    help=\"How many steps are performed in the denoising procedure. More usually result in better image quality.\",\n)\nargs_parser.add_argument(\n    \"--guidance-scale\",\n    type=float,\n    default=6.5,\n    help=\"How close it should follow the prompt. Lower values result in more freedom, while higher ones result in more consistency.\",\n)\nargs_parser.add_argument(\n    \"--images-count\",\n    type=int,\n    default=IMAGES_BATCH_COUNT,\n    help=\"How many images should be generated per prompt.\",\n)\nargs_parser.add_argument(\n    \"--seed\",\n    type=int,\n    default=None,\n    help=\"The seed used to set the state of the PRNG. If not set, chooses a random one.\",\n)\nargs_parser.add_argument(\n    \"--output-dirpath\",\n    type=Path,\n    default=Path(\"./txt2img_batch\"),\n    help=\"Where to store the generated images.\",\n)\n\nparsed_args = args_parser.parse_args()\n\n# Prepare the text-to-image pipeline to generate images.\n# ======================================================\nprompt = parsed_args.prompt_path.read_text()\nnegative_prompt = None if parsed_args.negative_prompt_path == None else parsed_args.negative_prompt_path.read_text()\n\npipeline = StableDiffusionXLPipeline.from_single_file(\n    parsed_args.model,\n    custom_pipeline=\"lpw_stable_diffusion_xl\",\n    torch_dtype=TORCH_DTYPE,\n    variant=\"bfp16\",\n    safety_checker=None,\n)\npipeline.scheduler = DPMSolverMultistepScheduler.from_config(\n    pipeline.scheduler.config,\n    use_karras_sigmas=True,\n    algorithm_type=\"dpmsolver++\",\n)\npipeline.enable_model_cpu_offload()\npipeline.enable_attention_slicing(slice_size=\"auto\")\npipeline.vae.enable_slicing()\n\n# Generate and save the images.\n# =============================\nbatch_count, leftover_count = divmod(parsed_args.images_count, IMAGES_BATCH_COUNT)\nimages_batch_sequence = []\n\nif batch_count > 0:\n    images_batch_sequence += [IMAGES_BATCH_COUNT] * batch_count\nif leftover_count > 0:\n    images_batch_sequence += [leftover_count]\n\ngenerator = None if parsed_args.seed == None else torch.manual_seed(parsed_args.seed)\n\nparsed_args.output_dirpath.mkdir(\n    parents=True,\n    exist_ok=True,\n)\nfor num_images_per_prompt in images_batch_sequence:\n    inference_result = pipeline(\n        prompt=prompt,\n        negative_prompt=negative_prompt,\n        num_inference_steps=parsed_args.sampling_steps,\n        guidance_scale=parsed_args.guidance_scale,\n        num_images_per_prompt=num_images_per_prompt,\n        generator=generator,\n    )\n\n    file_counter = 0\n    saved_image_paths = []\n    for generated_image in inference_result.images:\n        output_filepath = parsed_args.output_dirpath.joinpath(f\"image{file_counter}.png\")\n        while output_filepath.exists():\n            file_counter += 1\n            output_filepath = parsed_args.output_dirpath.joinpath(f\"image{file_counter}.png\")\n\n        generated_image.save(output_filepath)\n        saved_image_paths.append(output_filepath.name)\n\n    print(f\"Saved the generated {num_images_per_prompt} image/s {saved_image_paths} into the directory '{parsed_args.output_dirpath}'.\")\n```\n\nThis is what i use to generate AI images with Stable Diffusion XL models. It has a solid good enough CLI interface, and the script can be adapted for ones individual needs if something is insufficient.\n\nOne important thing to note. *Most* Stable Diffusion XL models are *around 7 GB in size*. Add the pipeline components, and we are talking 8+ GB of VRAM taken.\n\nSo reducing the VRAM usage is a must, and this is usually done with [offloading](https://huggingface.co/docs/diffusers/v0.40.0/en/optimization/memory#offloading), [attention slicing](https://huggingface.co/docs/diffusers/v0.40.0/en/optimization/memory#memory-efficient-attention) and [VAE slicing](https://huggingface.co/docs/diffusers/v0.40.0/en/optimization/memory#vae-slicing).\n\nFor optimal speed, i recommend using [model offloading](https://huggingface.co/docs/diffusers/v0.40.0/en/optimization/memory#model-offloading), but keep in mind that the **whole model** is loaded,\nwith the pipeline components being swapped out as needed. This means, it will use *more VRAM*, and if you are short on it, get ready for *out of memory exceptions*.\nIn the case of my GTX 1070, i need to ensure that *almost nothing is filling the GPUs VRAM* other then Diffusers doing its thing, otherwise the script will likely exit with a *out of memory exception when saving the generated images*, to give you an idea.\n\nNow, to save on VRAM as much as possible (and sacrificing some extra speed), use [cpu offloading](https://huggingface.co/docs/diffusers/v0.40.0/en/optimization/memory#cpu-offloading).\nThis will make it run *even on 6 GB of VRAM*, but as warned, will take longer.\n\nBest i have found (speed and reliability wise) is using [group offloading](https://huggingface.co/docs/diffusers/v0.40.0/en/optimization/memory#group-offloading) with CUDA streaming enabled, but this required having more RAM. In my experience, 16 GB is not enough for Stable Diffusion XL models. You need around 24 GB minimum.\n\nRegarding attention slicing, due to PyTorch being kinda a PITA in dealing with Pascal based Nvidia GPU's, you must enable it using the [enable_attention_slicing](https://huggingface.co/docs/diffusers/v0.40.0/en/api/pipelines/overview#diffusers.DiffusionPipeline.enable_attention_slicing) method from the base class [DiffusionPipeline](https://huggingface.co/docs/diffusers/v0.40.0/en/api/pipelines/overview#diffusers.DiffusionPipeline). Why is that? Due to [FlashAttention](https://huggingface.co/docs/diffusers/v0.40.0/en/optimization/fp16#scaled-dot-product-attention) (the enabled default on newer PyTorch builds) simply not working (atleast on my GPU), xFormers not working properly (atleast on my GPU, again), so...was left with only this.\n\nVAE slicing saves memory by splitting large batches of inputs into a single batch of data and separately processing them. This method works best when generating more than one image at a time, which will be the case usually, but keep in mind that inference will take longer the more images are generated in a run.\n\n## Generating images faster using Ays\n\nI was researching how to speed up the inference of Stable Diffusion XL models on my GPU, \nand found [this gem](https://research.nvidia.com/labs/toronto-ai/AlignYourSteps/) by Nvidia.\nAnd to my luck, its supported by Diffusers.\n\nNow, how useful is using the AYS schedule for inference? In my experience, so-so.\n\nYes, it is *much faster* and does generate good enough images.\nBut...the image quality in regards to prompt adherence *is worse*,\nand this depends *alot on the model you are using*.\nIf it was fine tuned very well, it will perform *mostly fine*.\nIf not, the images *will most likely look weird*.\n\nAlso (from my findings), it is best to use the [sde-dpmsolver++ scheduler](https://huggingface.co/docs/diffusers/v0.40.0/en/api/schedulers/multistep_dpm_solver#dpmsolvermultistepscheduler) for this, \ndue to it converging faster and giving good quality results (most of the time) in 10 steps of inference.\nAnything else will be less usable, but maybe i missed something.\n\nI personally recommend using this to experiment with new ideas.\n\nTo get you started on using AYS with Diffusers, i give you another script.\n\n``` python\n#!./venv/bin/python3\nimport argparse\nfrom pathlib import Path\n\nimport torch\nfrom diffusers import (\n    DPMSolverMultistepScheduler,\n    StableDiffusionXLPipeline,\n)\nfrom diffusers.schedulers import AysSchedules\n\n# Constants.\n# ==========\nIMAGES_BATCH_COUNT = 2\nTORCH_DTYPE = torch.bfloat16\n\n# Prepare and parse the terminal arguments.\n# =========================================\nargs_parser = argparse.ArgumentParser(\n    description=\"Custom program for text-to-image using normal Stable Diffusion models.\"\n)\nargs_parser.add_argument(\n    \"--model\",\n    type=str,\n    required=True,\n    help=\"Huggingface repository path to the SDXL model weights.\",\n)\nargs_parser.add_argument(\n    \"--prompt-path\",\n    type=Path,\n    default=Path(\"./prompt.txt\"),\n    help=\"Filepath to the prompt on what to generate.\",\n)\nargs_parser.add_argument(\n    \"--negative-prompt-path\",\n    type=Path,\n    default=None,\n    help=\"Filepath to the prompt on what NOT to generate. Ignored if not set.\",\n)\nargs_parser.add_argument(\n    \"--guidance-scale\",\n    type=float,\n    default=3.0,\n    help=\"How close it should follow the prompt. Lower values result in more freedom, while higher ones result in more consistency.\",\n)\nargs_parser.add_argument(\n    \"--images-count\",\n    type=int,\n    default=IMAGES_BATCH_COUNT,\n    help=\"How many images should be generated per prompt.\",\n)\nargs_parser.add_argument(\n    \"--seed\",\n    type=int,\n    default=None,\n    help=\"The seed used to set the state of the PRNG. If not set, chooses a random one.\",\n)\nargs_parser.add_argument(\n    \"--output-dirpath\",\n    type=Path,\n    default=Path(\"./txt2img_batch\"),\n    help=\"Where to store the generated images.\",\n)\n\nparsed_args = args_parser.parse_args()\n\n# Prepare the text-to-image pipeline to generate images.\n# ======================================================\nprompt = parsed_args.prompt_path.read_text()\nnegative_prompt = None if parsed_args.negative_prompt_path == None else parsed_args.negative_prompt_path.read_text()\n\npipeline = StableDiffusionXLPipeline.from_single_file(\n    parsed_args.model,\n    custom_pipeline=\"lpw_stable_diffusion_xl\",\n    torch_dtype=TORCH_DTYPE,\n    safety_checker=None,\n)\npipeline.scheduler = DPMSolverMultistepScheduler.from_config(\n    pipeline.scheduler.config,\n    algorithm_type=\"sde-dpmsolver++\",\n    solver_order=2,\n)\npipeline.enable_model_cpu_offload()\npipeline.enable_attention_slicing(slice_size=\"auto\")\npipeline.vae.enable_slicing()\n\n# Generate and save the images.\n# =============================\nbatch_count, leftover_count = divmod(parsed_args.images_count, IMAGES_BATCH_COUNT)\nimages_batch_sequence = []\n\nif batch_count > 0:\n    images_batch_sequence += [IMAGES_BATCH_COUNT] * batch_count\nif leftover_count > 0:\n    images_batch_sequence += [leftover_count]\n\ngenerator = None if parsed_args.seed == None else torch.manual_seed(parsed_args.seed)\n\nparsed_args.output_dirpath.mkdir(\n    parents=True,\n    exist_ok=True,\n)\nfor num_images_per_prompt in images_batch_sequence:\n    inference_result = pipeline(\n        prompt=prompt,\n        negative_prompt=negative_prompt,\n        guidance_scale=parsed_args.guidance_scale,\n        num_images_per_prompt=num_images_per_prompt,\n        generator=generator,\n        timesteps=AysSchedules[\"StableDiffusionXLTimesteps\"],\n    )\n\n    file_counter = 0\n    for generated_image in inference_result.images:\n        output_filepath = parsed_args.output_dirpath.joinpath(f\"image{file_counter}.png\")\n        while output_filepath.exists():\n            file_counter += 1\n            output_filepath = parsed_args.output_dirpath.joinpath(f\"image{file_counter}.png\")\n\n        generated_image.save(output_filepath)\n\n    print(f\"Saved the generated {num_images_per_prompt} image/s into the directory '{parsed_args.output_dirpath}'.\")\n```\n\nEnjoy.\n\n## Conclusion\n\nAt the time of writing this article, it is doable and works OK-ish.\n\nSoftware support (unless obtained from other sources) will get harder with time,\nsometimes forcing one to compile their own stuff to get it working.\nNow, why am i this opinion? [cuML](https://github.com/NVIDIA/cuml) and [cuDF](https://github.com/NVIDIA/cudf) not supporting Nvidia Pascal GPU's, \nbut [cupy](https://github.com/cupy/cupy/) for whatever reason working fine.\n\nOn to the topic of inference speed. It's slow, atleast on my GTX 1070. 2 image take normally around 3:40 minutes to generate with model offloading, and around 1:30 with AYS and model offloading. This is the fastest possible at the moment, atleast on a GPU as mine. I think it will be much faster on a GTX 1080ti, but these are not exacly budget cards.\n\nIf you ask me, buy a used Nvidia 2000 or 3000 series GPU. Best i can recommend is the RTX 3060 12 GB model, price to performance for model inference.", "url": "https://wpnews.pro/news/setup-diffusers-for-stable-diffusion-xl-inference-using-nvidia-pascal-gpu-in", "canonical_source": "https://zansf-personal-blog.statichost.page/setup-diffusers-for-stable-diffusion-xl-inference-using-nvidia-pascal-gpu-in-linux-mint.html", "published_at": "2026-09-30 22:00:00+00:00", "updated_at": "2026-10-24 23:43:48.555900+00:00", "lang": "en", "topics": ["generative-ai", "ai-tools", "large-language-models"], "entities": ["Nvidia", "GTX 1070", "Linux Mint", "Hugging Face", "Diffusers", "Stable Diffusion XL", "PyTorch", "transformers"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/setup-diffusers-for-stable-diffusion-xl-inference-using-nvidia-pascal-gpu-in", "markdown": "https://wpnews.pro/news/setup-diffusers-for-stable-diffusion-xl-inference-using-nvidia-pascal-gpu-in.md", "text": "https://wpnews.pro/news/setup-diffusers-for-stable-diffusion-xl-inference-using-nvidia-pascal-gpu-in.txt", "jsonld": "https://wpnews.pro/news/setup-diffusers-for-stable-diffusion-xl-inference-using-nvidia-pascal-gpu-in.jsonld"}}