Quark Support for HuggingFace Diffusers and SVDQuant AMD Quark now supports SVDQuant and native Hugging Face Diffusers integration for diffusion models, enabling 4-bit quantization of both weights and activations. On an AMD Instinct MI350 GPU, plain native FP4 quantization of FLUX.1-dev achieves a 1.12× speedup over BF16 with a 33% reduction in model memory, while native SVDQuant trades speed (0.92×) for accuracy and a 27% memory reduction. Quark Support for HuggingFace Diffusers and SVDQuant quark-support-for-huggingface-diffusers-and-svdquant Diffusion models are heavy on memory and compute: a single text-to-image call runs a large transformer or UNet dozens of times. Quantization — storing weights and sometimes activations in low precision — is one of the most effective ways to cut both the memory footprint and the latency of these models. In a previous post 1 references we showed how AMD Quark 2 references brings MXFP4 quantization to Diffusers 3 references and xDiT FLUX.1-dev 4 references image generation, reaching up to 1.92× speedup over the BF16 eager baseline on a single AMD Instinct™ MI350 GPU while preserving quality. This post covers two new capabilities for diffusion models: SVDQuant 5 references , for accurate low-bit 4-bit quantization, and Quark’s native Hugging Face Diffusers support . Quark now allows quantized diffusion models to be saved and reloaded through the standard Diffusers save pretrained / from pretrained APIs. SVDQuant Support in Quark svdquant-support-in-quark Pushing diffusion models to 4-bit — quantizing not just weights but activations too — is much harder than the FP8 or INT8 case. Both the weights and the activations in diffusion transformers contain outlier channels whose magnitudes dwarf the rest of the tensor. A naive 4-bit grid has to stretch to cover those outliers, which crushes the resolution available for the bulk of the values and shows up as visible artifacts in generated images. SVDQuant addresses this with two ideas working together: Smoothing migrates part of the activation dynamic range into the weights as in SmoothQuant 6 references , so neither side has to absorb the full outlier magnitude alone.A high-precision low-rank correction branch captures the residual outliers that low-bit quantization cannot represent. The weight matrix is decomposed via SVD; at inference, the output is the result of the low-bit residual GEMM plus a small rank-16-to-32 correction. That correction is cheap in compute and memory, yet recovers most of the accuracy lost at 4 bits. This is what makes 4-bit activation quantization viable, not just weight-only compression: the w4a4 , mxfp4 , and nvfp4 modes below quantize activations as well as weights. In Quark, SVDQuant is configured through SVDQuantConfig , with ready-made schemes from build quant layer config : Mode | Weights | Activations | Notes | |---|---|---|---| | INT4 per-group | fp16 / bf16 | weight-only 4-bit | | INT4 per-group | INT4 per-group dynamic | fully 4-bit | | MXFP4 | MXFP4 dynamic | | | FP4 block-16 | FP4 block-16 dynamic | FP4 with FP8 block scales | Native inference. On MI300 / MI350 GPUs, quark.torch.enable native inference runs the low-bit residual GEMM on AMD AITER 10 references matrix-core kernels. For SVDQuant, the low-rank correction runs as a separate branch alongside that GEMM optionally overlapped on a second CUDA stream via RuntimeOptions svdquant overlap streams=True ; it is not yet fused into the GEMM kernel — a fused SVDQuant kernel is planned and will further reduce its latency. This path is part of AMD Quark’s diffusion support. The table below shows Quark quantization of FLUX.1-dev on a gfx950 / MI350 GPU, reported relative to the BF16 baseline: Config FLUX.1-dev | Speedup vs BF16 ↑ | Model mem vs BF16 ↓ | Peak mem vs BF16 ↓ | |---|---|---|---| BF16 | 1.00× | 1.00× | 1.00× | Native FP4 plain RTN | 1.12× | 0.67× | 0.69× | Native SVDQuant | 0.92× | 0.73× | 0.75× | Plain native FP4 runs about 1.12× faster than BF16 while cutting model memory by roughly a third. Native SVDQuant trades a little speed currently ~ 0.92× of BF16, due to the extra correction branch that is not yet fused for the accuracy that makes 4-bit activations viable, while still trimming model memory by ~27%. Once the correction branch is fused into the GEMM, SVDQuant’s latency should move toward the plain FP4 path. The images below are FLUX.1-dev generations produced with Quark SVDQuant across all four low-bit formats, using the same prompt and seed. Quality holds across INT4 and FP4: SVDQuant W4A16 | SVDQuant W4A4 | |---|---| | | Figure 1. FLUX.1-dev generated with Quark SVDQuant across four low-bit formats INT4 weight-only, INT4 W4A4, MXFP4, NVFP4 , same prompt and seed — image quality holds across all four. These samples were produced with examples/torch/diffusers/testSVDQuant.py multi-mode SVDQuant for SDXL / FLUX / SD3 . On FLUX.1-dev, the SVDQuant W4A4, MXFP4, and NVFP4 variants all maintain CLIP scores 11 references within about half a point of the FP16 reference: SVDQuant FLUX.1-dev | Residual rounding | CLIP ↑ | |---|---|---| FP16 reference | — | 27.82 | W4A4 | RTN | 27.50 | W4A4 | GPTQ | 27.40 | MXFP4 | RTN | 27.41 | MXFP4 | GPTQ | 27.36 | NVFP4 | RTN | 27.72 | NVFP4 | GPTQ | 28.01 | CLIP score with openai/clip-vit-large-patch14 on 1,000 MJHQ 12 prompts higher is better . “RTN” and “GPTQ” 13 denote how the 4-bit residual weights are rounded. Diffusers Support diffusers-support A single import quark.integrations.diffusers statement registers Quark as a Hugging Face Diffusers quantizer, so quantized diffusion models behave like any other Diffusers checkpoint. There are two ways to use it. Offline quantization — quantize once, reload anywhere. Quantize a pipeline submodule with ModelQuantizer , save it with export safetensors which routes through save pretrained and embeds the serialized Quark QConfig under quantization config in config.json , and reload later with plain from pretrained . The checkpoint is self-describing: the loader reads the config, rebuilds the quantized layers with meta-device and low cpu mem usage loading supported , loads the weights, and freezes the model for inference — no QConfig needed at the call site. Native online quantization — quantize at load time. Pass quantization config=... to from pretrained against a plain fp16/bf16 checkpoint, and Quark applies weight-only quantization in-process, with no export/reload round-trip. We already offered online quantization through xDiT 1 references — where Quark replaces a transformer’s linear layers with FP8 or MXFP4 implementations at load time and routes them to AITER kernels — and that same online path is now available directly through Diffusers. Availability. Today this is enabled by importing quark.integrations.diffusers , which self-registers monkeypatches the "quark" method into the Diffusers registries at runtime. We also have a PR to add Quark to Diffusers upstream huggingface/diffusers 14077 https://github.com/huggingface/diffusers/pull/14077 ; until it merges, the one-line import is all that’s needed — nothing else in your code changes. Tutorial: Quantize, Save, and Reload a Diffusion Model tutorial-quantize-save-and-reload-a-diffusion-model This tutorial is adapted from the scripts in examples/torch/diffusers quantize diffusers.py , testSVDQuant.py . It uses SDXL 14 references and FLUX.1-dev 4 references as running examples; the same pattern applies to SD1.5 15 references , SD3 16 references , and PixArt 17 references by swapping the pipeline class and the target submodule. Environment environment The setup mirrors the earlier xDiT blog. On an AMD Instinct GPU with a recent ROCm PyTorch image: docker run -it \ --cap-add=SYS PTRACE --security-opt seccomp=unconfined \ --device=/dev/kfd --device=/dev/dri --group-add video \ --ipc=host --network host --shm-size 128G \ -v /shareddata/:/data -w /workspace \ rocm/pytorch-xdit:v26.5 Inside the container, install Diffusers and Quark: pip install diffusers transformers accelerate Install Quark from source public repo, main branch git clone https://github.com/amd/quark.git Quark && cd Quark && pip install -e . Which submodule do you quantize? For SDXL / SD1.5it is pipe.unet ; forFLUX / SD3 / PixArtit is pipe.transformer . Quantization is applied to that submodule, not the VAE or text encoders. 1. Quantize, Save, and Reload through Diffusers quantize-save-and-reload-through-diffusers This is the headline workflow. Weight-only quantization needs no calibration data, so it is the simplest way to see the full round-trip: quantize the target submodule with ModelQuantizer , save it with export safetensors which routes through save pretrained and embeds the Quark QConfig in config.json , and reload it later with plain from pretrained . python import torch from diffusers import DiffusionPipeline from quark.torch import ModelQuantizer, export safetensors from quark.torch.quantization.config.config import Int8PerTensorSpec, QConfig, QLayerConfig pipe = DiffusionPipeline.from pretrained "stabilityai/stable-diffusion-xl-base-1.0", torch dtype=torch.float16, variant="fp16", .to "cuda" INT8 weight-only. dataloader=None is allowed because there are no activation quantizers. weight spec = Int8PerTensorSpec observer method="min max", symmetric=True, scale type="float", round method="half even", is dynamic=False, .to quantization spec qconfig = QConfig global quant config=QLayerConfig weight=weight spec pipe.unet = ModelQuantizer qconfig .quantize model pipe.unet, dataloader=None Save through the standard Diffusers API: writes diffusion pytorch model.safetensors plus a config.json carrying the serialized QConfig under quantization config . export safetensors pipe.unet, "./sdxl-unet-quark-int8" Reloading is a two-liner. Importing the integration self-registers the "quark" method; from pretrained then reads quantization config , reconstructs the quantized layers via process model transformation , loads the checkpoint, and freezes the model for inference — no QConfig needed at the call site: python import quark.integrations.diffusers registers "quark" into the Diffusers registries from diffusers import UNet2DConditionModel unet = UNet2DConditionModel.from pretrained "./sdxl-unet-quark-int8" pipe.unet = unet drop the reloaded, quantized module back into the pipeline image = pipe "A cat on a windowsill", num inference steps=30, guidance scale=8.0 .images 0 image.save "sdxl int8.png" The same config.json mechanism works when the quantized submodule lives inside a full pipeline directory: DiffusionPipeline.from pretrained "