Following the information in the pull request docs MiniMax-H3 · Hugging Face based on the 24-32GB consumer card section, I’m trying to load the t2va model on a machine with 96GB RAM and an R9700 32GB.
Even with the 8bit quant of transformer and text_encoder the model is taking up more than 80GB of ram and as soon the the code for enable group offload runs, my machine OOM’s and kills the process.
Is 8bit quant only done during moving tensors from ram to vram rather than on initial load into cpu ram?
I know the docs are talking about a setup for a single 24-32GB card and I have two 32GB so should hopefully be able to optimise further, but right now I’m just trying to get the basic recipe going with a single 32GB card as the docs suggest and failing.
Is there a minimum amount of cpu ram required to load/quant this model for usage with off on a single 32GB gpu? Any other options I can try, or should I look for a 8 or 4 bit diffusers format pre-quant model?