I recommend using the same memory management system ComfyUI uses: GitHub - Comfy-Org/comfy-aimdo: AI Model Demand Off Allocator · GitHub. It does on-demand “paging” of model weights from disk to the GPU. It works with mmap’d weights which is important with multi-GPU inferencing engines in PyTorch as your different per-rank processes share the memory instead of making a bunch of copies in CPU RAM.
I am using this approach in my own image/video-gen software and it works well.
It does have a cost, you do have to wait for the weights to transfer over if they are being loaded in - but this ends up being a small amount of time compared to the compute time and it does attempt to overlap during computation.
That said, you do need to have enough VRAM to keep your activations and the current/next block of the model weights. With H3 Ref2VA generations, if you include ~15 seconds of reference video at a high resolution, you can easily reach 300k-400k tokens to denoise and that will exceed your VRAM budget. 24GB would be fine for smaller reference sizes (or doing FL2VA or T2VA generations).