Building Federated Multimodal AI Workflows with NVIDIA FLARE

NVIDIA FLARE, an open-source federated learning framework, enables federated training of vision-language models (VLMs) across distributed sites by coordinating model updates through externalization, tensor streaming, and disk-backed aggregation. The design decisions focus on what model state crosses the network and how it is transferred and aggregated efficiently, with FedUMM, a collaboration between William & Mary and NVIDIA, serving as a concrete example by federating lightweight adapters over a frozen multimodal backbone. FedUMM received an Outstanding Student Paper Award at the FL@FM workshop at TheWebConf 2026 and is supported by the NVIDIA Academic Grant Program.

Modern vision-language models VLMs https://www.nvidia.com/en-us/glossary/vision-language-models/ can support tasks such as visual question answering, captioning, and image-text reasoning. In practice, however, the data needed to adapt these models may be distributed across institutions or organizations that cannot centralize their raw records. Federated learning https://nvflare.readthedocs.io/en/2.4/fl introduction.html provides a way to coordinate training across these data-local sites. For VLMs, the challenge is not only orchestration. Sites may contribute different task or modality mixes, and model updates can be large enough to strain network bandwidth and server memory. This post focuses on two design decisions for federated multimodal AI workflows: what model state should cross the network, and how should it be transferred and aggregated efficiently? It shows how NVIDIA FLARE https://github.com/NVIDIA/NVFlare coordinates federated training across sites and handles large model updates through externalization, tensor streaming, and disk-backed aggregation. Those design questions also apply to unified multimodal models UMMs , which support multiple modalities and tasks within a shared architecture. FedUMM https://arxiv.org/abs/2601.15390 , developed through a collaboration between William & Mary and NVIDIA, provides a concrete example by federating lightweight adapters over a frozen multimodal backbone. FedUMM is supported by the NVIDIA Academic Grant Program https://www.nvidia.com/en-us/industries/higher-education-research/academic-grant-program/ , and received an Outstanding Student Paper Award at the FL@FM workshop https://federated-learning.org/fl@fm-www-2026/ at TheWebConf 2026. Why are VLMs difficult to federate? In a centralized vision-language experiment, images, captions, visual question answering examples, and generation prompts can feed one training pipeline. In a federated setting, those examples are distributed across sites with different data, task mixes, and operating constraints. This creates two engineering problems. First, sites may train on different task or modality mixes, so the workflow must define what each client updates and how those updates are combined. Second, full-model updates can be expensive to serialize, transfer, and hold in server memory. The first decision is therefore what to federate. Some approaches exchange distilled knowledge rather than model weights. Others freeze a pretrained backbone and aggregate only lightweight trainable components. CreamFL https://arxiv.org/abs/2302.08888 illustrates the first approach, while FedCLIP https://arxiv.org/abs/2302.13485 , FedPIA https://arxiv.org/abs/2412.14424 , and FedUMM https://arxiv.org/abs/2601.15390 illustrate the second. When larger updates are required, the system must also support streaming and memory-efficient aggregation. NVIDIA FLARE, an open source, extensible Python SDK and framework for federated learning and collaborative computing , can support both parameter-efficient and full-model communication patterns. Figure 1 shows the general workflow. Sites keep different mixes of images, text, and prompts local, while the server coordinates training and aggregates approved model updates. Large-object externalization, tensor streaming, and disk-backed aggregation help manage larger payloads. Coordinating training across clients Every NVIDIA FLARE job separates global coordination from local execution. The server schedules rounds and aggregates updates, while each client trains or evaluates against its local data. Site-specific preprocessing, prompt construction, and batching remain inside the client. The NVIDIA FLARE Recipe API https://nvflare.readthedocs.io/en/main/user guide/data scientist guide/recipe api.html provides a concise starting point. The FedAvg https://nvflare.readthedocs.io/en/main/apidocs/nvflare.recipe.fedavg.html nvflare.recipe.fedavg.FedAvgRecipe recipe pairs a model with a client training script. The same recipe can be run in simulation or in a real provisioned multi-site deployment. Before implementing the model, define the client update contract: what remains local, what may leave the site, which model components each client may update, and which metrics return to the server. When clients update different model components, the contract should also specify how those component-level updates are combined. Moving and aggregating large model updates efficiently A common baseline method for federated VLM training is fine-tuning and aggregating the whole model parameters. This will lead to large model updates. With many clients joining the federation, they create two distinct memory pressures: serializing and transferring one update, and holding several client updates in memory during aggregation. NVIDIA FLARE supports several features to tackle this challenge. Externalize large objects NVIDIA FLARE can replace large objects in a message with lightweight references and transfer the underlying data separately. This keeps the control message small and supports payloads that exceed the ordinary serialized-message limit. Built-in decomposers cover PyTorch tensors, NumPy arrays, and common FLARE structures; custom decomposers are needed only for application-specific object types. Stream tensors For PyTorch workflows, FLARE Tensor Downloader https://nvflare.readthedocs.io/en/main/programming guide/tensor downloader.html streams tensors incrementally using a pull-based protocol. Only the requested chunk is serialized at a time, reducing peak memory during model distribution. Chunk size can be tuned to balance request overhead against per-chunk memory. TensorFlow workflows use the traditional serialization path. Offload aggregation to disk Streaming reduces memory pressure during transfer, but the server may still need to hold several client updates during aggregation. This causes the server’s peak memory to grow linearly with the number of clients. In NVIDIA FLARE 2.8.0, tensor disk offload https://nvflare.readthedocs.io/en/main/design/tensor disk offload.html writes incoming PyTorch FedAvg updates to temporary safetensors files and loads them as needed, thus preventing the linear growth in CPU memory. These mechanisms complement payload reduction from adapter-based training to enable full-model training, larger adapters, or federated learning with many clients. For tested configuration examples, see the NVIDIA FLARE Recipe API https://nvflare.readthedocs.io/en/main/user guide/data scientist guide/recipe api.html , FLARE Tensor Downloader https://nvflare.readthedocs.io/en/main/programming guide/tensor downloader.html , and tensor disk offload documentation https://nvflare.readthedocs.io/en/main/design/tensor disk offload.html . Federating lightweight adapters over a frozen VLM with FedUMM FedUMM https://arxiv.org/abs/2601.15390 provides a concrete example of minimizing what crosses the network in NVIDIA FLARE. Each simulated client keeps a frozen BLIP backbone https://huggingface.co/BLIP3o and trains LoRA adapters locally. NVIDIA FLARE coordinates the rounds and aggregates only the adapter updates. FedUMM is designed for generality with modality-specific encoders for vision, audio, and text, while its current experiments focus on vision-language. The reported experiments evaluate VQA v2 https://huggingface.co/datasets/HuggingFaceM4/VQAv2 and GenEval https://arxiv.org/abs/2310.11513 under Dirichlet-controlled heterogeneity with up to 16 clients. In an eight-client comparison, adapter-only federation reduced per-client communication from 28.6 GB to 0.094 GB per round and improved VQA v2 by 0.7 points relative to full-model FedAvg. At eight clients, performance remained about 97% of the centralized reference on both benchmarks Figure 2 . The evaluation uses simulated sites, synthetic partitions, and public general-domain benchmarks. It does not establish clinical performance or formal privacy guarantees; it shows that raw training data remains local within the simulated federated workflow. FedUMM reduces the system burdens at its source by exchanging only small LoRA adapters. Not every AI workflow can do that. When clients must send larger updates, tensor streaming reduces memory pressure during transfer; when the server must aggregate updates from many clients, disk-backed aggregation reduces how much data must be held in memory. Checklist for designing federated multimodal AI workflows When designing a federated multimodal workflow, consider both what each client should contribute and how those updates will move through the system. The following checklist summarizes the key decisions for balancing model quality, communication cost, and system memory requirements. Define the update contract: Decide what stays local, what each client sends, and how the updates are combined. Minimize the payload: Exchange lightweight adapters when possible, and full-model updates only when the task requires them. Choose how updates move and aggregate: Use externalization and tensor streaming for large in-memory updates, together with disk offload on server when aggregation several updates would exceed server memory. Evaluate end-to-end: Measure model quality together with communication, runtime, memory use, data heterogeneity, and failures. Get started building federated multimodal AI workflows Start by defining the update contract for your workflow: what remains local, what model state each client may return, and which clients should contribute to each aggregation. Use the NVIDIA FLARE Recipe API https://nvflare.readthedocs.io/en/main/user guide/data scientist guide/recipe api.html to implement the workflow and validate it in simulation at the expected client count. Next, choose the payload-handling mechanism that matches your bottleneck. Use https://nvflare.readthedocs.io/en/2.8.0/programming guide/file streaming.html large-object externalization https://nvflare.readthedocs.io/en/main/programming guide/decomposer for large object.html and FLARE Tensor Downloader https://nvflare.readthedocs.io/en/main/programming guide/tensor downloader.html for large model updates, together with tensor disk offload when server-side aggregation memory becomes a constraint. For a concrete adapter-based example, explore the paper, FedUMM: A General Framework for Federated Learning with Unified Multimodal Models https://arxiv.org/abs/2601.15390 , and its implementation in the NVIDIA FLARE repository https://github.com/NVIDIA/NVFlare/tree/main/research/fedumm . After establishing a working baseline, use Auto-FL https://developer.nvidia.com/blog/accelerating-federated-learning-research-with-ai-agents-and-nvidia-flare-auto-fl/ how to adapt auto-fl to your datasets and tasks to adapt and tune the federated experiment for your own datasets and tasks. To learn more, join us for NVIDIA Flare Day 2026 https://events.nvidia.com/flare-day-2026 , a free online event that explores cutting-edge applications of federated learning across industries.