JUICE: Unified Multimodal Modeling in Pretrained Representation Spaces

Researchers introduced JUICE (Joint Unified multimodal modeling In Continuous Encoder spaces), a single flow-matching Transformer that models images and text jointly in frozen pretrained encoder spaces, reaching up to 0.88 on GenEval and 73.1 on VQAv2 at 2.1B parameters with no language model initialization and at 224x224 resolution, according to the arXiv paper 2610.28801v1. Generating text as continuous states left image generation intact at 48.41 GenEval versus 47.86 for a text-to-image-only model at equal guidance, though captions lost some exact wording. Coupled sampling from the joint distribution raised joint success on multi-solution mazes from 54.6% with independent sampling to 93.7%, and JUICE solved 99.8% of visual spatial planning maps at its training sizes.

arXiv:2610.28801v1 Announce Type: new Abstract: Most unified multimodal models treat images and text asymmetrically: a pretrained language model generates text, an attached diffusion head generates images, and outputs that contain both are produced in a fixed order, so the first output cannot adapt to the second. We ask whether both modalities can instead be generated by one process that is trained on them jointly from scratch. Pretrained encoder spaces hold both modalities in one continuous form and carry knowledge learned at scale, and recent work finds them effective as generative targets for vision and language. We therefore model the joint distribution directly in these spaces. We introduce JUICE Joint Unified multimodal modeling In Continuous Encoder spaces , a single flow matching Transformer over frozen image and text representations, in which each modality has its own noise level. Which modalities are clean and which are generated then decides the task: image generation, captioning, visual question answering, or joint generation. Controlled pretraining runs test each design choice. At equal guidance, generating text as continuous states leaves image generation intact 48.41 GenEval, against 47.86 for a model trained only for text-to-image generation , while captions lose some exact wording. Training on joint and conditional denoising together beats either alone, and semantic encoder spaces beat VAE latents and raw pixels as targets. At 2.1B parameters, with no language model initialization and at 224x224, JUICE reaches up to 0.88 on GenEval and 73.1 on VQAv2. By sampling both modalities from the joint distribution coupled sampling , it can also draw a route on a map while writing out the matching moves. On mazes with several valid solutions, joint success rises from 54.6% with independent sampling to 93.7%, and no modality has to go first. On visual spatial planning, JUICE solves 99.8% of maps at its training sizes.