Language-Instructed Vision Embeddings for Controllable and Generalizable Perception

Researchers introduced Language-Instructed Vision Embeddings (LIVE), a method that uses language to dynamically guide vision encoders, eliminating the need for task-specific retraining. LIVE reduced visual hallucinations by 34 points on MMVP and outperformed larger vision-language models on visual question answering, demonstrating controllable and generalizable perception.

arXiv:2606.19584v1 Announce Type: new Abstract: Vision foundation models are typically trained as static feature extractors, placing the burden of task adaptation onto large downstream models. We propose an alternative paradigm: instead of solely feeding visual features into language models, we use language itself to dynamically guide the vision encoder. Our method, Language-Instructed Vision Embeddings LIVE , leverages language as high-level guidance to produce task-centric embeddings at inference time, removing the need for task-specific retraining. This enables the encoder to focus on contextually relevant aspects of the input, yielding more controllable and generalizable representations. Empirically, LIVE reduces visual hallucinations +34 points on MMVP , surpasses vision-language models with orders of magnitude more parameters on visual question answering, and generalizes to unseen instructions and tasks -- offering a direct path toward adaptive, instruction-driven visual intelligence.