Cross-modal Steering Vectors Exist Across Both Image and Text Generation

A research study operationalizes the question of whether concepts and mechanisms are unified across modalities in current AI models by testing cross-modal steering, finding vectors that successfully steer generation on tokens of one modality. The work examines whether a single steering vector can influence both image and text generation, addressing whether models implement concepts separately per modality or share them. The finding that cross-modal steering vectors exist across both image and text generation bears on how multimodal models represent concepts internally.

Are concepts and mechanisms unified across modalities in current AI models, or instead implemented separately for each one? We operationalize this broad question of modality unification by studying cross-modal steering: 1 finding vectors that successfully steer generation on tokens of one modality,