Member-only story
Introduction #
The widespread adoption of LLMs solidified prompting as the standard interface between users and foundation models. At first, prompts were strictly text-based instructions in natural language used to guide a model’s underlying reasoning. The introduction of multimodal foundation models fundamentally disrupted this paradigm. Visuals like images, documents, and user interfaces are no longer just things models look at, they are now an active part of the instruction itself.
The transition from text-only models to multimodal architectures requires a precise redefinition of how inputs guide model behavior. Visual prompting should not be understood simply as "providing an image to an AI." Every multimodal interaction includes visual input, but not every interaction constitutes visual prompting. Visual prompting is the deliberate design of visual context to steer a model’s attention, constrain its hypothesis space, improve grounding, and shape downstream reasoning. The image functions as an instructional artifact rather than merely an observation. Developers must treat visual inputs with the same structural intentionality as engineered textual prompts.
We can say that visual prompting represents a change in the interaction paradigm, not merely an expansion of the input modality.