AI video tools change quickly. A prompt that works well in one model may behave differently in another, and model-specific syntax can become obsolete after an update. A more durable approach is to represent the creative intent first, then translate that representation into the vocabulary of a chosen generator.
This article describes a model-agnostic prompt architecture for reference-based video workflows.
Instead of sending a long natural-language paragraph directly to a model, store the shot as structured fields:
{
"duration": 5,
"aspect_ratio": "9:16",
"subject": "a cyclist wearing a yellow rain jacket",
"environment": "wet city street at dusk",
"composition": "medium tracking shot, subject in right third",
"camera": "truck left at cycling speed, stable horizon",
"action": "cyclist passes two parked cars and looks over shoulder",
"lighting": "cool ambient light with warm shop reflections",
"end_state": "cyclist exits frame left"
}
This is not a prompt yet. It is an intermediate representation: a stable description of what the shot must accomplish.
The advantages are practical:
Not every detail has equal importance. Divide the representation into two groups.
Invariants are requirements that should survive every model translation. Examples include the subject, action order, camera direction, duration, and final state.
Preferences are desirable but negotiable details: a subtle film grain, a specific color mood, shallow depth of field, or a minor background prop.
When generation fails, preserve invariants first. A visually beautiful result is still wrong if the subject moves in the opposite direction or never completes the required action.
P0: action order, camera direction, end state
P1: framing, environment, lighting
P2: texture, secondary props, stylistic accents
A model adapter can transform the intermediate representation in two stages.
The first stage describes the opening frame: subject, wardrobe, environment, framing, lens cues, lighting, and spatial relationships. It should be possible to sketch the first frame from this section alone.
The second stage describes motion: what the subject does, how the camera moves, when events occur, and what the final frame should contain.
Keeping these stages separate makes debugging easier. If the first frame is wrong, revise the visual state. If the action stalls or the camera drifts, revise the temporal instructions.
Each model adapter should map the same fields into the model’s preferred style. One generator may respond well to concise cinematic prose. Another may need explicit timing beats. A third may benefit from negative constraints.
The adapter should be responsible for ordering fields by importance, converting durations into supported values, translating camera vocabulary, omitting unsupported controls, and adding only constraints the target model understands.
The source representation remains unchanged. This prevents model quirks from leaking into the analysis layer.
A lightweight validator catches many expensive errors:
- duration is supported by the target model
- aspect ratio is valid
- subject and action are present
- camera instruction has direction or lock behavior
- action has an observable end state
- timing beats do not exceed total duration
For reference-video workflows, validation should also compare every field against the source. Do not invent wardrobe, dialogue, or camera movement just to make the description sound richer.
Pixel similarity is a poor measure for generative video. Evaluate structure instead:
These checks create useful revision signals. If camera direction is wrong, change the camera layer. If the action is incomplete, simplify the action or allocate more time. If identity drifts, strengthen the visual-state constraints.
Store the intermediate representation, rendered prompt, target model, generation settings, and revision notes together. This produces an audit trail and makes successful prompts reusable.
source segment -> structured shot -> model adapter -> rendered prompt
-> generation result -> evaluation -> revision
We use this separation between analysis and model-specific rendering in promptvv, where reference videos can be decomposed into shot-level tasks before prompts are adapted and refined.
The main idea is simple: treat prompts as compiled output, not as the source of truth. A stable intermediate representation makes an AI video workflow easier to test, debug, and migrate when models change.