The Problem that Revision Prompting solves #
We prompt LLMs in two ways:
- Ad-hoc prompting
Prompts LLMs manually, with a custom instruction per call.
Examples #
-
Asking a coding agent to implement a new feature.
-
Asking a chatbot to draft an email.
-
Industrial prompting
Prompts LLMs as part of an automated process, with the same instruction across calls.
Examples #
- Extracting structured information from invoices as part of an accounting pipeline.
- Translating documentation pages into other languages as part of a release process.
Industrial prompting typically processes some Input data with an Instruction to produce some Output.
Whenever the Input gets updated, industrial prompting naively re-runs the Instruction on the UpdatedInput to produce the UpdatedOutput.
This approach has two downsides:
- Lack of consistency
- LLMs are non-deterministic, so the
UpdatedOutputdiffers from the originalOutputbeyond what theUpdatedInputnecessitates. - Full processing time and token costs - Although only parts of the input have changed, we produce the
UpdatedOutputin full. This is as expensive as the production of the originalOutput.
Revision prompting resolves both downsides of naive re-runs by operating on the input and output revisions instead of the full input and output.
How Revision Prompting works #
Assume you have processed some Input with an Instruction to produce some Output by prompting an LLM with
Instruction: Input
Now, Input has been updated, and you also want to process the UpdatedInput.
Revision Prompting processes the UpdatedInput by constructing the RevisionPrompt as
Instruction: Input produced Output.
The input got updated as follows: diff(
.Input, UpdatedInput)
Please produce a patch to update the output.
The LLM responds to the RevisionPrompt with the OutputPatch that we apply to the Output to obtain the UpdatedOutput.
Example #
You translate the product page of an e-bike to German with the prompt
The Vela 3 e-bike has a range of 80 km.
Its battery recharges in three hours.
The frame is made from recycled aluminium.
Every Vela 3 includes a two-year warranty.
and the LLM produces
Das E-Bike Vela 3 hat eine Reichweite von 80 km.
Sein Akku lädt in drei Stunden auf.
Der Rahmen besteht aus recyceltem Aluminium.
Jedes Vela 3 hat zwei Jahre Garantie.
Later, a battery upgrade increases the range from 80 km to 100 km. Instead of re-translating the whole page, you prompt
The Vela 3 e-bike has a range of 80 km.
Its battery recharges in three hours.
The frame is made from recycled aluminium.
Every Vela 3 includes a two-year warranty.
produced
OutputDas E-Bike Vela 3 hat eine Reichweite von 80 km.
Sein Akku lädt in drei Stunden auf.
Der Rahmen besteht aus recyceltem Aluminium.
Jedes Vela 3 hat zwei Jahre Garantie.
The input got updated as follows: diff(Input, UpdatedInput)- The Vela 3 e-bike has a range of 80 km.+ The Vela 3 e-bike has a range of 100 km. Please produce a patch to update the output. The LLM responds with
Applying the OutputPatch to the original Output produces the updated translation.
The OutputPatch contains only two lines of text instead of a full re-translation. Unchanged content stays consistent with the original translation.
Why Revision Prompting works #
- Consistency
- By supplying the LLM with the input revision
diff(
, we ensure that theInput, UpdatedInput) OutputPatchis limited to what the input changes necessitate. Everything not touched by theOutputPatch remains identical to the originalOutput. Therefore, the UpdatedOutputisconsistent with the original Output. - Time & cost savings
- Revision Prompting feeds the original
Outputback in as part of the prompt, so the LLM only generates the shortOutputPatch. Most tokens thereby move from the output to the input. Since processing time scales roughly with the length of the output, this eliminates most of the processing time. It also converts most of the output token cost into much cheaper input token cost. If the re-run happens within a couple of minutes of the original run, prompt caching reduces part of the input token cost as well.
Revision Prompting in practice #
- Revision Formats
- The ideal formats for encoding
diff(
and theInput, UpdatedInput) OutputPatchdepend on the Instruction. ThePOSIXis a useful generic format. For JSON outputs, thediff
utilityJSON Patchformat works well. - Expected Savings
- The time & cost reduction scales with the size of the input changes and the sensitivity of the output to changes in the input. In our own industrial prompts, Revision Prompting reduces time by ~80%, and costs by ~65%.