DSPy's pitch is that you stop hand-writing prompts and let an optimizer compile them for you. You write a program out of modules, hand it a metric and a trainset, run a teleprompter, and it finds good few-shot examples for each step. It genuinely works. The part the tutorials don't put a number on is what "found good few-shot examples" costs you — not once, but on every call your compiled program makes in production.
When you compile with the default optimizer, DSPy bootstraps few-shot demonstrations and pins them onto each predictor:
def __init__(self, ..., max_bootstrapped_demos=4, max_labeled_demos=16, ...):
That's up to 20 demos per predictor by default — 4 bootstrapped (full input→output traces, including the chain-of-thought rationale) plus up to 16 labeled examples. They live on the compiled program, not in any prompt string you wrote.
Then, on every inference, the module hands all of them to the adapter:
demos = kwargs.pop("demos", self.demos)
self.demos
is the full set the optimizer attached. There's no "use them for the first call only" — each forward()
defaults to sending the whole list.
The adapter turns every demo into a pair of chat messages — a user turn and an assistant turn — and appends all of them ahead of your real input:
messages.append({"role": "system", "content": system_message})
messages.extend(self.format_demos(signature, demos))
...
messages.append({"role": "user", "content": self.format_user_message_content(signature, demo)})
messages.append({"role": "assistant", "content": ...})
So a predictor compiled with 12 demos prepends 24 messages to every call. Because bootstrapped demos carry the full reasoning trace, those messages aren't small. This is fixed overhead you pay on request #1 and request #1,000,000 alike — and it's invisible in your code, because you never wrote those messages. The optimizer did.
DSPy's whole point is composition: a pipeline is several modules — a couple of ChainOfThought
steps, a retriever-reader, a router. Each is a predictor, each gets its own demo set, each re-sends it on every call. The per-call prompt overhead is roughly:
demos_per_predictor × predictors × (rationale is long, so each demo is not cheap)
Compile a 4-module pipeline with the defaults and you can be prepending 60–80 demo messages across the pipeline for a single end-user request — none of which appear anywhere in your source.
This isn't a bug and it isn't a strawman: DSPy hands you the lever at compile time. Choose the demo budget instead of inheriting 4 + 16:
from dspy.teleprompt import BootstrapFewShot
optimizer = BootstrapFewShot(metric=my_metric,
max_bootstrapped_demos=2,
max_labeled_demos=2)
compiled = optimizer.compile(program, trainset=trainset)
And check what actually got attached before you ship it:
for p in compiled.predictors():
print(len(p.demos)) # how many few-shot pairs ride along on every call
The point isn't "DSPy is expensive" — it's that the number of demos re-sent per call is a decision the optimizer makes for you, and the default is generous.
Before you tune anything, put a dollar figure on one real run of the compiled program — priced, not guessed. That's what @wartzar-bee/tokenscope does (
npm i @wartzar-bee/tokenscope
): it takes real usage and prices each bucket — input, output, cache-write (~1.25×), cache-read (~0.1×) — into an actual per-run cost, so "the compiled pipeline costs N× the zero-shot one" stops being a hunch.If it runs in CI, gate it: wartzar-bee/ci-guardrail is an Apache-2.0 GitHub Action (built on tokenscope) that fails the check when a run crosses an absolute
max-usd
ceiling — so a recompile that bumps the demo count doesn't ship as a silent 3× before anyone notices.
- uses: wartzar-bee/ci-guardrail@v1
with:
max-usd: "0.50"
If you run compiled DSPy programs: how many demos are on each predictor, and how long is each one? Worth pricing one real run before the next invoice does it for you.