Your compiled DSPy program re-sends up to 20 few-shot demos on every single call

A developer found that DSPy's compiled programs re-send up to 20 few-shot demonstrations on every inference call, adding significant token overhead that is invisible in the source code. The default optimizer attaches 4 bootstrapped and 16 labeled demos per predictor, which are converted into chat messages and prepended to every request. The developer recommends setting explicit demo budgets and using tools like tokenscope to price the real cost.

DSPy's pitch is that you stop hand-writing prompts and let an optimizer compile them for you. You write a program out of modules, hand it a metric and a trainset, run a teleprompter, and it finds good few-shot examples for each step. It genuinely works. The part the tutorials don't put a number on is what "found good few-shot examples" costs you — not once, but on every call your compiled program makes in production. When you compile with the default optimizer, DSPy bootstraps few-shot demonstrations and pins them onto each predictor: python dspy/teleprompt/bootstrap.py def init self, ..., max bootstrapped demos=4, max labeled demos=16, ... : That's up to 20 demos per predictor by default — 4 bootstrapped full input→output traces, including the chain-of-thought rationale plus up to 16 labeled examples. They live on the compiled program, not in any prompt string you wrote. Then, on every inference, the module hands all of them to the adapter: dspy/predict/predict.py demos = kwargs.pop "demos", self.demos self.demos is the full set the optimizer attached. There's no "use them for the first call only" — each forward defaults to sending the whole list. The adapter turns every demo into a pair of chat messages — a user turn and an assistant turn — and appends all of them ahead of your real input: dspy/adapters/base.py — format messages.append {"role": "system", "content": system message} messages.extend self.format demos signature, demos ... format demos , per complete demo: messages.append {"role": "user", "content": self.format user message content signature, demo } messages.append {"role": "assistant", "content": ...} So a predictor compiled with 12 demos prepends 24 messages to every call. Because bootstrapped demos carry the full reasoning trace, those messages aren't small. This is fixed overhead you pay on request 1 and request 1,000,000 alike — and it's invisible in your code, because you never wrote those messages. The optimizer did. DSPy's whole point is composition: a pipeline is several modules — a couple of ChainOfThought steps, a retriever-reader, a router. Each is a predictor, each gets its own demo set, each re-sends it on every call. The per-call prompt overhead is roughly: demos per predictor × predictors × rationale is long, so each demo is not cheap Compile a 4-module pipeline with the defaults and you can be prepending 60–80 demo messages across the pipeline for a single end-user request — none of which appear anywhere in your source. This isn't a bug and it isn't a strawman: DSPy hands you the lever at compile time. Choose the demo budget instead of inheriting 4 + 16: python from dspy.teleprompt import BootstrapFewShot optimizer = BootstrapFewShot metric=my metric, max bootstrapped demos=2, max labeled demos=2 compiled = optimizer.compile program, trainset=trainset And check what actually got attached before you ship it: for p in compiled.predictors : print len p.demos how many few-shot pairs ride along on every call The point isn't "DSPy is expensive" — it's that the number of demos re-sent per call is a decision the optimizer makes for you, and the default is generous. Before you tune anything, put a dollar figure on one real run of the compiled program — priced, not guessed. That's what @wartzar-bee/tokenscope https://www.npmjs.com/package/@wartzar-bee/tokenscope does npm i @wartzar-bee/tokenscope : it takes real usage and prices each bucket — input, output, cache-write ~1.25× , cache-read ~0.1× — into an actual per-run cost, so "the compiled pipeline costs N× the zero-shot one" stops being a hunch.If it runs in CI, gate it: wartzar-bee/ci-guardrail https://github.com/wartzar-bee/ci-guardrail is an Apache-2.0 GitHub Action built on tokenscope that fails the check when a run crosses an absolute max-usd ceiling — so a recompile that bumps the demo count doesn't ship as a silent 3× before anyone notices. - uses: wartzar-bee/ci-guardrail@v1 with: max-usd: "0.50" If you run compiled DSPy programs: how many demos are on each predictor, and how long is each one? Worth pricing one real run before the next invoice does it for you.