{"slug": "your-compiled-dspy-program-re-sends-up-to-20-few-shot-demos-on-every-single-call", "title": "Your compiled DSPy program re-sends up to 20 few-shot demos on every single call", "summary": "A developer found that DSPy's compiled programs re-send up to 20 few-shot demonstrations on every inference call, adding significant token overhead that is invisible in the source code. The default optimizer attaches 4 bootstrapped and 16 labeled demos per predictor, which are converted into chat messages and prepended to every request. The developer recommends setting explicit demo budgets and using tools like tokenscope to price the real cost.", "body_md": "DSPy's pitch is that you stop hand-writing prompts and let an optimizer *compile* them for you. You write a program out of modules, hand it a metric and a trainset, run a teleprompter, and it finds good few-shot examples for each step. It genuinely works. The part the tutorials don't put a number on is what \"found good few-shot examples\" costs you — not once, but on every call your compiled program makes in production.\n\nWhen you compile with the default optimizer, DSPy bootstraps few-shot demonstrations and pins them onto each predictor:\n\n``` python\n# dspy/teleprompt/bootstrap.py\ndef __init__(self, ..., max_bootstrapped_demos=4, max_labeled_demos=16, ...):\n```\n\nThat's **up to 20 demos per predictor** by default — 4 bootstrapped (full input→output traces, including the chain-of-thought rationale) plus up to 16 labeled examples. They live on the compiled program, not in any prompt string you wrote.\n\nThen, on every inference, the module hands all of them to the adapter:\n\n```\n# dspy/predict/predict.py\ndemos = kwargs.pop(\"demos\", self.demos)\n```\n\n`self.demos`\n\nis the full set the optimizer attached. There's no \"use them for the first call only\" — each `forward()`\n\ndefaults to sending the whole list.\n\nThe adapter turns every demo into a **pair of chat messages** — a user turn and an assistant turn — and appends all of them ahead of your real input:\n\n```\n# dspy/adapters/base.py — format()\nmessages.append({\"role\": \"system\", \"content\": system_message})\nmessages.extend(self.format_demos(signature, demos))\n...\n# format_demos(), per complete demo:\nmessages.append({\"role\": \"user\", \"content\": self.format_user_message_content(signature, demo)})\nmessages.append({\"role\": \"assistant\", \"content\": ...})\n```\n\nSo a predictor compiled with 12 demos prepends 24 messages to *every* call. Because bootstrapped demos carry the full reasoning trace, those messages aren't small. This is fixed overhead you pay on request #1 and request #1,000,000 alike — and it's invisible in your code, because you never wrote those messages. The optimizer did.\n\nDSPy's whole point is composition: a pipeline is several modules — a couple of `ChainOfThought`\n\nsteps, a retriever-reader, a router. Each is a predictor, each gets its own demo set, each re-sends it on every call. The per-call prompt overhead is roughly:\n\n```\ndemos_per_predictor  ×  predictors  ×  (rationale is long, so each demo is not cheap)\n```\n\nCompile a 4-module pipeline with the defaults and you can be prepending 60–80 demo messages across the pipeline for a single end-user request — none of which appear anywhere in your source.\n\nThis isn't a bug and it isn't a strawman: DSPy hands you the lever at compile time. Choose the demo budget instead of inheriting 4 + 16:\n\n``` python\nfrom dspy.teleprompt import BootstrapFewShot\n\noptimizer = BootstrapFewShot(metric=my_metric,\n                             max_bootstrapped_demos=2,\n                             max_labeled_demos=2)\ncompiled = optimizer.compile(program, trainset=trainset)\n```\n\nAnd check what actually got attached before you ship it:\n\n```\nfor p in compiled.predictors():\n    print(len(p.demos))   # how many few-shot pairs ride along on every call\n```\n\nThe point isn't \"DSPy is expensive\" — it's that the *number of demos re-sent per call* is a decision the optimizer makes for you, and the default is generous.\n\nBefore you tune anything, put a dollar figure on one real run of the *compiled* program — priced, not guessed. That's what [ @wartzar-bee/tokenscope](https://www.npmjs.com/package/@wartzar-bee/tokenscope) does (\n\n`npm i @wartzar-bee/tokenscope`\n\n): it takes real usage and prices each bucket — input, output, cache-write (~1.25×), cache-read (~0.1×) — into an actual per-run cost, so \"the compiled pipeline costs N× the zero-shot one\" stops being a hunch.If it runs in CI, gate it: [ wartzar-bee/ci-guardrail](https://github.com/wartzar-bee/ci-guardrail) is an Apache-2.0 GitHub Action (built on tokenscope) that fails the check when a run crosses an absolute\n\n`max-usd`\n\nceiling — so a recompile that bumps the demo count doesn't ship as a silent 3× before anyone notices.\n\n```\n- uses: wartzar-bee/ci-guardrail@v1\n  with:\n    max-usd: \"0.50\"\n```\n\nIf you run compiled DSPy programs: how many demos are on each predictor, and how long is each one? Worth pricing one real run before the next invoice does it for you.", "url": "https://wpnews.pro/news/your-compiled-dspy-program-re-sends-up-to-20-few-shot-demos-on-every-single-call", "canonical_source": "https://dev.to/wartzarbee/your-compiled-dspy-program-re-sends-up-to-20-few-shot-demos-on-every-single-call-3a74", "published_at": "2026-08-21 07:17:22+00:00", "updated_at": "2026-08-21 07:44:03.301513+00:00", "lang": "en", "topics": ["machine-learning", "developer-tools"], "entities": ["DSPy", "BootstrapFewShot", "tokenscope", "ci-guardrail"], "alternates": {"html": "https://wpnews.pro/news/your-compiled-dspy-program-re-sends-up-to-20-few-shot-demos-on-every-single-call", "markdown": "https://wpnews.pro/news/your-compiled-dspy-program-re-sends-up-to-20-few-shot-demos-on-every-single-call.md", "text": "https://wpnews.pro/news/your-compiled-dspy-program-re-sends-up-to-20-few-shot-demos-on-every-single-call.txt", "jsonld": "https://wpnews.pro/news/your-compiled-dspy-program-re-sends-up-to-20-few-shot-demos-on-every-single-call.jsonld"}}