Declare the task, score it against labeled data, and let DSPy's MIPROv2 compile a better prompt.
What you'll build #
A support-ticket classifier where you never hand-write the prompt: you declare the task as a DSPy signature, score it against a small labeled dataset, and let the MIPROv2 optimizer compile an instruction + few-shot demos that measurably beat your baseline.
Prerequisites #
- Python 3.10+ (any OS; commands below assume macOS/Linux — on Windows, activate the venv with
.venv\Scripts\activate
) - DSPy 3.3.0 (verified against the docs for the August 2026 release)
- An Anthropic API key from the Anthropic Console. DSPy talks to providers throughLiteLLM, which is installed automatically as a dependency — no separate SDK needed.
Expect the optimization run to make a few hundred model calls. DSPy caches LM responses on disk by default, so re-runs are nearly free.
1. Install DSPy and configure the model #
python -m venv .venv && source .venv/bin/activate
pip install "dspy==3.3.0"
export ANTHROPIC_API_KEY="sk-ant-..."
Create optimize.py
and start with the LM setup:
import dspy
lm = dspy.LM("anthropic/claude-opus-5", max_tokens=16000)
dspy.configure(lm=lm)
Two deliberate choices here: we don't pass temperature
— DSPy 3.3 omits it unless you set it, and Anthropic's current models (Opus 4.7 and later) reject sampling parameters with a 400. And max_tokens=16000
gives headroom because Claude Opus 5 reasons before answering by default, and max_tokens
caps reasoning plus response together.
2. Declare the task as a signature, not a prompt #
In DSPy you specify what goes in and out; the framework renders the actual prompt. Typed output fields with Literal
constrain the model to your label set:
from typing import Literal
Label = Literal["billing", "bug", "how_to", "feature_request"]
class ClassifyTicket(dspy.Signature):
"""Classify a customer support ticket."""
ticket: str = dspy.InputField()
label: Label = dspy.OutputField()
classify = dspy.Predict(ClassifyTicket)
That docstring is the entire "prompt" you write by hand. MIPROv2 will rewrite it for you in step 4.
3. Build the scored dataset and a metric #
Optimization needs labeled examples and a metric that scores predictions. Twenty-four short tickets are enough to see real movement:
raw = [
("I was charged twice this month", "billing"),
("Refund hasn't shown up after 10 days", "billing"),
("Why did my plan price go up?", "billing"),
("Invoice PDF shows the wrong company name", "billing"),
("Card declined but you still billed me", "billing"),
("Need a receipt for my last payment", "billing"),
("App crashes when I upload a PNG", "bug"),
("Export button does nothing on Safari", "bug"),
("Getting a 500 error on login since today", "bug"),
("Dark mode resets after every restart", "bug"),
("Search returns results from deleted projects", "bug"),
("Notifications arrive twice on Android", "bug"),
("How do I invite a teammate?", "how_to"),
("Where do I change my email address?", "how_to"),
("Can I export my data to CSV?", "how_to"),
("How do I set up SSO with Okta?", "how_to"),
("What's the keyboard shortcut for search?", "how_to"),
("How do I cancel my subscription?", "how_to"),
("Please add a calendar view", "feature_request"),
("Would love webhook support for Zapier", "feature_request"),
("Any plans for an offline mode?", "feature_request"),
("A bulk-edit option would save me hours", "feature_request"),
("Please support markdown in comments", "feature_request"),
("It would be great to pin favorite projects", "feature_request"),
]
data = [dspy.Example(ticket=t, label=l).with_inputs("ticket") for t, l in raw]
train, dev = data[::2], data[1::2] # 12 train, 12 dev, balanced by label
def accuracy(example, pred, trace=None):
return example.label == pred.label
with_inputs("ticket")
tells DSPy which fields are inputs; everything else is treated as the gold label.
4. Score the baseline, then compile with MIPROv2 #
evaluate = dspy.Evaluate(devset=dev, metric=accuracy,
num_threads=8, display_progress=True)
baseline = evaluate(classify)
print(f"Baseline: {baseline.score}")
optimizer = dspy.MIPROv2(metric=accuracy, auto="light", num_threads=8)
optimized = optimizer.compile(classify, trainset=train, valset=dev,
minibatch=False)
print(f"Optimized: {evaluate(optimized).score}")
optimized.save("ticket_classifier.json")
Run it with python optimize.py
. MIPROv2 bootstraps candidate few-shot demos from train
, proposes rewritten instructions, and searches over combinations, scoring each candidate program on valset
with your metric — automated prompt engineering against numbers instead of vibes. auto="light"
bounds the budget; minibatch=False
matters because the default minibatch size (35) exceeds our tiny valset and would error. The saved JSON contains the winning instruction and demos.
5. Verify it works #
The run takes a few minutes. You should see evaluation progress bars and MIPROv2's trial logging, ending with something like:
Average Metric: 8.00 / 12 (66.7%): 100%|██████████| 12/12
Baseline: 66.67
...
===== Trial 7 / 10 =====
Best full score so far! Score: 91.67
...
Optimized: 91.67
Exact numbers vary between runs, but optimized should beat baseline. Confirm the compiled program reloads cleanly by appending:
loaded = dspy.Predict(ClassifyTicket)
loaded.load("ticket_classifier.json")
print(loaded(ticket="You billed my old credit card again"))
dspy.inspect_history(n=1)
inspect_history
prints the last actual prompt sent to Claude — you'll see an instruction you never wrote, plus selected few-shot examples. That's the compiled artifact.
Troubleshooting #
** litellm.AuthenticationError: AnthropicException - authentication_error ... invalid x-api-key** — the key isn't visible to the script. Export
ANTHROPIC_API_KEY
in the same shell you run python
from (or pass api_key=...
to dspy.LM
), and check for a stale key from an old shell profile.** ValueError: Minibatch size cannot exceed the size of the valset** — you removed
minibatch=False
(or shrank the dataset). Either keep minibatch=False
, pass a smaller minibatch_size
to compile()
, or use a valset with 35+ examples.** litellm.BadRequestError: AnthropicException - ... invalid_request_error mentioning temperature** — something is setting sampling parameters, which Anthropic's newest models reject. Remove any
temperature
/top_p
arguments from your dspy.LM(...)
call; on older DSPy versions (pre-3.3, where temperature
defaulted to 0.0
) upgrade with pip install -U dspy
.** AdapterParseError or truncated/missing label field** — the response hit the token cap before the answer, usually because reasoning consumed the budget. Raise
max_tokens
in dspy.LM(...)
.## Next steps
MIPROv2 is the workhorse, but DSPy 3.x's GEPA optimizer often does better on harder tasks by using a strong reflection model to evolve instructions from natural-language feedback — your metric can return dspy.Prediction(score=..., feedback="...")
instead of a bare number. Swap dspy.Predict
for dspy.ChainOfThought
or dspy.ReAct
and recompile — the optimizer works on any module unchanged. For real projects, grow the dataset (50–300 examples), hold out a test split the optimizer never sees, and check the compiled JSON into version control so prompt changes get code review like everything else. The MIPROv2 API reference documents every knob used here, including the medium
and heavy
budgets for larger datasets.
Sources & further reading #
Setting up DSPy— dspy.ai - MIPROv2 API Reference— dspy.ai - dspy.Evaluate API Reference— dspy.ai - dspy.LM API Reference— dspy.ai - GEPA Optimizer Overview— dspy.ai - dspy 3.3.0 on PyPI— pypi.org
Mariana Souza· Senior Editor
Mariana covers the fast-moving world of machine learning and generative AI, with a particular focus on how these technologies are reshaping development workflows. When she isn't stress-testing the latest foundation models, she's usually at a local hackathon.
Discussion 0 #
No comments yet
Be the first to weigh in.