Optimize LLM Prompts Automatically with DSPy Instead of Hand-Tuning

DSPy 3.3.0's MIPROv2 optimizer can automatically compile a support-ticket classifier prompt that beats a hand-tuned baseline, according to a tutorial by Mariana Souza. The approach uses a DSPy signature, a 24-example labeled dataset, and a metric to guide optimization, with the framework rewriting the instruction and selecting few-shot demos. The tutorial specifies using Anthropic's Claude Opus 5 model via LiteLLM, with max_tokens set to 16000 and no temperature parameter due to Anthropic's rejection of sampling parameters.

Optimize LLM Prompts Automatically with DSPy Instead of Hand-Tuning Declare the task, score it against labeled data, and let DSPy's MIPROv2 compile a better prompt. Mariana Souza https://sourcefeed.dev/u/mariana souza What you'll build A support-ticket classifier where you never hand-write the prompt: you declare the task as a DSPy https://dspy.ai signature, score it against a small labeled dataset, and let the MIPROv2 optimizer compile an instruction + few-shot demos that measurably beat your baseline. Prerequisites - Python 3.10+ any OS; commands below assume macOS/Linux — on Windows, activate the venv with .venv\Scripts\activate - DSPy 3.3.0 verified against the docs for the August 2026 release - An Anthropic API key from the Anthropic Console https://console.anthropic.com . DSPy talks to providers through LiteLLM https://docs.litellm.ai , which is installed automatically as a dependency — no separate SDK needed. Expect the optimization run to make a few hundred model calls. DSPy caches LM responses on disk by default, so re-runs are nearly free. 1. Install DSPy and configure the model python -m venv .venv && source .venv/bin/activate pip install "dspy==3.3.0" export ANTHROPIC API KEY="sk-ant-..." Create optimize.py and start with the LM setup: python import dspy lm = dspy.LM "anthropic/claude-opus-5", max tokens=16000 dspy.configure lm=lm Two deliberate choices here: we don't pass temperature — DSPy 3.3 omits it unless you set it, and Anthropic's current models Opus 4.7 and later reject sampling parameters with a 400. And max tokens=16000 gives headroom because Claude Opus 5 reasons before answering by default, and max tokens caps reasoning plus response together. 2. Declare the task as a signature, not a prompt In DSPy you specify what goes in and out; the framework renders the actual prompt. Typed output fields with Literal constrain the model to your label set: python from typing import Literal Label = Literal "billing", "bug", "how to", "feature request" class ClassifyTicket dspy.Signature : """Classify a customer support ticket.""" ticket: str = dspy.InputField label: Label = dspy.OutputField classify = dspy.Predict ClassifyTicket That docstring is the entire "prompt" you write by hand. MIPROv2 will rewrite it for you in step 4. 3. Build the scored dataset and a metric Optimization needs labeled examples and a metric that scores predictions. Twenty-four short tickets are enough to see real movement: raw = "I was charged twice this month", "billing" , "Refund hasn't shown up after 10 days", "billing" , "Why did my plan price go up?", "billing" , "Invoice PDF shows the wrong company name", "billing" , "Card declined but you still billed me", "billing" , "Need a receipt for my last payment", "billing" , "App crashes when I upload a PNG", "bug" , "Export button does nothing on Safari", "bug" , "Getting a 500 error on login since today", "bug" , "Dark mode resets after every restart", "bug" , "Search returns results from deleted projects", "bug" , "Notifications arrive twice on Android", "bug" , "How do I invite a teammate?", "how to" , "Where do I change my email address?", "how to" , "Can I export my data to CSV?", "how to" , "How do I set up SSO with Okta?", "how to" , "What's the keyboard shortcut for search?", "how to" , "How do I cancel my subscription?", "how to" , "Please add a calendar view", "feature request" , "Would love webhook support for Zapier", "feature request" , "Any plans for an offline mode?", "feature request" , "A bulk-edit option would save me hours", "feature request" , "Please support markdown in comments", "feature request" , "It would be great to pin favorite projects", "feature request" , data = dspy.Example ticket=t, label=l .with inputs "ticket" for t, l in raw train, dev = data ::2 , data 1::2 12 train, 12 dev, balanced by label def accuracy example, pred, trace=None : return example.label == pred.label with inputs "ticket" tells DSPy which fields are inputs; everything else is treated as the gold label. 4. Score the baseline, then compile with MIPROv2 evaluate = dspy.Evaluate devset=dev, metric=accuracy, num threads=8, display progress=True baseline = evaluate classify print f"Baseline: {baseline.score}" optimizer = dspy.MIPROv2 metric=accuracy, auto="light", num threads=8 optimized = optimizer.compile classify, trainset=train, valset=dev, minibatch=False print f"Optimized: {evaluate optimized .score}" optimized.save "ticket classifier.json" Run it with python optimize.py . MIPROv2 bootstraps candidate few-shot demos from train , proposes rewritten instructions, and searches over combinations, scoring each candidate program on valset with your metric — automated prompt engineering against numbers instead of vibes. auto="light" bounds the budget; minibatch=False matters because the default minibatch size 35 exceeds our tiny valset and would error. The saved JSON contains the winning instruction and demos. 5. Verify it works The run takes a few minutes. You should see evaluation progress bars and MIPROv2's trial logging, ending with something like: Average Metric: 8.00 / 12 66.7% : 100%|██████████| 12/12 Baseline: 66.67 ... ===== Trial 7 / 10 ===== Best full score so far Score: 91.67 ... Optimized: 91.67 Exact numbers vary between runs, but optimized should beat baseline. Confirm the compiled program reloads cleanly by appending: loaded = dspy.Predict ClassifyTicket loaded.load "ticket classifier.json" print loaded ticket="You billed my old credit card again" Prediction label='billing' dspy.inspect history n=1 inspect history prints the last actual prompt sent to Claude — you'll see an instruction you never wrote, plus selected few-shot examples. That's the compiled artifact. Troubleshooting litellm.AuthenticationError: AnthropicException - authentication error ... invalid x-api-key — the key isn't visible to the script. Export ANTHROPIC API KEY in the same shell you run python from or pass api key=... to dspy.LM , and check for a stale key from an old shell profile. ValueError: Minibatch size cannot exceed the size of the valset — you removed minibatch=False or shrank the dataset . Either keep minibatch=False , pass a smaller minibatch size to compile , or use a valset with 35+ examples. litellm.BadRequestError: AnthropicException - ... invalid request error mentioning temperature — something is setting sampling parameters, which Anthropic's newest models reject. Remove any temperature / top p arguments from your dspy.LM ... call; on older DSPy versions pre-3.3, where temperature defaulted to 0.0 upgrade with pip install -U dspy . AdapterParseError or truncated/missing label field — the response hit the token cap before the answer, usually because reasoning consumed the budget. Raise max tokens in dspy.LM ... . Next steps MIPROv2 is the workhorse, but DSPy 3.x's GEPA optimizer https://dspy.ai/api/optimizers/GEPA/overview/ often does better on harder tasks by using a strong reflection model to evolve instructions from natural-language feedback — your metric can return dspy.Prediction score=..., feedback="..." instead of a bare number. Swap dspy.Predict for dspy.ChainOfThought or dspy.ReAct and recompile — the optimizer works on any module unchanged. For real projects, grow the dataset 50–300 examples , hold out a test split the optimizer never sees, and check the compiled JSON into version control so prompt changes get code review like everything else. The MIPROv2 API reference https://dspy.ai/api/optimizers/MIPROv2/ documents every knob used here, including the medium and heavy budgets for larger datasets. Sources & further reading - Setting up DSPy https://dspy.ai/getting-started/installation/ — dspy.ai - MIPROv2 API Reference https://dspy.ai/api/optimizers/MIPROv2/ — dspy.ai - dspy.Evaluate API Reference https://dspy.ai/api/evaluation/Evaluate/ — dspy.ai - dspy.LM API Reference https://dspy.ai/api/models/LM/ — dspy.ai - GEPA Optimizer Overview https://dspy.ai/api/optimizers/GEPA/overview/ — dspy.ai - dspy 3.3.0 on PyPI https://pypi.org/project/dspy/ — pypi.org Mariana Souza https://sourcefeed.dev/u/mariana souza · Senior Editor Mariana covers the fast-moving world of machine learning and generative AI, with a particular focus on how these technologies are reshaping development workflows. When she isn't stress-testing the latest foundation models, she's usually at a local hackathon. Discussion 0 No comments yet Be the first to weigh in.