{"slug": "using-dspy-to-evaluate-and-improve-datasette-agent-s-sql-system-prompts", "title": "Using DSPy to evaluate and improve Datasette Agent's SQL system prompts", "summary": "Simon Willison used DSPy to evaluate and improve the system prompts for Datasette Agent, an AI tool that executes read-only SQL queries. The research identified that including column names in schema listings could reduce error-retry loops caused by column-name guessing. The work was conducted via Claude Code using Claude Fable 5 and tested with GPT 4.1 mini and nano.", "body_md": "**Research:** [Using DSPy to evaluate and improve Datasette Agent's SQL system prompts](https://github.com/simonw/research/tree/main/dspy-datasette-agent-prompts#readme)\n\nOne of this morning's AIE keynotes covered [dspy](https://github.com/stanfordnlp/dspy), which reminded me I've been meaning to see if it could help me improve the system prompt used by [Datasette Agent](https://agent.datasette.io) - so I fired off an asynchronous research task in Claude Code for web using Claude Fable 5:\n\n`Pip install the latest Datasette alpha and datasette-agent and dspy - then figure out how to use dspy to evaluate and improve the main system prompts used by Datasette Agent for the feature where it can execute read only SQL queries to answer user questions about data.`\n\nFable chose to test using GPT 4.1 mini and nano, and identified several promising looking directions for improvements. I particularly like this one:\n\nThe schema listing gives only table names; the \"don't call describe_table if you already have the information\" advice caused column-name guessing (page_count, o.order_id, first_name) and error-retry loops in baseline traces. Either include column names in the prompt's schema listing or soften that advice.\n\nTags: [ai](https://simonwillison.net/tags/ai), [datasette](https://simonwillison.net/tags/datasette), [generative-ai](https://simonwillison.net/tags/generative-ai), [llms](https://simonwillison.net/tags/llms), [evals](https://simonwillison.net/tags/evals), [dspy](https://simonwillison.net/tags/dspy), [datasette-agent](https://simonwillison.net/tags/datasette-agent), [claude-mythos-fable](https://simonwillison.net/tags/claude-mythos-fable)", "url": "https://wpnews.pro/news/using-dspy-to-evaluate-and-improve-datasette-agent-s-sql-system-prompts", "canonical_source": "https://simonwillison.net/2026/Jul/2/dspy-datasette-agent-prompts/#atom-everything", "published_at": "2026-07-02 18:25:00+00:00", "updated_at": "2026-07-03 21:50:11.551429+00:00", "lang": "en", "topics": ["ai-research", "ai-tools", "large-language-models", "generative-ai", "natural-language-processing"], "entities": ["DSPy", "Datasette Agent", "Simon Willison", "Claude Code", "Claude Fable 5", "GPT 4.1 mini", "GPT 4.1 nano", "Stanford NLP"], "alternates": {"html": "https://wpnews.pro/news/using-dspy-to-evaluate-and-improve-datasette-agent-s-sql-system-prompts", "markdown": "https://wpnews.pro/news/using-dspy-to-evaluate-and-improve-datasette-agent-s-sql-system-prompts.md", "text": "https://wpnews.pro/news/using-dspy-to-evaluate-and-improve-datasette-agent-s-sql-system-prompts.txt", "jsonld": "https://wpnews.pro/news/using-dspy-to-evaluate-and-improve-datasette-agent-s-sql-system-prompts.jsonld"}}