{"slug": "smevals-a-small-eval-suite-for-evaluating-models-prompts-and-harnesses", "title": "smevals - a small eval suite for evaluating models, prompts, and harnesses", "summary": "Simon Willison released smevals, a new open-source tool for running small eval suites across different model configurations, grading results, and generating static HTML reports. The tool, available via `uvx smevals`, supports commands like `run`, `grade`, `serve`, and `build`, and is designed to evaluate models, prompts, and harnesses. Willison describes it as his third iteration on eval tooling, which he plans to expand and apply to his own projects.", "body_md": "[smevals - a small eval suite for evaluating models, prompts, and harnesses](https://primeradiant.com/blog/2026/smevals.html)\n\nThe result is ** smevals**, a new tool for running small eval suites across different model configurations and grading the results.\n\nThis blog entry describes the tool in detail. Here's the 10 second version:\n\n`run uvx smevals docs`\n\nto learn the tool (this outputs Once you've created an eval - which takes the form of a directory with some YAML files - you can run it against models like this:\n\n```\nuvx smevals run path-to-eval/ -m gpt-5.5 -m claude-opus-4.6\n```\n\nRuns are treated separately from grading operations - you can grade your runs (against your defined set of checks) using:\n\n```\nuvx smevals grade path-to-eval/\n```\n\nThen you can run a localhost web server to explore the results:\n\n```\nuvx smevals serve path-to-eval/\n```\n\nOr run the `smevals build`\n\ncommand to build that report as static HTML, which you can then host anywhere. Here's [an example](https://static.simonwillison.net/static/2026/smevals-haiku-build/#/haiku) showing an eval suite I built to evaluate how well models can write haikus.\n\nI've been trying to figure out an approach I like for evals for several years now. `smevals`\n\nis my third iteration on the idea and it feels right to me. I'm looking forward to expanding this more in the future, as well as pointing it at some of my own projects.\n\nTags: [projects](https://simonwillison.net/tags/projects), [ai](https://simonwillison.net/tags/ai), [generative-ai](https://simonwillison.net/tags/generative-ai), [llms](https://simonwillison.net/tags/llms), [llm](https://simonwillison.net/tags/llm), [evals](https://simonwillison.net/tags/evals), [jesse-vincent](https://simonwillison.net/tags/jesse-vincent)", "url": "https://wpnews.pro/news/smevals-a-small-eval-suite-for-evaluating-models-prompts-and-harnesses", "canonical_source": "https://simonwillison.net/2026/Jul/31/smevals/#atom-everything", "published_at": "2026-07-31 21:15:23+00:00", "updated_at": "2026-07-31 21:32:08.085916+00:00", "lang": "en", "topics": ["ai-tools", "ai-research", "developer-tools"], "entities": ["Simon Willison", "smevals", "gpt-5.5", "claude-opus-4.6"], "alternates": {"html": "https://wpnews.pro/news/smevals-a-small-eval-suite-for-evaluating-models-prompts-and-harnesses", "markdown": "https://wpnews.pro/news/smevals-a-small-eval-suite-for-evaluating-models-prompts-and-harnesses.md", "text": "https://wpnews.pro/news/smevals-a-small-eval-suite-for-evaluating-models-prompts-and-harnesses.txt", "jsonld": "https://wpnews.pro/news/smevals-a-small-eval-suite-for-evaluating-models-prompts-and-harnesses.jsonld"}}