{"slug": "run-and-compare-ai-evaluations-with-a-cli-for-developers-and-coding-agents", "title": "Run and Compare AI Evaluations with a CLI for Developers and Coding Agents", "summary": "Quantiles, an open-source AI evaluation platform, released a CLI tool called 'qt' that lets developers and coding agents run, analyze, and compare AI evaluations locally. The tool supports built-in benchmarks like SimpleQA Verified and uses local embedding models for scoring without requiring an LLM provider.", "body_md": "**TL;DR:** This walkthrough shows how developers and coding agents can use [Quantiles](https://github.com/quantiles-evals/quantiles), an open-source AI evaluation platform licensed under Apache 2.0, to quickly run, analyze, and compare AI evaluations locally. We'll use the SimpleQA Verified benchmark as an example throughout this post, letting you follow the commands, inspect the evaluation results, and configure your own model for the same workflow.\n\nRunning an AI evaluation is rarely as simple as sending prompts to a model. Developers must connect datasets, model APIs, scoring logic, result storage, and comparison tooling before they can answer a basic question: did the system get better? When those pieces are spread across scripts, notebooks, and logs, every rerun becomes harder to reproduce and diagnose. A score alone cannot reveal whether the model changed or whether the dataset, prompt, scorer, or sample set changed with it.\n\nThe Quantiles CLI is called `qt`\n\non the command line. A simple `curl ... | bash`\n\ncommand supports macOS and Linux on X86-64 and Arm64 systems. First, use it to install the CLI:\n\n```\ncurl -fsSL https://cli.quantiles.io/install.sh | bash\n```\n\nIf you don't want to run code directly sourced from the internet, see the\n\n[first.]`install.sh`\n\nsource code\n\nNext, let’s run a built-in benchmark from start to finish using a single command. [SimpleQA Verified](https://huggingface.co/datasets/google/simpleqa-verified) is a 1,000-prompt benchmark created by Google DeepMind and Google Research. It re-curates questions from OpenAI's [SimpleQA](https://openai.com/index/introducing-simpleqa/) benchmark to reduce problems such as incorrect labels, topical bias, redundant questions, and ambiguous source evidence. Each example includes a short factual question in `problem`\n\n, its reference `answer`\n\n, topic and answer-type metadata, and supporting URLs.\n\nUse the following command to run `simpleqa-verified`\n\nusing the built-in Quantiles demo model, which doesn't incur any usage charges:\n\n```\nqt run simpleqa-verified\n```\n\nResults from the demo model are intended only to demonstrate the evaluation workflow because its outputs are generated randomly. See the configuration section to run evaluations with your own model.\n\nThe current Quantiles integration reads `problem`\n\nand `answer`\n\ncolumns from the [quantiles/simpleqa-verified](https://huggingface.co/datasets/quantiles/simpleqa-verified) dataset, which is a fork of the canonical Google dataset. The benchmark remains the work of the original authors.\n\nThe above `qt run`\n\ncommand embeds each response and reference answer using a local embedding model powered by [fastembed](https://docs.rs/fastembed/latest/fastembed/), and then compares the two using cosine similarity. It then records a sample-level similarity score and aggregates the score distribution. This flow exercises dataset loading, sample execution, scoring, aggregation, and evaluation metadata and metrics storage, without using an LLM provider. It does not reproduce the benchmark's canonical GPT-4.1 autorater, so does not report the resulting similarity metrics as SimpleQA Verified model performance.\n\nWhen you’re ready to evaluate LLMs, refer to the model configuration documentation section below for detailed setup instructions.\n\nQuantiles calculates and displays the aggregate metrics after all samples have been evaluated and reports the evaluation name, status, timestamps, duration, workflow input and output, error state, and aggregate metrics.\n\n``` bash\n$ qt run simpleqa-verified\n\nCreated run 1\nsimpleqa-verified: 100%|████████████████████| 1000/1000 [00:18<00:00, 1195.52it/s]\nRun 1 completed successfully in 19.21s\n\nAggregate metrics\n METRIC                VALUE     UNIT\n max_similarity        0.8351    -\n mean_similarity       0.5647    -\n median_similarity     0.5589    -\n min_similarity        0.4497    -\n p95_similarity        0.6359    -\n p99_similarity        0.6734    -\n stdev_similarity      0.0396    -\n variance_similarity   0.0015    -\n\nRun `qt show 1 --json` for sample-level details.\n```\n\nEach completed evaluation reports its `run_id`\n\n. If the ID is no longer visible, use the following command to list previous runs and their IDs:\n\n```\nqt list\n```\n\nUse the following command to display the sample-level details for a specific run. Here, `run_id`\n\n1 refers to the SimpleQA Verified evaluation completed above.\n\n```\nqt show 1 --json\n```\n\nUse the `--json`\n\nflag to retrieve sample-level results and execution details as structured, machine-readable output. The JSON document has three top-level fields: `run`\n\n, `metrics`\n\n, and `samples`\n\n. Depending on the recorded step, a sample might include its step key, status, input hash, timestamps, stored output, error, and associated metrics.\n\nMany system changes can affect model behavior, including changes to prompts, datasets, code, infrastructure, and the model itself. Comparing two runs makes it possible to measure and inspect the effects of a change.\n\nRun the same `simpleqa-verified`\n\nevaluation a second time and retain both run IDs:\n\n```\nqt run simpleqa-verified\n```\n\nIn this example, the two SimpleQA Verified runs have `run_id`\n\nvalues of 1 and 2:\n\n```\nqt compare 1 2\nComparing runs 1 and 2\n                       Run 1              Run 2              Delta\nEval                   simpleqa-verified  simpleqa-verified  SAME\nStatus                 COMPLETE           COMPLETE\nDuration               23.201s            18.720s            -4.481s\nModel                  demo-builtin       demo-builtin       SAME\nmax_similarity         0.7409             0.7618             +0.0209\nmean_similarity        0.5634             0.5647             +0.0013\nmedian_similarity      0.56               0.56               -0.000041\nmin_similarity         0.4501             0.4341             -0.0161\np95_similarity         0.6387             0.6397             +0.001\np99_similarity         0.6779             0.6798             +0.0018\nstdev_similarity       0.0409             0.0407             -0.000228\nvariance_similarity    0.0017             0.0017             -0.000019\n\nRun 'qt compare 1 2 --json' for sample-level details\n```\n\n`qt compare`\n\nexits with status `1`\n\nwhen any checked input, output, step, or aggregate metric differs, and `0`\n\nwhen the runs match, which can be useful for scripts or CI jobs. It's important to remember, however, that a `1`\n\nstatus means \"different,\" and doesn't necessarily indicate that the model got worse. Even a metric improvement counts as a difference. Make sure to apply your own release policy to the JSON output and the status code.\n\nSo far, the examples have used the default demo model and the full evaluation dataset. You can instead configure a model from a supported provider, such as OpenAI or Anthropic, and set a sample limit when you do not need to evaluate all 1,000 prompts. Provider-backed evaluations send prompts to an external service and may incur usage charges, so start with a small smoke test.\n\n`--input`\n\nflag\nThe quickest way to customize a built-in benchmark such as SimpleQA Verified is to use `--input`\n\nto override the model or sample limit. These overrides apply only to the current run and the value passed must be a JSON object. To reuse the settings in future SimpleQA Verified runs, add them to `quantiles.toml`\n\nconfiguration file.\n\nFor example, this command limits SimpleQA Verified to 10 samples while still using the built-in demo model:\n\n```\nqt run simpleqa-verified --input '{\"limit\":10}'\n```\n\nTo evaluate the same subset with a specific model, such as one from OpenAI, add the model to the JSON object and ensure that `OPENAI_API_KEY`\n\nis already set:\n\n```\nqt run simpleqa-verified --input '{\"limit\":10,\"model\":\"openai:gpt-5.6-luna\"}'\n```\n\nYou can use a configuration file when you want the same model and sample settings to apply across repeated runs. To do this, create exactly one of either `quantiles.toml`\n\nor `.quantiles.toml`\n\nin the directory where you run `qt`\n\n(the CLI will throw an error if it finds both).\n\nThe below `quantiles.toml`\n\nconfiguration runs the first 10 SimpleQA Verified samples with an OpenAI model:\n\n```\n[benchmarks.simpleqa-verified]\nsamples = 10\nmodel = \"openai:gpt-5.6-luna\"\n```\n\nOmit the\n\n`samples`\n\nkey from configuration if you intend to run the evaluation over its complete dataset.\n\nQuantiles loads the matching benchmark section automatically when you run:\n\n```\nqt run simpleqa-verified\n```\n\nQuantiles currently supports four model provider prefixes. It forwards the text after the prefix as the provider's model ID, so the available models depend on the provider's current catalog and your account access. The repository examples below show identifiers that work with the current configuration format.\n\n| Model source |\n`model` value |\nRequired environment variables | Repository example |\n|---|---|---|---|\n| Built-in demo | Omit `model` , or use `random`\n|\nNone | Generates random text for workflow validation only |\n|\n\n`openai:<model-id>`\n\n`OPENAI_API_KEY`\n\n`anthropic:<model-id>`\n\n`ANTHROPIC_API_KEY`\n\n`gemini:<model-id>`\n\n`GEMINI_API_KEY`\n\n`cloudflare_ai_gateway:<model-id>`\n\n`CLOUDFLARE_API_KEY`\n\n, `CLOUDFLARE_ACCOUNT_ID`\n\n, and `CLOUDFLARE_GATEWAY_ID`\n\nCloudflare includes models from families such as OpenAI's gpt-oss, Llama, Mistral, Gemma, DeepSeek, Qwen, and GLM. Use the [Cloudflare model catalog](https://developers.cloudflare.com/workers-ai/models/) to choose an exact model ID because availability and access can change independently of Quantiles.\n\nKeep API keys out of `quantiles.toml`\n\n, `--input`\n\nJSON, and source control. Quantiles reads provider credentials from environment variables, so set the variable listed in the table for the selected provider.\n\nFor the OpenAI examples below, export the key in the same terminal session where you will run your `qt`\n\ncommands:\n\n```\nexport OPENAI_API_KEY=\"<your-openai-api-key>\"\n```\n\nCredentials are configured the same way whether you pass the model through a configuration file or the\n\n`--input`\n\nflag.\n\nQuantiles provides an [open-source agent skill](https://github.com/quantiles-evals/skill) that gives coding agents, such as Codex or Claude Code, reusable instructions for running evaluations, inspecting sample-level results, comparing runs, and recovering interrupted work with the `qt`\n\nCLI. Run the agent from your project root so it can find your Quantiles configuration and local run history.\n\nFirst, ask your coding agent to install the skill:\n\n```\nPlease install the Quantiles skill at github.com/quantiles-evals/skill\n```\n\nThen prompt your agent to run the SimpleQA Verified benchmark with the demo model:\n\n```\nRun the simpleqa-verified benchmark and summarize the results.\n```\n\nWhen you are ready to evaluate your model, replace the placeholder in the following prompt and run a small smoke test to verify the model configuration:\n\n```\nRun 10 samples of the simpleqa-verified benchmark using <model provider and model_id>. Confirm that the required provider credentials are available without showing their values; if they are not, stop and tell me what is missing. When finished, summarize the results. Do not run the evaluation until I confirm.\n```\n\nAfter you have two comparable runs, ask the agent to analyze the change:\n\n```\nCompare the two most recent runs for 'simpleqa-verified'. Summarize the aggregate metrics, sample-level results, failures, and any notable errors. Identify the highest-impact issues to review first, and recommend specific next steps.\n```\n\n`qt`\n\nCLI uses the default Hugging Face revision, so be sure to record the dataset repository, configuration, split, and revision in your configuration file when strict reproduction matters.The Quantiles stack provides a structured, local-first system for running, analyzing, comparing, and reproducing AI evaluations. You can use the built-in demo model to verify that your workflow is configured correctly, then switch to a model from a supported provider (e.g., OpenAI or Anthropic) with a fixed configuration to evaluate real model behavior consistently. Since running evaluations, analyzing results, and comparing runs all use the same CLI, the entire process can be carried out through your preferred coding agent.\n\nIf you’d like your benchmark included in Quantiles so others can run it with a single command, submit a request through the [Quantiles issue tracker](https://github.com/quantiles-evals/quantiles/issues).", "url": "https://wpnews.pro/news/run-and-compare-ai-evaluations-with-a-cli-for-developers-and-coding-agents", "canonical_source": "https://dev.to/quantiles-io/run-and-compare-ai-evaluations-with-a-cli-for-developers-and-coding-agents-385m", "published_at": "2026-07-28 21:10:55+00:00", "updated_at": "2026-07-28 21:35:44.680959+00:00", "lang": "en", "topics": ["developer-tools", "ai-tools", "machine-learning", "artificial-intelligence"], "entities": ["Quantiles", "Google DeepMind", "Google Research", "OpenAI", "SimpleQA Verified", "fastembed"], "alternates": {"html": "https://wpnews.pro/news/run-and-compare-ai-evaluations-with-a-cli-for-developers-and-coding-agents", "markdown": "https://wpnews.pro/news/run-and-compare-ai-evaluations-with-a-cli-for-developers-and-coding-agents.md", "text": "https://wpnews.pro/news/run-and-compare-ai-evaluations-with-a-cli-for-developers-and-coding-agents.txt", "jsonld": "https://wpnews.pro/news/run-and-compare-ai-evaluations-with-a-cli-for-developers-and-coding-agents.jsonld"}}