# Run and Compare AI Evaluations with a CLI for Developers and Coding Agents > Source: > Published: 2026-07-28 21:10:55+00:00 **TL;DR:** This walkthrough shows how developers and coding agents can use [Quantiles](https://github.com/quantiles-evals/quantiles), an open-source AI evaluation platform licensed under Apache 2.0, to quickly run, analyze, and compare AI evaluations locally. We'll use the SimpleQA Verified benchmark as an example throughout this post, letting you follow the commands, inspect the evaluation results, and configure your own model for the same workflow. Running an AI evaluation is rarely as simple as sending prompts to a model. Developers must connect datasets, model APIs, scoring logic, result storage, and comparison tooling before they can answer a basic question: did the system get better? When those pieces are spread across scripts, notebooks, and logs, every rerun becomes harder to reproduce and diagnose. A score alone cannot reveal whether the model changed or whether the dataset, prompt, scorer, or sample set changed with it. The Quantiles CLI is called `qt` on the command line. A simple `curl ... | bash` command supports macOS and Linux on X86-64 and Arm64 systems. First, use it to install the CLI: ``` curl -fsSL https://cli.quantiles.io/install.sh | bash ``` If you don't want to run code directly sourced from the internet, see the [first.]`install.sh` source code Next, let’s run a built-in benchmark from start to finish using a single command. [SimpleQA Verified](https://huggingface.co/datasets/google/simpleqa-verified) is a 1,000-prompt benchmark created by Google DeepMind and Google Research. It re-curates questions from OpenAI's [SimpleQA](https://openai.com/index/introducing-simpleqa/) benchmark to reduce problems such as incorrect labels, topical bias, redundant questions, and ambiguous source evidence. Each example includes a short factual question in `problem` , its reference `answer` , topic and answer-type metadata, and supporting URLs. Use the following command to run `simpleqa-verified` using the built-in Quantiles demo model, which doesn't incur any usage charges: ``` qt run simpleqa-verified ``` Results from the demo model are intended only to demonstrate the evaluation workflow because its outputs are generated randomly. See the configuration section to run evaluations with your own model. The current Quantiles integration reads `problem` and `answer` columns from the [quantiles/simpleqa-verified](https://huggingface.co/datasets/quantiles/simpleqa-verified) dataset, which is a fork of the canonical Google dataset. The benchmark remains the work of the original authors. The above `qt run` command embeds each response and reference answer using a local embedding model powered by [fastembed](https://docs.rs/fastembed/latest/fastembed/), and then compares the two using cosine similarity. It then records a sample-level similarity score and aggregates the score distribution. This flow exercises dataset loading, sample execution, scoring, aggregation, and evaluation metadata and metrics storage, without using an LLM provider. It does not reproduce the benchmark's canonical GPT-4.1 autorater, so does not report the resulting similarity metrics as SimpleQA Verified model performance. When you’re ready to evaluate LLMs, refer to the model configuration documentation section below for detailed setup instructions. Quantiles calculates and displays the aggregate metrics after all samples have been evaluated and reports the evaluation name, status, timestamps, duration, workflow input and output, error state, and aggregate metrics. ``` bash $ qt run simpleqa-verified Created run 1 simpleqa-verified: 100%|████████████████████| 1000/1000 [00:18<00:00, 1195.52it/s] Run 1 completed successfully in 19.21s Aggregate metrics METRIC VALUE UNIT max_similarity 0.8351 - mean_similarity 0.5647 - median_similarity 0.5589 - min_similarity 0.4497 - p95_similarity 0.6359 - p99_similarity 0.6734 - stdev_similarity 0.0396 - variance_similarity 0.0015 - Run `qt show 1 --json` for sample-level details. ``` Each completed evaluation reports its `run_id` . If the ID is no longer visible, use the following command to list previous runs and their IDs: ``` qt list ``` Use the following command to display the sample-level details for a specific run. Here, `run_id` 1 refers to the SimpleQA Verified evaluation completed above. ``` qt show 1 --json ``` Use the `--json` flag to retrieve sample-level results and execution details as structured, machine-readable output. The JSON document has three top-level fields: `run` , `metrics` , and `samples` . Depending on the recorded step, a sample might include its step key, status, input hash, timestamps, stored output, error, and associated metrics. Many system changes can affect model behavior, including changes to prompts, datasets, code, infrastructure, and the model itself. Comparing two runs makes it possible to measure and inspect the effects of a change. Run the same `simpleqa-verified` evaluation a second time and retain both run IDs: ``` qt run simpleqa-verified ``` In this example, the two SimpleQA Verified runs have `run_id` values of 1 and 2: ``` qt compare 1 2 Comparing runs 1 and 2 Run 1 Run 2 Delta Eval simpleqa-verified simpleqa-verified SAME Status COMPLETE COMPLETE Duration 23.201s 18.720s -4.481s Model demo-builtin demo-builtin SAME max_similarity 0.7409 0.7618 +0.0209 mean_similarity 0.5634 0.5647 +0.0013 median_similarity 0.56 0.56 -0.000041 min_similarity 0.4501 0.4341 -0.0161 p95_similarity 0.6387 0.6397 +0.001 p99_similarity 0.6779 0.6798 +0.0018 stdev_similarity 0.0409 0.0407 -0.000228 variance_similarity 0.0017 0.0017 -0.000019 Run 'qt compare 1 2 --json' for sample-level details ``` `qt compare` exits with status `1` when any checked input, output, step, or aggregate metric differs, and `0` when the runs match, which can be useful for scripts or CI jobs. It's important to remember, however, that a `1` status means "different," and doesn't necessarily indicate that the model got worse. Even a metric improvement counts as a difference. Make sure to apply your own release policy to the JSON output and the status code. So far, the examples have used the default demo model and the full evaluation dataset. You can instead configure a model from a supported provider, such as OpenAI or Anthropic, and set a sample limit when you do not need to evaluate all 1,000 prompts. Provider-backed evaluations send prompts to an external service and may incur usage charges, so start with a small smoke test. `--input` flag The quickest way to customize a built-in benchmark such as SimpleQA Verified is to use `--input` to override the model or sample limit. These overrides apply only to the current run and the value passed must be a JSON object. To reuse the settings in future SimpleQA Verified runs, add them to `quantiles.toml` configuration file. For example, this command limits SimpleQA Verified to 10 samples while still using the built-in demo model: ``` qt run simpleqa-verified --input '{"limit":10}' ``` To evaluate the same subset with a specific model, such as one from OpenAI, add the model to the JSON object and ensure that `OPENAI_API_KEY` is already set: ``` qt run simpleqa-verified --input '{"limit":10,"model":"openai:gpt-5.6-luna"}' ``` You can use a configuration file when you want the same model and sample settings to apply across repeated runs. To do this, create exactly one of either `quantiles.toml` or `.quantiles.toml` in the directory where you run `qt` (the CLI will throw an error if it finds both). The below `quantiles.toml` configuration runs the first 10 SimpleQA Verified samples with an OpenAI model: ``` [benchmarks.simpleqa-verified] samples = 10 model = "openai:gpt-5.6-luna" ``` Omit the `samples` key from configuration if you intend to run the evaluation over its complete dataset. Quantiles loads the matching benchmark section automatically when you run: ``` qt run simpleqa-verified ``` Quantiles currently supports four model provider prefixes. It forwards the text after the prefix as the provider's model ID, so the available models depend on the provider's current catalog and your account access. The repository examples below show identifiers that work with the current configuration format. | Model source | `model` value | Required environment variables | Repository example | |---|---|---|---| | Built-in demo | Omit `model` , or use `random` | None | Generates random text for workflow validation only | | `openai:` `OPENAI_API_KEY` `anthropic:` `ANTHROPIC_API_KEY` `gemini:` `GEMINI_API_KEY` `cloudflare_ai_gateway:` `CLOUDFLARE_API_KEY` , `CLOUDFLARE_ACCOUNT_ID` , and `CLOUDFLARE_GATEWAY_ID` Cloudflare includes models from families such as OpenAI's gpt-oss, Llama, Mistral, Gemma, DeepSeek, Qwen, and GLM. Use the [Cloudflare model catalog](https://developers.cloudflare.com/workers-ai/models/) to choose an exact model ID because availability and access can change independently of Quantiles. Keep API keys out of `quantiles.toml` , `--input` JSON, and source control. Quantiles reads provider credentials from environment variables, so set the variable listed in the table for the selected provider. For the OpenAI examples below, export the key in the same terminal session where you will run your `qt` commands: ``` export OPENAI_API_KEY="" ``` Credentials are configured the same way whether you pass the model through a configuration file or the `--input` flag. Quantiles provides an [open-source agent skill](https://github.com/quantiles-evals/skill) that gives coding agents, such as Codex or Claude Code, reusable instructions for running evaluations, inspecting sample-level results, comparing runs, and recovering interrupted work with the `qt` CLI. Run the agent from your project root so it can find your Quantiles configuration and local run history. First, ask your coding agent to install the skill: ``` Please install the Quantiles skill at github.com/quantiles-evals/skill ``` Then prompt your agent to run the SimpleQA Verified benchmark with the demo model: ``` Run the simpleqa-verified benchmark and summarize the results. ``` When you are ready to evaluate your model, replace the placeholder in the following prompt and run a small smoke test to verify the model configuration: ``` Run 10 samples of the simpleqa-verified benchmark using . Confirm that the required provider credentials are available without showing their values; if they are not, stop and tell me what is missing. When finished, summarize the results. Do not run the evaluation until I confirm. ``` After you have two comparable runs, ask the agent to analyze the change: ``` Compare the two most recent runs for 'simpleqa-verified'. Summarize the aggregate metrics, sample-level results, failures, and any notable errors. Identify the highest-impact issues to review first, and recommend specific next steps. ``` `qt` CLI uses the default Hugging Face revision, so be sure to record the dataset repository, configuration, split, and revision in your configuration file when strict reproduction matters.The Quantiles stack provides a structured, local-first system for running, analyzing, comparing, and reproducing AI evaluations. You can use the built-in demo model to verify that your workflow is configured correctly, then switch to a model from a supported provider (e.g., OpenAI or Anthropic) with a fixed configuration to evaluate real model behavior consistently. Since running evaluations, analyzing results, and comparing runs all use the same CLI, the entire process can be carried out through your preferred coding agent. If you’d like your benchmark included in Quantiles so others can run it with a single command, submit a request through the [Quantiles issue tracker](https://github.com/quantiles-evals/quantiles/issues).