Run and Compare AI Evaluations with a CLI for Developers and Coding Agents Quantiles, an open-source AI evaluation platform, released a CLI tool called 'qt' that lets developers and coding agents run, analyze, and compare AI evaluations locally. The tool supports built-in benchmarks like SimpleQA Verified and uses local embedding models for scoring without requiring an LLM provider. TL;DR: This walkthrough shows how developers and coding agents can use Quantiles https://github.com/quantiles-evals/quantiles , an open-source AI evaluation platform licensed under Apache 2.0, to quickly run, analyze, and compare AI evaluations locally. We'll use the SimpleQA Verified benchmark as an example throughout this post, letting you follow the commands, inspect the evaluation results, and configure your own model for the same workflow. Running an AI evaluation is rarely as simple as sending prompts to a model. Developers must connect datasets, model APIs, scoring logic, result storage, and comparison tooling before they can answer a basic question: did the system get better? When those pieces are spread across scripts, notebooks, and logs, every rerun becomes harder to reproduce and diagnose. A score alone cannot reveal whether the model changed or whether the dataset, prompt, scorer, or sample set changed with it. The Quantiles CLI is called qt on the command line. A simple curl ... | bash command supports macOS and Linux on X86-64 and Arm64 systems. First, use it to install the CLI: curl -fsSL https://cli.quantiles.io/install.sh | bash If you don't want to run code directly sourced from the internet, see the first. install.sh source code Next, let’s run a built-in benchmark from start to finish using a single command. SimpleQA Verified https://huggingface.co/datasets/google/simpleqa-verified is a 1,000-prompt benchmark created by Google DeepMind and Google Research. It re-curates questions from OpenAI's SimpleQA https://openai.com/index/introducing-simpleqa/ benchmark to reduce problems such as incorrect labels, topical bias, redundant questions, and ambiguous source evidence. Each example includes a short factual question in problem , its reference answer , topic and answer-type metadata, and supporting URLs. Use the following command to run simpleqa-verified using the built-in Quantiles demo model, which doesn't incur any usage charges: qt run simpleqa-verified Results from the demo model are intended only to demonstrate the evaluation workflow because its outputs are generated randomly. See the configuration section to run evaluations with your own model. The current Quantiles integration reads problem and answer columns from the quantiles/simpleqa-verified https://huggingface.co/datasets/quantiles/simpleqa-verified dataset, which is a fork of the canonical Google dataset. The benchmark remains the work of the original authors. The above qt run command embeds each response and reference answer using a local embedding model powered by fastembed https://docs.rs/fastembed/latest/fastembed/ , and then compares the two using cosine similarity. It then records a sample-level similarity score and aggregates the score distribution. This flow exercises dataset loading, sample execution, scoring, aggregation, and evaluation metadata and metrics storage, without using an LLM provider. It does not reproduce the benchmark's canonical GPT-4.1 autorater, so does not report the resulting similarity metrics as SimpleQA Verified model performance. When you’re ready to evaluate LLMs, refer to the model configuration documentation section below for detailed setup instructions. Quantiles calculates and displays the aggregate metrics after all samples have been evaluated and reports the evaluation name, status, timestamps, duration, workflow input and output, error state, and aggregate metrics. bash $ qt run simpleqa-verified Created run 1 simpleqa-verified: 100%|████████████████████| 1000/1000 00:18<00:00, 1195.52it/s Run 1 completed successfully in 19.21s Aggregate metrics METRIC VALUE UNIT max similarity 0.8351 - mean similarity 0.5647 - median similarity 0.5589 - min similarity 0.4497 - p95 similarity 0.6359 - p99 similarity 0.6734 - stdev similarity 0.0396 - variance similarity 0.0015 - Run qt show 1 --json for sample-level details. Each completed evaluation reports its run id . If the ID is no longer visible, use the following command to list previous runs and their IDs: qt list Use the following command to display the sample-level details for a specific run. Here, run id 1 refers to the SimpleQA Verified evaluation completed above. qt show 1 --json Use the --json flag to retrieve sample-level results and execution details as structured, machine-readable output. The JSON document has three top-level fields: run , metrics , and samples . Depending on the recorded step, a sample might include its step key, status, input hash, timestamps, stored output, error, and associated metrics. Many system changes can affect model behavior, including changes to prompts, datasets, code, infrastructure, and the model itself. Comparing two runs makes it possible to measure and inspect the effects of a change. Run the same simpleqa-verified evaluation a second time and retain both run IDs: qt run simpleqa-verified In this example, the two SimpleQA Verified runs have run id values of 1 and 2: qt compare 1 2 Comparing runs 1 and 2 Run 1 Run 2 Delta Eval simpleqa-verified simpleqa-verified SAME Status COMPLETE COMPLETE Duration 23.201s 18.720s -4.481s Model demo-builtin demo-builtin SAME max similarity 0.7409 0.7618 +0.0209 mean similarity 0.5634 0.5647 +0.0013 median similarity 0.56 0.56 -0.000041 min similarity 0.4501 0.4341 -0.0161 p95 similarity 0.6387 0.6397 +0.001 p99 similarity 0.6779 0.6798 +0.0018 stdev similarity 0.0409 0.0407 -0.000228 variance similarity 0.0017 0.0017 -0.000019 Run 'qt compare 1 2 --json' for sample-level details qt compare exits with status 1 when any checked input, output, step, or aggregate metric differs, and 0 when the runs match, which can be useful for scripts or CI jobs. It's important to remember, however, that a 1 status means "different," and doesn't necessarily indicate that the model got worse. Even a metric improvement counts as a difference. Make sure to apply your own release policy to the JSON output and the status code. So far, the examples have used the default demo model and the full evaluation dataset. You can instead configure a model from a supported provider, such as OpenAI or Anthropic, and set a sample limit when you do not need to evaluate all 1,000 prompts. Provider-backed evaluations send prompts to an external service and may incur usage charges, so start with a small smoke test. --input flag The quickest way to customize a built-in benchmark such as SimpleQA Verified is to use --input to override the model or sample limit. These overrides apply only to the current run and the value passed must be a JSON object. To reuse the settings in future SimpleQA Verified runs, add them to quantiles.toml configuration file. For example, this command limits SimpleQA Verified to 10 samples while still using the built-in demo model: qt run simpleqa-verified --input '{"limit":10}' To evaluate the same subset with a specific model, such as one from OpenAI, add the model to the JSON object and ensure that OPENAI API KEY is already set: qt run simpleqa-verified --input '{"limit":10,"model":"openai:gpt-5.6-luna"}' You can use a configuration file when you want the same model and sample settings to apply across repeated runs. To do this, create exactly one of either quantiles.toml or .quantiles.toml in the directory where you run qt the CLI will throw an error if it finds both . The below quantiles.toml configuration runs the first 10 SimpleQA Verified samples with an OpenAI model: benchmarks.simpleqa-verified samples = 10 model = "openai:gpt-5.6-luna" Omit the samples key from configuration if you intend to run the evaluation over its complete dataset. Quantiles loads the matching benchmark section automatically when you run: qt run simpleqa-verified Quantiles currently supports four model provider prefixes. It forwards the text after the prefix as the provider's model ID, so the available models depend on the provider's current catalog and your account access. The repository examples below show identifiers that work with the current configuration format. | Model source | model value | Required environment variables | Repository example | |---|---|---|---| | Built-in demo | Omit model , or use random | None | Generates random text for workflow validation only | | openai: