Show HN: Jev-align, an OSS library for calibrating Jev to your judgement

Sutro released jev-align, an experimental open-source CLI that calibrates TypeSafe's Jev AI Functions by having users label uncertain examples and then applying GEPA to improve the function. The tool requires Python 3.11 or newer, defaults to the first 1,000 rows of a dataset, and offers a GEPA metric-call budget defaulting to 300, with an optional 20% held-out evaluation set and 5, 10, 15, or 20 training annotations per round. Every label comes from the user, and a higher training score never accepts a proposal automatically, according to Sutro.

jev-align is an experimental CLI from Sutro https://sutro.sh/ for building AI Functions with TypeSafe's Jev https://docs.typesafe.ai/introduction . It finds uncertain examples, asks you to label them, and uses GEPA https://gepa-ai.github.io/gepa/ to improve the function. Use it in your application and keep learning from production examples. jev-align-3.mp4 Requires Python 3.11 or newer. uv tool install jev-align export TYPESAFE API KEY="..." Or use Vercel or Cloudflare below export OPENAI API KEY="..." or ANTHROPIC API KEY / GEMINI API KEY jeva Start the CLI with either jeva or jev-align . Use pip install jev-align if you do not use uv https://docs.astral.sh/uv/ . The guided setup discovers local CSV, Parquet, and JSONL files and includes three ready-to-run examples. Each round: 1. Evaluates the configured dataset and measures uncertainty. 2. Selects ambiguous rows plus a random audit sample for you to label. 3. Uses your accumulated labels and optional rationales to run GEPA. 4. Shows the score, certainty change, and proposed definition diff. 5. Lets you accept, reject, rewind, or resume later. Every label comes from you. A higher training score never accepts a proposal automatically. | Type | Output | |---|---| | Binary | True or False | | Multiclass | Exactly one fixed label | | Multilabel | Zero or more fixed labels | | Score | One level from an ordered rubric | The guided Advanced menu configures: - 5, 10, 15, or 20 training annotations per round. - An optional 20% held-out evaluation set. - GEPA's metric-call budget, which defaults to 300. By default, jev-align uses the first 1,000 rows—or the entire dataset when it is smaller—and lets you concatenate all fields or select specific columns. Everything can also be configured with flags: jeva optimize posts.csv \ --question "Is the post related to aviation?" \ --column title \ --column text \ --pool-size 1000 Use repeated --class "NAME=DESCRIPTION" options for multiclass or multilabel tasks, and repeated --score-level options for scoring tasks. Run jeva optimize --help for the complete flag reference. Jev can run directly through TypeSafe AI, Vercel AI Gateway, or Cloudflare Workers AI. The guided setup detects configured providers and lets you choose. Vercel AI Gateway export AI GATEWAY API KEY="..." jeva optimize data.csv --question "Is this relevant?" --column text \ --backend vercel Cloudflare Workers AI export CLOUDFLARE ACCOUNT ID="..." export CLOUDFLARE API TOKEN="..." jeva optimize data.csv --question "Is this relevant?" --column text \ --backend cloudflare These routes do not require a TYPESAFE API KEY . The chosen provider is saved with the AI Function, so later runtime calls use the same provider. GEPA's reflection model is configured separately. GEPA's reflection model is separate from the JEV model evaluating your data. OpenAI, Anthropic, and Gemini models are detected automatically. Any LiteLLM provider https://docs.litellm.ai/docs/providers —including Fireworks, local vLLM, and other OpenAI-compatible endpoints—can be supplied with --reflection-model provider/model . export HOSTED VLLM API BASE="http://localhost:8000/v1" jeva optimize data.csv --question "Is this relevant?" --column text \ --reflection-model "hosted vllm/Qwen/Qwen3-8B" - Arrow keys and Enter navigate menus. - b returns to the previous label; /back leaves the rationale prompt. - Space toggles choices in multilabel tasks. jeva functions jeva optimize --resume .jev-align/runs/<run-id Load an AI Function in your application and capture useful production examples: python from jev align import AIFunction is aviation = AIFunction.load ".jev-align/runs/<run-id ", capture=True, prediction = is aviation title="Airport expansion", text="A new runway opens next year.", Later, resume the AI Function and label the captured examples. GEPA uses that feedback to propose the next version: jeva functions See AGENTS.md https://github.com/sutro-sh/jev-align/blob/main/AGENTS.md for detailed setup, provider configuration, CLI operation, and development guidance for coding agents. Sutro is not affiliated with TypeSafe AI, the makers of Jev.