{"slug": "show-hn-jev-align-an-oss-library-for-calibrating-jev-to-your-judgement", "title": "Show HN: Jev-align, an OSS library for calibrating Jev to your judgement", "summary": "Sutro released jev-align, an experimental open-source CLI that calibrates TypeSafe's Jev AI Functions by having users label uncertain examples and then applying GEPA to improve the function. The tool requires Python 3.11 or newer, defaults to the first 1,000 rows of a dataset, and offers a GEPA metric-call budget defaulting to 300, with an optional 20% held-out evaluation set and 5, 10, 15, or 20 training annotations per round. Every label comes from the user, and a higher training score never accepts a proposal automatically, according to Sutro.", "body_md": "`jev-align` is an experimental CLI from [Sutro](https://sutro.sh/) for building\nAI Functions with [TypeSafe's Jev](https://docs.typesafe.ai/introduction).\n\nIt finds uncertain examples, asks you to label them, and uses\n[GEPA](https://gepa-ai.github.io/gepa/) to improve the function. Use it in your\napplication and keep learning from production examples.\n\n## jev-align-3.mp4\n\nRequires Python 3.11 or newer.\n\n```\nuv tool install jev-align\nexport TYPESAFE_API_KEY=\"...\" # Or use Vercel or Cloudflare below\nexport OPENAI_API_KEY=\"...\" # or ANTHROPIC_API_KEY / GEMINI_API_KEY\njeva\n```\n\nStart the CLI with either `jeva` or `jev-align`.\n\nUse `pip install jev-align` if you do not use\n[uv](https://docs.astral.sh/uv/). The guided setup discovers local CSV,\nParquet, and JSONL files and includes three ready-to-run examples.\n\nEach round:\n\n1. Evaluates the configured dataset and measures uncertainty.\n2. Selects ambiguous rows plus a random audit sample for you to label.\n3. Uses your accumulated labels and optional rationales to run GEPA.\n4. Shows the score, certainty change, and proposed definition diff.\n5. Lets you accept, reject, rewind, or resume later.\n\nEvery label comes from you. A higher training score never accepts a proposal automatically.\n\n| Type | Output | \n|---|---|\n| Binary | `True` or`False` | \n| Multiclass | Exactly one fixed label | \n| Multilabel | Zero or more fixed labels | \n| Score | One level from an ordered rubric | \n\nThe guided **Advanced** menu configures:\n\n- 5, 10, 15, or 20 training annotations per round.\n- An optional 20% held-out evaluation set.\n- GEPA's metric-call budget, which defaults to 300.\n\nBy default, `jev-align` uses the first 1,000 rows—or the entire dataset when it\nis smaller—and lets you concatenate all fields or select specific columns.\n\nEverything can also be configured with flags:\n\n```\njeva optimize posts.csv \\\n  --question \"Is the post related to aviation?\" \\\n  --column title \\\n  --column text \\\n  --pool-size 1000\n```\n\nUse repeated `--class \"NAME=DESCRIPTION\"` options for multiclass or multilabel\ntasks, and repeated `--score-level` options for scoring tasks. Run\n`jeva optimize --help` for the complete flag reference.\n\nJev can run directly through TypeSafe AI, Vercel AI Gateway, or Cloudflare Workers AI. The guided setup detects configured providers and lets you choose.\n\n```\n# Vercel AI Gateway\nexport AI_GATEWAY_API_KEY=\"...\"\njeva optimize data.csv --question \"Is this relevant?\" --column text \\\n  --backend vercel\n\n# Cloudflare Workers AI\nexport CLOUDFLARE_ACCOUNT_ID=\"...\"\nexport CLOUDFLARE_API_TOKEN=\"...\"\njeva optimize data.csv --question \"Is this relevant?\" --column text \\\n  --backend cloudflare\n```\n\nThese routes do not require a `TYPESAFE_API_KEY`. The chosen provider is saved\nwith the AI Function, so later runtime calls use the same provider. GEPA's\nreflection model is configured separately.\n\nGEPA's reflection model is separate from the JEV model evaluating your data.\nOpenAI, Anthropic, and Gemini models are detected automatically. Any\n[LiteLLM provider](https://docs.litellm.ai/docs/providers)—including Fireworks,\nlocal vLLM, and other OpenAI-compatible endpoints—can be supplied with\n`--reflection-model provider/model`.\n\n```\nexport HOSTED_VLLM_API_BASE=\"http://localhost:8000/v1\"\njeva optimize data.csv --question \"Is this relevant?\" --column text \\\n  --reflection-model \"hosted_vllm/Qwen/Qwen3-8B\"\n```\n\n- Arrow keys and Enter navigate menus.\n- `b` returns to the previous label;`/back` leaves the rationale prompt.\n- Space toggles choices in multilabel tasks.\n\n```\njeva functions\njeva optimize --resume .jev-align/runs/<run-id>\n```\n\nLoad an AI Function in your application and capture useful production examples:\n\n``` python\nfrom jev_align import AIFunction\n\nis_aviation = AIFunction.load(\n    \".jev-align/runs/<run-id>\",\n    capture=True,\n)\n\nprediction = is_aviation(\n    title=\"Airport expansion\",\n    text=\"A new runway opens next year.\",\n)\n```\n\nLater, resume the AI Function and label the captured examples. GEPA uses that feedback to propose the next version:\n\n```\njeva functions\n```\n\nSee [AGENTS.md](https://github.com/sutro-sh/jev-align/blob/main/AGENTS.md) for detailed setup, provider configuration, CLI\noperation, and development guidance for coding agents.\n\nSutro is not affiliated with TypeSafe AI, the makers of Jev.", "url": "https://wpnews.pro/news/show-hn-jev-align-an-oss-library-for-calibrating-jev-to-your-judgement", "canonical_source": "https://github.com/sutro-sh/jev-align", "published_at": "2026-09-19 23:05:55+00:00", "updated_at": "2026-09-19 23:24:17.812154+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "ai-products", "machine-learning"], "entities": ["Sutro", "jev-align", "TypeSafe", "Jev", "GEPA", "Python", "Vercel", "Cloudflare Workers AI"], "alternates": {"html": "https://wpnews.pro/news/show-hn-jev-align-an-oss-library-for-calibrating-jev-to-your-judgement", "markdown": "https://wpnews.pro/news/show-hn-jev-align-an-oss-library-for-calibrating-jev-to-your-judgement.md", "text": "https://wpnews.pro/news/show-hn-jev-align-an-oss-library-for-calibrating-jev-to-your-judgement.txt", "jsonld": "https://wpnews.pro/news/show-hn-jev-align-an-oss-library-for-calibrating-jev-to-your-judgement.jsonld"}}