{"slug": "ranking-a-backlog-with-jev", "title": "Ranking a Backlog with Jev", "summary": "TypeSafe's decision model Jev, accessed through TypeSafe's API at $0.042 per million input tokens, was tested as the ranking engine behind backlog-ranker 0.1.2, a CLI that scores GitHub and Linear issues against a YAML rubric by asking Jev four typed questions per issue. Run unchanged on 447 open issues across the Hindsight and Biome backlogs, the shipped impact-roi rubric reached Kendall's tau agreement of 0.54 against a blind ranking of 35 Biome issues and 0.71 between two rankers on 15 issues, and the tool's calibration step marked results trusted at 25 and 35 issues but not at 15, where the same Biome backlog scored 0.14 on one random sample and 0.51 on another against a 0.4 trust bar.", "body_md": "Article\n\n# Ranking a Backlog with Jev\n\nAn introduction to TypeSafe's Jev, a model that answers typed questions with probabilities, through a small tool that ranks a GitHub backlog with it and checks the result against a blind ranking.\n\n[Jev](https://docs.typesafe.ai/) is TypeSafe’s decision model. It writes no\ntext: you send it a piece of state and typed questions, and it returns a\nprobability for each possible answer. The [last article](https://feniix-hq.net/blog/hindsight-llm-comparison/)\nused it as a grader. This one is about building on it. I wrote\n[backlog-ranker](https://feniix-hq.net/projects/backlog-ranker/), a CLI that ranks a GitHub or\nLinear backlog by a YAML rubric: Jev answers four narrow questions about one\nissue at a time, code does the rest, and a calibration step refuses to call\nthe result trusted until it agrees with a ranking made without the tool.\n\nI ran the rubric that ships with the tool, unchanged, on two public\nbacklogs, [Hindsight](https://github.com/vectorize-io/hindsight) and\n[Biome](https://github.com/biomejs/biome), and calibrated it against blind\nrankings of 15, 25 and 35 issues. At 25 and 35 it was trusted on both. At\n15 the same Biome backlog scored 0.14 on one random sample and 0.51 on\nanother, on a scale where the tool’s trust bar is 0.4, which says more\nabout its 12-issue minimum than about the rubric. The tool’s own `explain`\noutput traced each miss to one question or one gate.\n\n447 open issues across Biome and Hindsight, jev-1.14 through TypeSafe's own API at $0.042 per million input tokens, backlog-ranker 0.1.2, the shipped impact-roi rubric. Both agreement figures are on Biome: the rubric against a blind ranking of 35 issues, and two rankers against each other on 15. The article measures agreement as Kendall's tau, explained below, where they are 0.54 and 0.71.\n\n## What Jev answers\n\nTypeSafe calls Jev a System One model: trained to return calibrated decisions, not generated text. Its API has three question types for text (a fourth, Box, is for images).\n\n- **Score** : the state is placed on ordered levels you write as sentences.\nThe answer is a probability per level and their weighted mean, plus a\nconfidence that says how concentrated the probabilities are, not how\nlikely the answer is to be right. The rubric asks two,`impact` and`effort` , five levels each.\n- **Noul** : a yes-or-no question. The answer is the probability of yes; a\nvalue near 0.5 means unsure. The rubric asks two:`security` pins an\nissue to the top at 0.5 or above,`judgeable` holds it out for a person\nbelow 0.5.\n- **Choice** : one option from a set, with a probability per option. The\ntool’s`--tiebreak` uses it to compare pairs at the top of the ranking,\nin both orders.\n\nOne recorded answer from the tool’s test fixtures, for a synthetic issue:\nall four questions in one request, 628 input tokens, $0.000026 at list\nprice. The `impact` Score shows what an answer is made of. The model saw\nfive sentences and returned a probability for each.\n\n| Level | Sentence the model saw | Probability | \n|---|---|---|\n| 0 | Cosmetic: wording or a nicety nobody is slowed by | 0.12 | \n| 1 | Removes minor friction in a rarely used command | 0.70 | \n| 2 | Removes recurring friction in a workflow used on most goals | 0.13 | \n| 3 | Fixes a failure that blocks a workflow used on most goals | 0.05 | \n| 4 | Prevents a leaked secret, lost work, or silently wrong results | 0 | \n\nThe test fixture’s wording; the shipped rubric’s levels are longer. The Score is each level number times its probability, added up: 0×0.12 + 1×0.70 + 2×0.13 + 3×0.05 = 1.11, and its confidence, 0.71, says how concentrated the probabilities are: 1 when all on one level, lower as they spread. The model sees only the sentences, never the numbers, so levels describe situations, not degrees.\n\n| Question | Probabilities | Answer | Confidence | Used as | \n|---|---|---|---|---|\n| impact (Score) | levels 0–4: 0.12, 0.70, 0.13, 0.05, 0 | 1.11 | 0.71 | benefit in the composite | \n| effort (Score) | levels 0–4: 1, 0, 0, 0, 0 | 0 | 1.00 | cost in the composite | \n| security (Noul) | yes: 0.01 | no | — | under 0.5, not pinned | \n| judgeable (Noul) | yes: 0.97 | yes | — | over 0.5, not held | \n\nThe rubric flags a Score under confidence 0.5 as low-confidence. A Noul’s answer is its probability of yes, judged against the gate’s threshold; it has no separate confidence.\n\nThe adapter checks every answer before it is used: each distribution must\ncarry every requested key and sum to one, and the response must name the\npinned model version or it is discarded. TypeSafe’s own advice is the\nsame: the `jev-latest` alias moves on each release, so pin the version\nyour thresholds were tuned on. These runs used jev-1.14.\n\n## What the code does\n\nJev judges four things about an issue: which impact level fits, which effort level fits, whether it leaks a secret, and whether it is concrete enough to judge. Everything else is code. The facts are counted, not asked, and age goes as a bucket so a cached answer does not expire overnight. The composite divides mean benefit level by mean cost level, both on a one-based scale so the lowest cost divides by one, not zero. A pin beats a hold, and held issues are listed apart. Nothing is sent unless the source is on a committed allowlist, and comments are never sent.\n\nThe cache is what makes experimenting cheap. Changing a weight, the composition or a gate re-asks nothing; rewording one question re-asks only that question. Ranking all 279 of Biome’s open issues took 279 requests and 30 seconds, about two cents by my estimate from the request sizes; every calibration of the shipped rubric after that took 0 requests, and only the rewording rounds described later asked anything new.\n\n## Calibration and Kendall’s tau\n\nThe tool’s `calibrate` verb takes a hand ranking of 12 or more issues, made\nbefore seeing any tool output, scores the same issues, and reports\nKendall’s tau between the two orders.\n\n**Kendall’s tau** looks at every pair of issues. If both orderings put the\nsame one first, the pair is concordant; otherwise discordant. Tau is\nconcordant minus discordant, divided by the number of pairs: 1 is the same\norder, 0 no relation, −1 reversed. The tool calls a rubric trusted at tau\n0.4 or more.\n\n| Hand ranking | Pairs | Worst issue | Chance spread | Pairs for 0.4 | \n|---|---|---|---|---|\n| 12 issues | 66 | 0.33 | ±0.22 | 47 | \n| 15 issues | 105 | 0.27 | ±0.19 | 74 | \n| 25 issues | 300 | 0.16 | ±0.14 | 210 | \n| 35 issues | 595 | 0.11 | ±0.12 | 417 | \n\n12 is the tool’s minimum. “Worst issue” is the drop in tau from one issue placed first by one order and last by the other. “Chance spread” is the standard deviation of tau when the two orders are unrelated: at 12 issues the 0.4 bar is under two standard deviations from chance, at 25 nearly three. One disagreeing pair is worth 0.02 at 15 issues and 0.007 at 25.\n\n## Two backlogs, six samples each\n\nFor each repository I took a snapshot of the open issues and drew random samples with fixed seeds: two of 15, two of 25, two of 35. The rankers were not people. Each sample was ordered by a fresh Claude Fable 5.1 agent that read the issues on GitHub, comments and reactions included, by the rubric’s own tie-break question: which issue gives the project’s users more value for the engineering work it takes. Each 15-issue sample was also ranked by an agent that saw only what the tool sends.\n\n| Backlog | Open issues | Held by `judgeable` | Median `judgeable` | Low-confidence rows | \n|---|---|---|---|---|\n| Hindsight | 168 | 9 (5%) | 0.92 | 28% | \n| Biome | 279 | 52 (19%) | 0.77 | 39% | \n\nWhole backlog, shipped rubric. Hindsight’s issues are technical reports, more than half filed on a form; Biome’s are mostly lint-rule and editor reports from users. A row is low-confidence when a Score answer’s confidence is under the rubric’s floor of 0.5.\n\nAt 25 and 35 issues every sample is trusted, and the two samples of each size agree within 0.02 on both backlogs. The two Biome 15-issue samples are the outliers, and both contained held issues the ranker had near the top: the hold gate is applied to a fifth of Biome’s backlog, and whether the held issues in a 15-issue sample are its worst tickets or its messiest important bugs is luck. In the first 35-issue sample the rubric held eight issues and the ranker placed all eight in the bottom half.\n\nThe comment thread matters on Biome and not on Hindsight. Against the body-only rankings, Biome’s 15-issue samples scored 0.37 and 0.37 instead of 0.14 and 0.16; on Hindsight the body-only and thread rankings agreed with each other at 0.77 and 0.91, and the tool scored 0.75 and 0.47 against them. Two agents ranking Biome’s sample A from the threads agreed with each other at 0.71, so the Biome gap is not rater noise.\n\n## Where the 15-issue misses came from\n\n`explain` prints one issue’s level probabilities, each gate’s probability\nagainst its threshold, and what precedence let each gate do. Biome’s\ndisagreeing pairs had four causes.\n\n**The hold gate held the best reports.** Biome’s “extension at 99% CPU”\nreport, 22 reactions, was the thread ranker’s 2nd; the tool scored it\nimpact 2.56 and held it at `judgeable` 0.24, because the usable stack\nsample is in a comment. A confirmed wrong-fix bug the ranker put 5th was\nheld at 0.46. Held issues are listed last, so each cost up to a dozen\npairs.\n\n**Comments are not sent.** Both thread rankers put one issue last because\na comment says it is already fixed; the tool, reading the body, put it\nsixth.\n\n**Reactions are a fact without a level.** The tool sends `reactions: 46`\nwith the sample’s most-wanted issue, but no level mentions reactions, so\nthe model has nowhere to put them; the rankers weighted them heavily.\n\n**A ratio caps severity.** Biome’s unsigned Windows binary blocks the tool\non locked-down machines; the rankers had it 2nd and 3rd, the shipped\nrubric 12th. Even after rewording put it at impact level 3 (0.79) and\neffort level 2 (0.90), the rankers’ own description, it landed 11th:\ndividing by effort means a severe issue with moderate effort cannot\noutrank a trivial fix.\n\n## What changing the rubric did\n\nRemoving the hold gate re-ranks from the cache, so it could be tried on every sample at no cost.\n\n| Sample (issues) | Shipped rubric | Without the hold gate | Change | \n|---|---|---|---|\n| Biome 15, sample A | 0.14 | 0.24 | +0.10 | \n| Biome 15, sample B | 0.16 | 0.31 | +0.15 | \n| Biome 25, sample A | 0.51 | 0.50 | −0.01 | \n| Biome 25, sample B | 0.51 | 0.51 | 0 | \n| Biome 35, sample A | 0.54 | 0.56 | +0.02 | \n| Biome 35, sample B | 0.54 | 0.58 | +0.04 | \n| Hindsight 15, sample A | 0.68 | 0.68 | 0 | \n| Hindsight 15, sample B | 0.45 | 0.35 | −0.10 | \n| Hindsight 25, sample A | 0.65 | 0.51 | −0.13 | \n| Hindsight 25, sample B | 0.65 | 0.56 | −0.09 | \n| Hindsight 35, sample A | 0.48 | 0.48 | 0 | \n| Hindsight 35, sample B | 0.50 | 0.44 | −0.06 | \n\nKendall’s tau against the thread rankings, jev-1.14. Every row cost 0 new requests.\n\nRemoving the gate rescued the two small Biome samples and never helped a Hindsight one: four of the six got worse, because the issues it held there, such as a storage-quantization wish and a request for a Copilot provider, were the ones the ranker put last. On Biome’s larger samples it barely mattered. The gate is right when “hard to judge” means “low value” and wrong when important bugs are badly written; a large enough sample averages over both, and on a backlog of well-written reports it mostly helps.\n\nRewording the levels did worse. Three rounds of edits on Biome’s sample A, each re-asking the 15 issues, raised it from 0.14 to between 0.20 and 0.26 and lowered sample B to 0.11; switching the reworded rubric to a weighted sum with impact at four times effort reached 0.28 on sample A and 0.07 on sample B.\n\n## Limitations\n\n- The rankers are a model, not the maintainers; two agents agreed with each other at 0.71 on one sample, which bounds what any rubric could reach against them.\n- One ranker per sample (two on the sample where I measured rater agreement), two repositories, both developer tools, one day’s snapshot each.\n- Cost is an estimate; the tool does not log token usage.\n- The tool’s 12-issue minimum is the number I would change first.\n\n## Try it\n\nA decision model gives you a probability per answer and nothing else. That was the useful part: every disagreement traced to a number on a question, and every experiment after the first rank was free. What you still have to supply is the ranking a person would make.\n\n```\nexport GITHUB_TOKEN=$(gh auth token)\nexport TYPESAFE_API_KEY=...\ncurl -sO https://raw.githubusercontent.com/Spantree/backlog-ranker/main/rubrics/impact-roi.yaml\nnpx -y @feniix/backlog-ranker rank --rubric \"$PWD/impact-roi.yaml\" \\\n  --source github:OWNER/REPO --allow-once --backend typesafe\n```\n\n`--allow-once` admits the source for one run and prints what leaves the\nmachine. Then rank 25 issues yourself, before looking at the output, and\nrun `calibrate`. The pairs it lists are where you and the rubric disagree.", "url": "https://wpnews.pro/news/ranking-a-backlog-with-jev", "canonical_source": "https://feniix-hq.net/blog/ranking-a-backlog-with-jev/", "published_at": "2026-10-19 20:42:32+00:00", "updated_at": "2026-10-19 20:46:24.208553+00:00", "lang": "en", "topics": ["ai-tools", "ai-products", "artificial-intelligence", "developer-tools"], "entities": ["TypeSafe", "Jev", "backlog-ranker", "Hindsight", "Biome", "GitHub", "Linear"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/ranking-a-backlog-with-jev", "markdown": "https://wpnews.pro/news/ranking-a-backlog-with-jev.md", "text": "https://wpnews.pro/news/ranking-a-backlog-with-jev.txt", "jsonld": "https://wpnews.pro/news/ranking-a-backlog-with-jev.jsonld"}}