# Ranking a Backlog with Jev > Source: > Published: 2026-10-19 20:42:32+00:00 Article # Ranking a Backlog with Jev An introduction to TypeSafe's Jev, a model that answers typed questions with probabilities, through a small tool that ranks a GitHub backlog with it and checks the result against a blind ranking. [Jev](https://docs.typesafe.ai/) is TypeSafe’s decision model. It writes no text: you send it a piece of state and typed questions, and it returns a probability for each possible answer. The [last article](https://feniix-hq.net/blog/hindsight-llm-comparison/) used it as a grader. This one is about building on it. I wrote [backlog-ranker](https://feniix-hq.net/projects/backlog-ranker/), a CLI that ranks a GitHub or Linear backlog by a YAML rubric: Jev answers four narrow questions about one issue at a time, code does the rest, and a calibration step refuses to call the result trusted until it agrees with a ranking made without the tool. I ran the rubric that ships with the tool, unchanged, on two public backlogs, [Hindsight](https://github.com/vectorize-io/hindsight) and [Biome](https://github.com/biomejs/biome), and calibrated it against blind rankings of 15, 25 and 35 issues. At 25 and 35 it was trusted on both. At 15 the same Biome backlog scored 0.14 on one random sample and 0.51 on another, on a scale where the tool’s trust bar is 0.4, which says more about its 12-issue minimum than about the rubric. The tool’s own `explain` output traced each miss to one question or one gate. 447 open issues across Biome and Hindsight, jev-1.14 through TypeSafe's own API at $0.042 per million input tokens, backlog-ranker 0.1.2, the shipped impact-roi rubric. Both agreement figures are on Biome: the rubric against a blind ranking of 35 issues, and two rankers against each other on 15. The article measures agreement as Kendall's tau, explained below, where they are 0.54 and 0.71. ## What Jev answers TypeSafe calls Jev a System One model: trained to return calibrated decisions, not generated text. Its API has three question types for text (a fourth, Box, is for images). - **Score** : the state is placed on ordered levels you write as sentences. The answer is a probability per level and their weighted mean, plus a confidence that says how concentrated the probabilities are, not how likely the answer is to be right. The rubric asks two,`impact` and`effort` , five levels each. - **Noul** : a yes-or-no question. The answer is the probability of yes; a value near 0.5 means unsure. The rubric asks two:`security` pins an issue to the top at 0.5 or above,`judgeable` holds it out for a person below 0.5. - **Choice** : one option from a set, with a probability per option. The tool’s`--tiebreak` uses it to compare pairs at the top of the ranking, in both orders. One recorded answer from the tool’s test fixtures, for a synthetic issue: all four questions in one request, 628 input tokens, $0.000026 at list price. The `impact` Score shows what an answer is made of. The model saw five sentences and returned a probability for each. | Level | Sentence the model saw | Probability | |---|---|---| | 0 | Cosmetic: wording or a nicety nobody is slowed by | 0.12 | | 1 | Removes minor friction in a rarely used command | 0.70 | | 2 | Removes recurring friction in a workflow used on most goals | 0.13 | | 3 | Fixes a failure that blocks a workflow used on most goals | 0.05 | | 4 | Prevents a leaked secret, lost work, or silently wrong results | 0 | The test fixture’s wording; the shipped rubric’s levels are longer. The Score is each level number times its probability, added up: 0×0.12 + 1×0.70 + 2×0.13 + 3×0.05 = 1.11, and its confidence, 0.71, says how concentrated the probabilities are: 1 when all on one level, lower as they spread. The model sees only the sentences, never the numbers, so levels describe situations, not degrees. | Question | Probabilities | Answer | Confidence | Used as | |---|---|---|---|---| | impact (Score) | levels 0–4: 0.12, 0.70, 0.13, 0.05, 0 | 1.11 | 0.71 | benefit in the composite | | effort (Score) | levels 0–4: 1, 0, 0, 0, 0 | 0 | 1.00 | cost in the composite | | security (Noul) | yes: 0.01 | no | — | under 0.5, not pinned | | judgeable (Noul) | yes: 0.97 | yes | — | over 0.5, not held | The rubric flags a Score under confidence 0.5 as low-confidence. A Noul’s answer is its probability of yes, judged against the gate’s threshold; it has no separate confidence. The adapter checks every answer before it is used: each distribution must carry every requested key and sum to one, and the response must name the pinned model version or it is discarded. TypeSafe’s own advice is the same: the `jev-latest` alias moves on each release, so pin the version your thresholds were tuned on. These runs used jev-1.14. ## What the code does Jev judges four things about an issue: which impact level fits, which effort level fits, whether it leaks a secret, and whether it is concrete enough to judge. Everything else is code. The facts are counted, not asked, and age goes as a bucket so a cached answer does not expire overnight. The composite divides mean benefit level by mean cost level, both on a one-based scale so the lowest cost divides by one, not zero. A pin beats a hold, and held issues are listed apart. Nothing is sent unless the source is on a committed allowlist, and comments are never sent. The cache is what makes experimenting cheap. Changing a weight, the composition or a gate re-asks nothing; rewording one question re-asks only that question. Ranking all 279 of Biome’s open issues took 279 requests and 30 seconds, about two cents by my estimate from the request sizes; every calibration of the shipped rubric after that took 0 requests, and only the rewording rounds described later asked anything new. ## Calibration and Kendall’s tau The tool’s `calibrate` verb takes a hand ranking of 12 or more issues, made before seeing any tool output, scores the same issues, and reports Kendall’s tau between the two orders. **Kendall’s tau** looks at every pair of issues. If both orderings put the same one first, the pair is concordant; otherwise discordant. Tau is concordant minus discordant, divided by the number of pairs: 1 is the same order, 0 no relation, −1 reversed. The tool calls a rubric trusted at tau 0.4 or more. | Hand ranking | Pairs | Worst issue | Chance spread | Pairs for 0.4 | |---|---|---|---|---| | 12 issues | 66 | 0.33 | ±0.22 | 47 | | 15 issues | 105 | 0.27 | ±0.19 | 74 | | 25 issues | 300 | 0.16 | ±0.14 | 210 | | 35 issues | 595 | 0.11 | ±0.12 | 417 | 12 is the tool’s minimum. “Worst issue” is the drop in tau from one issue placed first by one order and last by the other. “Chance spread” is the standard deviation of tau when the two orders are unrelated: at 12 issues the 0.4 bar is under two standard deviations from chance, at 25 nearly three. One disagreeing pair is worth 0.02 at 15 issues and 0.007 at 25. ## Two backlogs, six samples each For each repository I took a snapshot of the open issues and drew random samples with fixed seeds: two of 15, two of 25, two of 35. The rankers were not people. Each sample was ordered by a fresh Claude Fable 5.1 agent that read the issues on GitHub, comments and reactions included, by the rubric’s own tie-break question: which issue gives the project’s users more value for the engineering work it takes. Each 15-issue sample was also ranked by an agent that saw only what the tool sends. | Backlog | Open issues | Held by `judgeable` | Median `judgeable` | Low-confidence rows | |---|---|---|---|---| | Hindsight | 168 | 9 (5%) | 0.92 | 28% | | Biome | 279 | 52 (19%) | 0.77 | 39% | Whole backlog, shipped rubric. Hindsight’s issues are technical reports, more than half filed on a form; Biome’s are mostly lint-rule and editor reports from users. A row is low-confidence when a Score answer’s confidence is under the rubric’s floor of 0.5. At 25 and 35 issues every sample is trusted, and the two samples of each size agree within 0.02 on both backlogs. The two Biome 15-issue samples are the outliers, and both contained held issues the ranker had near the top: the hold gate is applied to a fifth of Biome’s backlog, and whether the held issues in a 15-issue sample are its worst tickets or its messiest important bugs is luck. In the first 35-issue sample the rubric held eight issues and the ranker placed all eight in the bottom half. The comment thread matters on Biome and not on Hindsight. Against the body-only rankings, Biome’s 15-issue samples scored 0.37 and 0.37 instead of 0.14 and 0.16; on Hindsight the body-only and thread rankings agreed with each other at 0.77 and 0.91, and the tool scored 0.75 and 0.47 against them. Two agents ranking Biome’s sample A from the threads agreed with each other at 0.71, so the Biome gap is not rater noise. ## Where the 15-issue misses came from `explain` prints one issue’s level probabilities, each gate’s probability against its threshold, and what precedence let each gate do. Biome’s disagreeing pairs had four causes. **The hold gate held the best reports.** Biome’s “extension at 99% CPU” report, 22 reactions, was the thread ranker’s 2nd; the tool scored it impact 2.56 and held it at `judgeable` 0.24, because the usable stack sample is in a comment. A confirmed wrong-fix bug the ranker put 5th was held at 0.46. Held issues are listed last, so each cost up to a dozen pairs. **Comments are not sent.** Both thread rankers put one issue last because a comment says it is already fixed; the tool, reading the body, put it sixth. **Reactions are a fact without a level.** The tool sends `reactions: 46` with the sample’s most-wanted issue, but no level mentions reactions, so the model has nowhere to put them; the rankers weighted them heavily. **A ratio caps severity.** Biome’s unsigned Windows binary blocks the tool on locked-down machines; the rankers had it 2nd and 3rd, the shipped rubric 12th. Even after rewording put it at impact level 3 (0.79) and effort level 2 (0.90), the rankers’ own description, it landed 11th: dividing by effort means a severe issue with moderate effort cannot outrank a trivial fix. ## What changing the rubric did Removing the hold gate re-ranks from the cache, so it could be tried on every sample at no cost. | Sample (issues) | Shipped rubric | Without the hold gate | Change | |---|---|---|---| | Biome 15, sample A | 0.14 | 0.24 | +0.10 | | Biome 15, sample B | 0.16 | 0.31 | +0.15 | | Biome 25, sample A | 0.51 | 0.50 | −0.01 | | Biome 25, sample B | 0.51 | 0.51 | 0 | | Biome 35, sample A | 0.54 | 0.56 | +0.02 | | Biome 35, sample B | 0.54 | 0.58 | +0.04 | | Hindsight 15, sample A | 0.68 | 0.68 | 0 | | Hindsight 15, sample B | 0.45 | 0.35 | −0.10 | | Hindsight 25, sample A | 0.65 | 0.51 | −0.13 | | Hindsight 25, sample B | 0.65 | 0.56 | −0.09 | | Hindsight 35, sample A | 0.48 | 0.48 | 0 | | Hindsight 35, sample B | 0.50 | 0.44 | −0.06 | Kendall’s tau against the thread rankings, jev-1.14. Every row cost 0 new requests. Removing the gate rescued the two small Biome samples and never helped a Hindsight one: four of the six got worse, because the issues it held there, such as a storage-quantization wish and a request for a Copilot provider, were the ones the ranker put last. On Biome’s larger samples it barely mattered. The gate is right when “hard to judge” means “low value” and wrong when important bugs are badly written; a large enough sample averages over both, and on a backlog of well-written reports it mostly helps. Rewording the levels did worse. Three rounds of edits on Biome’s sample A, each re-asking the 15 issues, raised it from 0.14 to between 0.20 and 0.26 and lowered sample B to 0.11; switching the reworded rubric to a weighted sum with impact at four times effort reached 0.28 on sample A and 0.07 on sample B. ## Limitations - The rankers are a model, not the maintainers; two agents agreed with each other at 0.71 on one sample, which bounds what any rubric could reach against them. - One ranker per sample (two on the sample where I measured rater agreement), two repositories, both developer tools, one day’s snapshot each. - Cost is an estimate; the tool does not log token usage. - The tool’s 12-issue minimum is the number I would change first. ## Try it A decision model gives you a probability per answer and nothing else. That was the useful part: every disagreement traced to a number on a question, and every experiment after the first rank was free. What you still have to supply is the ranking a person would make. ``` export GITHUB_TOKEN=$(gh auth token) export TYPESAFE_API_KEY=... curl -sO https://raw.githubusercontent.com/Spantree/backlog-ranker/main/rubrics/impact-roi.yaml npx -y @feniix/backlog-ranker rank --rubric "$PWD/impact-roi.yaml" \ --source github:OWNER/REPO --allow-once --backend typesafe ``` `--allow-once` admits the source for one run and prints what leaves the machine. Then rank 25 issues yourself, before looking at the output, and run `calibrate`. The pairs it lists are where you and the rubric disagree.