Using Jev to make your agents write the code and design the app the way you do Filed cofounder Atul reported that coding agents pass lint, tests and CI while still producing code that human reviewers reject, because tests check what code does and lint checks how it looks, but nothing checks whether it is written the way a team writes things. Filed runs Perplexity's decider (pplx-decider), a decision model similar to TypeSafe's Jev that returns a probability per yes-or-no question and can inspect images to verify UI follows design rules, in place of prose conventions written in AGENTS.md, CLAUDE.md, best-practices docs and skills, which agents skim and which drift out of agreement. Using Jev to make your agents write the code and design the app the way you do: Moving beyond tests and lints I'm going to be honest with you: your coding agent passes lint, passes tests, goes green in CI, and still writes code you would reject in review. Every single day. If you've been told that a good enough test suite is all you need to let agents loose on your codebase, I'm sorry, but someone sold you a slightly smaller version of the truth. Tests check what the code does . Lint checks a few things about how it looks . Nobody is checking whether it's written the way we write things here . Up until recently that job rested on a few senior devs, in review, saying "...yeah, no" to the same five mistakes over and over again. I'm Atul, cofounder at Filed https://www.filed.com/ . We build agents for tax professionals, and a lot of our code is now written by other agents. A while back I wrote about how far you can push delegation to coding agents ./scaling-agentic-development.html and said the third leg of the stool was "codifying your best practices and having an agentic reviewer check them". This post is about what that turned into once we actually built it, and all the ways the first versions didn't work. Preface This is not a "tests are dead" post. Tests and lints are still the first thing you reach for, and if a script can decide it, a script should decide it. Everything below is about the leftovers: the stuff that's obviously wrong to a human on your team and invisible to every tool you own. Also, a confession about the title. Jev, from TypeSafe, is the decision model that got me hooked on this idea, but what we actually run in production is Perplexity's decider pplx-decider . Same kind of API, one big difference: the Perplexity decider can look at images , which turns out to be great for checking that UI follows your design rules more on that below . So I'm going to say "Perplexity decider / Jev" interchangeably in this post. Read it as "a decision model". Here's what a decision model does: you give it some state and a list of yes-or-no questions, and it gives you back a probability per question. That's it. No essays. Hold onto that, it matters later. The stuff that slips through is not exotic It's boring. That's the problem. Here's the kind of thing that goes green through every pipeline we have: - An except Exception: return None that turns a real failure into an empty screen and a very confused user. - A tool registered on an agent twice, because the agent was already getting it from another toolset. Works fine. Tests pass. The model now sees the same tool twice and gets to guess which one you meant. Lovely. - Shared code that grows another if / else branch every time a new case shows up, instead of that case getting its own module. Each branch looks harmless on its own. Ten of them later, the shared code is a pile of special cases nobody wants to touch. - Every new frontend page, even after you've written down your design guidelines and rules, coming out looking like a generic Claude-generated UI. Good at first glance. Clean, modern, nothing obviously wrong. It just doesn't look like your app. It lacks its soul. None of these break anything today. Every one of them makes the codebase a bit worse to work in tomorrow. And agents, being agents, will happily repeat the pattern the moment they see it in two places. We tried writing it down. The agents skimmed it. The obvious move is to write your conventions down. So we did. In AGENTS.md . In CLAUDE.md . In a best-practices doc. In skills. In review notes. Probably in a Notion page somewhere too. Here's what happens with that: the agent reads some of it, the context window eats the rest, and six months later you have four documents that disagree with each other about whether errors should bubble up. Prose drifts. Nobody tests prose. And a "do / don't" sentence buried in paragraph nine of a README is basically a suggestion. "Just ask an LLM to review the PR" is a vibes machine Ask it twice, get two reviews. The next obvious move is to throw the diff at a chat model and say "review this against our standards". We tried this too. You get back a nicely formatted essay. It flags a variable name. It praises your error handling. It suggests adding a docstring to a one-line function. Run it again and it flags a different variable name. You cannot put a threshold on an essay. You cannot tell whether your instructions got better or worse. You cannot tell whether it actually looked at the thing you cared about, or just felt like commenting on something. What I actually wanted was much dumber: one question per rule, one number per question. Enter Perplexity decider / Jev: it only answers yes or no This is where a decision model earns its keep. You send it two things: a state everything it needs to look at and a set of named questions . It sends back one number per question. But a model on its own isn't a system. You need something the agent can actually call, rules for it to read, and a loop that ties it together. So let's build it up, step by step: the CLI, the rules, what gets sent to the Perplexity decider, what comes back, and the loop the agent runs. Step 1: build your agent a CLI Remember the agents skimming our docs? That's the problem this step solves. Anything the agent is supposed to remember to read, it eventually won't. Anything it can call , it will, every time, because calling a tool is what agents are good at. So before anything else, you need a CLI. I'll come to exactly what goes inside it in step 5, but you need it first, because every other step is something the agent does by running it. Ours is called filed or filed.exe , for the poor souls on Windows . It's the one front door for everything an agent needs to build and verify a change in our codebase: start the local stack, sign into the app, take screenshots, record what was verified. This CLI is your harness . For this post, the part we care about is one command inside it: review . Step 2: write the rules Next, the rules. These live in rules.yaml files. Each rule is one claim the model can answer yes or no to, plus what to do about it. From here on the examples use an imaginary online shop, made-up diffs and illustrative numbers. The rule wording is our real wording. shop/docs/review/rules.yaml simplified: in reality these are split across folders, see below - id: code.errors-bubble-up applies: " / .py", " / .ts" mistake: The change catches an exception and then returns None, an empty list, a default value or a fallback instead of letting the error raise fix: Let the error raise; handle it only where the caller can act on it reference: payments/retry.py cases: violating: - file: orders/lookup.py diff: | + except Exception: + return None compliant: - file: orders/lookup.py diff: | + raise OrderNotFound order id - id: code.identity-from-token applies: " /graphql/ " mistake: A GraphQL field, mutation argument or input type accepts an account ID, user ID or role from the caller fix: Read the account and user from the auth token in the resolver reference: graphql/resolvers/me.py cases: ... - id: code.no-product-branches-in-shared-code applies: " /shared/ ", "orders/ " mistake: The change adds an if-check or filter on a specific product, task type, provider, surface or flavour inside shared or generic code fix: Move the specific behaviour into that product's or provider's own module or class reference: shipping/carriers/README.md cases: ... - id: code.one-code-path applies: " / .py", " / .ts" statement: The change keeps one code path, with no fallback, legacy, compatibility or workflow.patched branches for old data or old callers fix: Remove the old path and migrate its data or drain its runs reference: docs/migrations.md cases: ... A rule is either a mistake true when the change breaks it or a statement true when the change follows it . Never both. fix and reference are for the agent. cases are for testing the rule itself, which we'll get to. Above I showed all four rules in one file to keep things simple. In reality, rules are not one giant flat file. They're nested , and they live next to the code they judge. Any folder that cares can have its own docs/review/rules.yaml . A rule applies to its folder and everything below it. So the general stuff sits at the top, and the specific stuff sits right beside the code it's about: shop/ ├── docs/review/rules.yaml errors bubble up, one code path everything ├── graphql/ │ ├── docs/review/rules.yaml identity comes from the token only GraphQL │ └── inputs/create order.py ├── orders/ │ ├── docs/review/rules.yaml no N+1 queries on the order list only orders │ └── update.py └── shipping/ ├── docs/review/rules.yaml one module per carrier only shipping └── carriers/ ├── docs/review/rules.yaml every carrier implements prepare only carriers └── fedex.py When the agent is about to change shop/graphql/inputs/create order.py , the CLI walks up from that file and collects every rules.yaml on the way: the one in shop/graphql/ and the one in shop/ . It never sees the shipping rules or the carrier rules, because it isn't touching that code. If it later wanders into shipping/carriers/fedex.py , it asks again and gets a different, equally short list. This matters for three reasons: - The agent only reads what's relevant. A flat file of 200 rules is 200 rules the agent has to read, mostly ignore, and still be distracted by. A nested set gives it the five that apply to the folder it's in. - Rules live with their owners. The people who own the shipping code own the shipping rules, in the shipping folder. When the code moves or gets deleted, the rules go with it instead of rotting in a doc somewhere. - It keeps review fast and cheap. Every rule that applies is one more question per review, and as we'll see in the cost section every extra question makes the whole request bigger. Scoping a rule to its folder is the cheapest optimisation there is. We also keep the repo-wide rules on a diet: at most 15 of them may apply to any one file, and a test fails if that number creeps up, which is usually someone scoping a rule to / because they couldn't be bothered to narrow it. Ours are spread across about 27 of these files in six repositories right now, from repo-wide ones at the top down to a single folder of database code. The trick is that the same file has two readers. Step 3: what gets sent to the Perplexity decider The CLI finds the changed files, works out which rules apply to them, and turns each rule into a question. The question is the rule's own words, prefixed with where it applies: POST https://api.perplexity.ai/v1/decisions { "model": "pplx-decider-v1.1-27b", "state": { "context": "Changes in a pull request to the codebase, with the author's summary. Each question is about these changes only. references holds existing files that show the codebase's established pattern for a rule; use them to judge whether the change reuses or duplicates what already exists.", "summary": "Add a 'create order' mutation for the new checkout flow", "files": { "repo": "shop", "file": "orders/lookup.py", "diff": "+ try:\n+ return repo.get order id \n+ except Exception:\n+ return None" }, { "repo": "shop", "file": "graphql/inputs/create order.py", "diff": "+ account id: str\n+ items: list OrderItem " }, { "repo": "shop", "file": "orders/update.py", "diff": "+ if order.channel == Channel.WHOLESALE:\n+ notify wholesale team order " } , "references": { "repo": "shop", "file": "payments/retry.py", "content": "..." }, { "repo": "shop", "file": "graphql/resolvers/me.py", "content": "..." }, { "repo": "shop", "file": "shipping/carriers/README.md", "content": "..." } }, "questions": { "q0": { "type": "noul", "instructions": "In the shop repository, for files matching / .py, / .ts: The change catches an exception and then returns None, an empty list, a default value or a fallback instead of letting the error raise" }, "q1": { "type": "noul", "instructions": "In the shop repository, for files matching /graphql/ : A GraphQL field, mutation argument or input type accepts an account ID, user ID or role from the caller" }, "q2": { "type": "noul", "instructions": "In the shop repository, for files matching /shared/ , orders/ : The change adds an if-check or filter on a specific product, task type, provider, surface or flavour inside shared or generic code" }, "q3": { "type": "noul", "instructions": "In the shop repository, for files matching / .py, / .ts: The change keeps one code path, with no fallback, legacy, compatibility or workflow.patched branches for old data or old callers" } } } That's the whole prompt. No "you are a senior engineer", no rubric, no "think step by step". Just the change, the files that show the right pattern, and four claims. Step 4: what comes back { "answers": { "q0": { "noul": 0.96 }, "q1": { "noul": 0.93 }, "q2": { "noul": 0.88 }, "q3": { "noul": 0.94 } }, "usage": { "input tokens": 9412 } } Each number is the probability that the claim is true. For the three mistake rules, a high number is bad news. For the one statement rule, a high number is good news. So before showing anything to anyone, the CLI flips the mistakes 1 - p , so that every result reads the same way: the chance the change follows the rule. | Rule | Kind | Raw answer | Compliance | | |---|---|---|---|---| | errors-bubble-up | mistake | 0.96 | 4% | hint | | identity-from-token | mistake | 0.93 | 7% | hint | | no-product-branches-in-shared-code | mistake | 0.88 | 12% | hint | | one-code-path | statement | 0.94 | 94% | quiet | Anything under 80% becomes a hint, sorted worst first, with the rule's fix and reference attached. Everything else stays quiet. And here's what the agent actually gets back in its terminal: filed review: 3 hints 1 calls, 9412 input tokens 4% code.errors-bubble-up shop/orders/lookup.py fix: Let the error raise; handle it only where the caller can act on it see: shop/payments/retry.py 7% code.identity-from-token shop/graphql/inputs/create order.py fix: Read the account and user from the auth token in the resolver see: shop/graphql/resolvers/me.py 12% code.no-product-branches-in-shared-code shop/orders/update.py fix: Move the specific behaviour into that product's or provider's own module or class see: shop/shipping/carriers/README.md compliance 1/4 rules at 80% or above Because it's a number and not a paragraph, a bunch of things suddenly become possible: - You can threshold it. Under 80% is a hint, everything else stays quiet. - You can compare it. Same rule, same diff, different wording: which one separates good from bad better? - You can test it. More on that in a minute, because it's the part that made the whole thing work. Step 5: the loop: before code, code, after code Now that you've seen the rules and the decider, here's what goes inside the CLI from step 1. The review command has four subcommands: filed review rules --files