{"slug": "testing-jev-as-a-validation-gate-for-drug-discovery-agents", "title": "Testing Jev as a validation gate for drug-discovery agents", "summary": "A developer building a multi-agent drug-discovery system tested TypeSafe's Jev decision model as an independent validation gate for catching incorrect literature claims, molecular edits, and property records. Jev declined to generate a SMILES string for deucravacitinib when given an UNKNOWN option, but when that option was removed it produced decane's SMILES instead, illustrating that the model is suited to selecting among supplied candidates rather than free-form generation. The experiment compared Jev against GPT-6 Astra at low reasoning effort.", "body_md": "In my work building a multi-agent system for drug discovery, I see LLMs struggle repeatedly with the small, exact operations that hold a cheminformatics workflow together. An agent can explain the medicinal chemistry convincingly, propose a reasonable modification, and then return a structure that quietly changes something else. A stereocentre flips. A protonation state changes. A descriptor belongs to the previous molecule. The explanation remains plausible while the molecular object underneath it has changed.\n\nThose errors sometimes happen early enough to compromise everything that follows. If a requested analogue acquires an unintended stereochemical inversion, conformer generation, docking and free-energy calculations can proceed on the wrong stereoisomer. The calculations may execute correctly for the structure they received, but their results cannot simply be assigned to the compound the chemist intended. A polished final report can conceal an identity error introduced several steps earlier.\n\nThat is the motivation for this experiment: could Jev, a fast, inexpensive decision model act as an independent critic in an agentic drug-discovery workflow? A proposing agent would submit an edit, a compound record or a literature claim. A gate would challenge that submission before the next action proceeds. By “adversarial,” I mean that the critic is expected to test the proposing agent’s assertions.\n\nSeveral influential papers explore this idea. [CRITIC (Gou et al., ICLR 2024)](https://arxiv.org/abs/2305.11738) uses feedback from tools to correct model answers. [LLM-Modulo (Kambhampati et al., ICML 2024)](https://proceedings.mlr.press/v235/kambhampati24a.html) puts external verifiers in charge of checking proposed plans. In scientific research, [Towards an AI co-scientist (Gottweis et al., 2025)](https://arxiv.org/html/2502.18864v1) assigns separate agents to generate, review and rank hypotheses. These approaches support checking work as it proceeds.\n\nI tested whether **Jev could catch incorrect literature claims, molecular edits and property records while allowing correct ones through**. I compared it with GPT-6 Astra at low reasoning effort.\n\n## What Jev is built for\n\nIn its September 15 [announcement](https://typesafe.ai/blog/introducing-system-one-models-and-jev), TypeSafe introduced Jev as a model for fast, structured decisions inside software. You supply the information, the question and the allowed answers. Jev returns choices and probabilities that code can use to decide what happens next. It gives up free-form text generation to focus on those decisions.\n\nWe start with two SMILES examples to show how Jev is meant to be used: **generating a string is a poor fit; selecting from supplied strings is a better fit**.\n\n## 1. Poor fit: generate a SMILES string\n\nWe asked Jev to write the SMILES for **deucravacitinib**, one character at a time. Each call supplied the compound name, the characters written so far and the allowed next characters. Jev could also choose STOP when the string was complete or UNKNOWN if it could not construct the target reliably. This was a generation task assembled from repeated choices, the kind of use TypeSafe’s announcement discourages from Jev’s intended role.\n\n**Jev chose UNKNOWN on the first call, before writing any characters.** That was encouraging: it declined to supply a structure it could not produce reliably.\n\nWe then removed the UNKNOWN choice. Jev chose `C` ten times, followed by STOP, producing:\n\n```\nCCCCCCCCCC\n```\n\nThat is valid SMILES for **decane**, not **deucravacitinib**. The string could be parsed, but it represented the wrong molecule. Removing the uncertainty option produced an answer; it did not make the answer correct.\n\nWe also supplied the reference SMILES and asked Jev to copy it character by character. It failed at the second character. These small tests illustrate why we should not use Jev as a generative tool.\n\n## 2. Better fit: choose a supplied SMILES string\n\nHere, we supply complete structures and ask Jev to make one decision. This is closer to its intended use and resembles an agent choosing the right compound from search results before starting calculations.\n\nWe gave Jev the name “deucravacitinib” and the six candidate SMILES below, labelled A–F. The question was: **which letter represents deucravacitinib?** Only the letters and SMILES were supplied.\n\n*The correct answer in this displayed order is A.*\n\nWe repeated the question with the candidates in different orders. The correct answer is always be the letter assigned to deucravacitinib. NONE means no candidate matched and is wrong here, because the target is present. UNKNOWN means the model could not identify it reliably. This tests whether Jev can make the identity decision without having to write the structure itself.\n\n## From choosing structures to checking an agent’s work\n\nRecalling a molecule from its name is not the main job we want a gate to do of course. In a drug-discovery workflow, an LLM agent usually has something concrete to check: a claim extracted from a paper, a proposed molecular edit. **The more useful question is whether that submission is correct enough for the next step to proceed.**\n\nA specialized LLM agent could turn the task requirements into focused questions, gather the evidence and define the allowed answers. Jev would evaluate those checks; workflow rules would then let the main agent proceed, retrieve more evidence, revise its work or ask for review. The check belongs before an error can spread into later calculations and decisions.\n\nThe next tests examine those decisions directly: does the supplied evidence support the claim, did the molecule change only as requested, and do its properties match its SMILES? We supplied the inputs and answer choices to test the gates in isolation. We did not test an autonomous agent creating the checks or measure improvement across a complete workflow yet though.\n\n## How we compared Jev with Astra\n\nFor the measured literature and molecular-check tests, we compared Jev with **Astra low**. I chose that setting because it offers a useful balance of performance and cost. In the September 19 Artificial Analysis results, Astra low scored 46 on the Intelligence Index at $0.82 per benchmark task, compared with Sol max at 47 and $1.99. [Benchmark comparison](https://artificialanalysis.ai/models/comparisons/gpt-6-astra-low-vs-gpt-5-6-sol).\n\nBoth models received the same inputs, questions and answer choices. We required Astra to return a fixed JSON format matching the decisions requested from Jev. Neither model could use tools or look up answers. This let us score the same task for both models. Their probabilities still differ in how they are produced: Jev returns them directly, while Astra is prompted to estimate them.\n\n## 3. Does the paper support the claim?\n\nAgents use literature claims to choose compounds, explain results and plan experiments. A misread table can become an assumption that shapes the rest of the workflow. We tested whether a gate could catch that mistake before the claim was used.\n\nWe used six open-access medicinal-chemistry papers published between July 28 and September 10, 2026. They covered biochemical activity, selectivity and properties relevant to drug development. From them, we wrote 60 claims: 18 supported by the evidence and 42 that either conflicted with it or went beyond it.\n\nFor each question, we gave the model **one claim and related passages, tables and notes from the full paper**. We asked it to return `supported`, `contradicted` or `insufficient`, with a probability for each answer. The correct answer depended only on the supplied evidence. The model did not have to find or read the paper itself.\n\nA simple example is a unit error: **the paper says minutes, but the claim says hours**. \n\nThe number stayed the same, but the changed unit exaggerated the half-life by a factor of 60. An agent carrying that value into a compound comparison would be using the wrong stability result.\n\nThe actual Jev request used model `jev-1.13.0`, the claim, the full evidence packet. The question was:\n\nUsing only `evidence`, classify whether it supports `claim`. Do not use outside knowledge.\n\nBoth models caught the unit error. Jev answered **“contradicted” with 99% probability**; Astra gave the same answer with **99.98%**. Our gate required at least 95% for “supported” to approve a claim, so both stopped this one.\n\nWe compared two ways of checking the claims:\n\nWe also tested claims with added messages demanding approval, instructions hidden in the evidence, or evidence removed. The expected behavior was to judge the evidence, ignore those instructions and withhold approval when support was missing. Neither model approved the unsupported cases at 95%.\n\nThe extra checks made Jev much less willing to approve correct claims. Lowering the hardened gate’s requirement to 80% let 16/18 supported claims through, with no approvals among the 42 unsupported claims in the original set.\n\n## 4. Did the agent make only the requested molecular edit?\n\nAn agent asked to change one part of a molecule must preserve the rest. An unnoticed extra change could send the wrong compound into docking or free-energy calculations. We tested whether the model could distinguish the requested product from one containing an additional error.\n\nWe gave each model a starting structure, a proposed product and an instruction describing the one change allowed. Both structures were SMILES with numbered atoms so the model could match atoms between them. We asked for `approve`, `reject` or `review`, plus the type of error, if any. A correct product should receive `approve`; an extra change should receive `reject` and the appropriate error label. `Review` meant the model could not decide reliably.\n\nWe selected 15 pairs of molecules. For each pair, we submitted both the correct product and a version with an extra error. For example, the hydrocortisone acetate → prednisolone acetate question allowed one single bond to become a double bond. The incorrect product also flipped a stereocentre. The expected answer was **reject: stereochemistry changed**.\n\n## 5. Do the properties match the structure?\n\nAgents pass structures and property records between tools. If a structure changes but its old properties remain attached, later filtering and ranking can use the wrong data. We tested whether Jev could detect that mismatch.\n\n**How we built the tests:** we used 15 SMILES and used RDKit 2025.09.5 to calculate each molecule’s formula, average molecular weight, formal charge, and assigned and unassigned tetrahedral stereocentre counts. Those calculated values became the correct records. We then made an incorrect copy of each record by changing a property or replacing the SMILES while keeping the original properties. This gave us 15 correct and 15 incorrect test packets.\n\n**What the model received:** one SMILES, one set of claimed property values, and the calculation rules, including a molecular-weight. We did not identify the record as correct or incorrect, supply a separate answer key, or let the model call RDKit.\n\n**What it had to return:** two choices, with probabilities for every option:\n\nFor example, we supplied caffeine’s SMILES, `Cn1c(=O)c2c(ncn2C)n(C)c1=O`, with this record:\n\nMolecular weight computed with RDKit was **194.194 g/mol**. We changed only that value in the packet. The expected output was therefore `reject` **+** `molecular_weight`. In another test, we paired phenylbutazone’s SMILES with warfarin’s properties, so the expected output was `reject` **+** `multiple_fields`. \n\n## Results: molecular edits and property records\n\nThe gate approved a submission only when the model assigned at least 95% to `approve`:\n\nAstra chose the correct decision and error label on all 120 questions. The gate still stopped one correct record, miltefosine, because Astra gave `approve` only 78%. Jev let no incorrect submissions through, but it also stopped every correct property record.\n\n### Example: The stereo flip that Jev missed\n\nWe gave Jev **two SMILES: a starting structure and a proposed product** telling it that the **change was only mutating a single bond to a double bond**.\n\nWe deliberately added an extra error to the proposed product: **we also flipped one stereocentre**. We did not tell Jev about the flip. Its job was to compare the two structures and determine whether the product contained only the permitted bond-order change.\n\nThe correct answer was **“reject,” with “stereochemistry” as the error**.\n\nInstead, Jev chose **“approve” with 87% probability** and identified **no error**. It missed the unauthorized stereo flip.\n\n*Jev received only the starting and proposed structures as SMILES, plus the edit rules.*\n\nJev missed the extra change. Astra chose `reject` and identified the stereochemistry error correctly. \n\n## Speed and cost\n\nJev was fast. For the molecular questions, its median response time was **0.34 seconds**, versus **8.75 seconds** for Astra. Astra’s times included starting the Codex CLI and its agent though; Jev’s measured an HTTP request.\n\nThese estimates use September 19 standard prices: Jev charges $0.042 per million input tokens, with free output; Astra charges $10 per million uncached input tokens, $1 for cached input and $50 for output. [Jev pricing](https://docs.typesafe.ai/models) · [Astra pricing](https://developers.openai.com/api/docs/models/gpt-6-astra).\n\n## Can Jev Be Used As a Gate In Agentic Drug Discovery Workflow?\n\n**Jev could be a cheap additional gate alongside an LLM agent, especially for checking whether supplied literature evidence supports a claim.** Its simple literature gate approved 17 of 18 supported claims and stopped all 42 unsupported claims at our 95% threshold. Questionable claims could be sent back for correction before they influence later decisions.\n\n**I would not rely on Jev to detect molecular-structure or property errors.** In these tests, it missed an unauthorized stereo flip and stopped every correct property record. Lowering the threshold let chemistry errors through. It therefore did not provide a useful replacement for explicit chemistry checks.\n\nThe practical role is to complement an LLM’s reasoning with inexpensive evidence checks. This remains a small study of individual checks but was enough to give me an early signal on where Jev is relevant.", "url": "https://wpnews.pro/news/testing-jev-as-a-validation-gate-for-drug-discovery-agents", "canonical_source": "https://frederickparsons.substack.com/p/can-a-fast-ai-gate-catch-chemistry", "published_at": "2026-09-19 19:35:13+00:00", "updated_at": "2026-09-19 19:54:52.914965+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "large-language-models", "ai-research"], "entities": ["Jev", "TypeSafe", "GPT-6 Astra", "deucravacitinib", "CRITIC", "LLM-Modulo", "Gou", "Kambhampati"], "alternates": {"html": "https://wpnews.pro/news/testing-jev-as-a-validation-gate-for-drug-discovery-agents", "markdown": "https://wpnews.pro/news/testing-jev-as-a-validation-gate-for-drug-discovery-agents.md", "text": "https://wpnews.pro/news/testing-jev-as-a-validation-gate-for-drug-discovery-agents.txt", "jsonld": "https://wpnews.pro/news/testing-jev-as-a-validation-gate-for-drug-discovery-agents.jsonld"}}