{"slug": "jev-beyond-the-demos-testing-system-one-models-in-orca-signal", "title": "Jev Beyond the Demos: Testing System One Models in ORCA Signal", "summary": "TypeSafe's early-access System One model, Jev, is drawing developer attention for returning typed decisions from supplied state rather than generating answer strings, according to the company's published account. TypeSafe, co-founded by former OpenAI researcher Diogo Almeida, describes its training approach as reinforcement learning for calibrated decisions (RLCD) and exposes three primitives — Choice, Score, and Noul — with the ORCA Signal team testing Choice and Score on claim-support, evidence-selection, composed-decision, and sentiment tasks as of a September 18, 2026 reporting cutoff. TypeSafe's published material does not disclose a complete neural schematic, training corpus, or parameter count.", "body_md": "Jev has set the AI corner of the internet on fire.\n\nWithin days of its early-access launch, demos and walkthroughs were multiplying across developer feeds: real-time game control, chess experiments, email routing. The [official launch demonstration](https://typesafe.ai/blog/introducing-system-one-models-and-jev), a developer’s [chess harness](https://dev.to/maximsaplin/typesafe-jev-played-chess-and-landed-next-to-reasoning-models-28ga), and this [email-routing prototype](https://github.com/GiesN/typesafe-jev-workflow) give a sense of the activity. Behind them is an unusual interface: send a state, ask a question, get a decision your code can use.\n\nThe company is TypeSafe, co-founded by Diogo Almeida, a former OpenAI researcher whose work on RLHF and InstructGPT helped underpin ChatGPT. The [team’s background](https://typesafe.ai/team) explains some of the attention. Its [manifesto](https://typesafe.ai/manifesto) ends with a line I rather like: “We’re building prod, not God.” In a debate crowded with AGI promises, that is a position a working engineer can do something with.\n\nWe watched the demos too. Then we went straight to the material already moving through our own products.\n\nIn ORCA Signal, that means claims AI assistants make about brands, passages that should support those claims, objections that affect a recommendation, and sentiment that has to be attributed to the right speaker and context. These are small judgments with downstream consequences. Get one wrong and a dashboard can faithfully present a conclusion the evidence never earned.\n\nOur first experiment asked whether a reference supported a claim. Subsequent experiments tested evidence selection, composed decisions, and sentiment. The reporting cutoff here is September 18, 2026. What follows is an early engineering investigation: enough to expose useful behavior and design the next tests, with the limits of the evidence kept visible.\n\nJev is TypeSafe’s first [System One model](https://docs.typesafe.ai/concepts/system-one). The name points to quick, bounded judgments. An application supplies the relevant state and defines the question and permissible outcomes. The model returns a typed result.\n\nThe architectural difference, in TypeSafe’s [published account](https://typesafe.ai/blog/introducing-system-one-models-and-jev), is a parallel sampler that produces decisions without autoregressively generating an answer string. This moves the output contract into the model interface. The familiar “please return valid JSON” instruction disappears from the application prompt.\n\nThat account should be read at the level of detail actually disclosed. It does not give us a complete neural schematic, training corpus, or parameter count. We can explain the public mechanism without pretending to reconstruct the internals.\n\nTypeSafe describes its training approach as reinforcement learning for calibrated decisions, or RLCD. The [AI primer](https://docs.typesafe.ai/introduction/machine-learning-primer) explains the distinction between training for useful decisions and training for generated responses. Calibration concerns how predicted probabilities behave across many labeled examples. Whether that calibration transfers to our tasks still needs measurement.\n\nThere are three primitives. [Choice](https://docs.typesafe.ai/primitives/choice) selects a named outcome and returns its distribution. [Score](https://docs.typesafe.ai/primitives/score) evaluates an ordered rubric and returns a probability-weighted mean of the level indices. [Noul](https://docs.typesafe.ai/primitives/noul) returns a value between zero and one for a proposition. Our experiments used Choice and Score.\n\nA support question can look like this. This is an original illustration of the public TypeSafe request shape, not our production evaluator or the experimental OpenRouter payload:\n\n```\n{ \"model\": \"jev-latest\", \"state\": { \"claim\": \"The Enterprise plan includes SSO.\", \"reference\": \"Enterprise features: SSO, audit logs.\" }, \"questions\": { \"support\": { \"type\": \"choice\", \"instructions\": \"Assess this claim against this reference only.\", \"criteria\": { \"supported\": \"The reference supports the complete factual claim.\", \"unsupported\": \"The reference leaves a factual detail undocumented.\", \"contradicted\": \"The reference explicitly conflicts with the claim.\", \"outside_scope\": \"The statement cannot be assessed as a factual claim.\" } } } }\n```\n\nThe composition rule matters. Questions in one request are evaluated independently against the same state. A second question cannot read the first question’s returned answer. The [documentation](https://docs.typesafe.ai/introduction) makes this explicit. Dependencies belong in code or a subsequent step.\n\nI have used the principle “LLMs classify, Python calculates” in earlier work. Jev fits it neatly. A score is a position on a defined rubric; code decides how that position affects the application. Even the returned confidence is a [statistic derived from the answer distribution](https://docs.typesafe.ai/confidence), distinct from the probability assigned to one option. Neither is a certificate of correctness for our domain.\n\nFor example, a criticism rubric could distinguish no expressed criticism, a mild reservation, clear criticism, and emphatic criticism. Those descriptions occupy indices zero through three. A fractional result is a weighted position on that scale. It does not measure the percentage of unhappy buyers or the financial importance of an objection. The descriptions are part of the measurement instrument; their interpretation and the consequences attached to them are application choices.\n\nA parallel [experiment by Andrea Volpini and the WordLift team](https://lnkd.in/p/eUTcyyKy) makes the role of the supplied world especially clear. In their entity-disambiguation pipeline, Jev was tested only as the final decision layer over a fixed candidate set. Andrea reports 31/37 correct decisions with labels alone and 37/37 in the first run after explicit candidate semantics were added, with similar results across repeated and shuffled runs. The task, gold set, and evaluation design differ from ours, so the figures are not directly comparable. The useful convergence is architectural: the candidate world and the option descriptions are part of the measurement instrument.\n\nORCA Signal examines how AI assistants represent brands. Visibility, associated attributes, objections, sentiment, and factual alignment answer different questions. A prominently recommended brand can still be described with an invented feature or an undocumented qualification.\n\nThe Truth Analysis Index addresses factual alignment. At a high level, we identify claims in generated answers, associate them with suitable reference material, evaluate their relationship to that material, and aggregate admissible outcomes into an inspectable assessment. The useful object behind the index is the claim and its evidence trail.\n\nA supported claim, an undocumented claim, a contradicted claim, and a statement outside the applicable factual assessment need different treatment. If a reference omits a feature, that omission alone does not prove the feature absent in the world. If the statement is advice or speculation, forcing it into a factual verdict can manufacture an apparent error.\n\nJev could contribute to that semantic classification. Source selection, reference suitability, provenance, application checks, and aggregation remain work the system must perform. Jev’s Score primitive is not the Truth Analysis Index formula.\n\nThere is a related opportunity in Brand Perception Market: determining which attributes an answer actually associates with which brand. That judgment describes the answer. Verifying whether the association is factually justified is another task.\n\nBrand Performance adds a further boundary. Counts, rates, and comparisons with consistent denominators should remain deterministic calculations. A model might help check whether accompanying commentary faithfully describes those facts. We have designed that possibility, but did not run a matching commentary experiment in this corpus.\n\nThese distinctions are practical. They let us ask where a decision model can improve a workflow without handing it responsibility for the entire measurement.\n\nFor the initial Truth Analysis Index pilot, we used existing material from a professional-services workflow. The historical corpus contained 197 eligible unique claim/reference pairs. We selected sixteen deterministically, stratified by ORCA’s saved final verdict: six supported, six unsupported, and four outside the applicable verification rubric. The sampled final verdicts contained no contradiction cases.\n\nEach pair went through three repetitions, yielding 48 valid responses from 49 attempts. Those repetitions tell us about observed stability. They do not turn sixteen cases into 48 independent examples.\n\nWe froze the sample, reference content, configurations, and provenance before inference, then retained request and response records. The runs used [Jev 1.13 through OpenRouter](https://openrouter.ai/typesafe/jev-1.13), with the resolved model version recorded. The provider exposes a decisions interface; a chat-completions integration is not interchangeable with it.\n\nThe reference packets were provisional and still needed review. The saved ORCA labels were automatic comparator outputs. Nobody had independently annotated these sixteen cases into a human gold set.\n\nThere was another limitation: we did not rerun the existing evaluator. Its original context and rubric differed from Jev’s simpler claim/reference configuration. This pilot therefore compares component outcomes under those conditions, rather than isolating model choice in a controlled head-to-head benchmark.\n\nA reproducible sample gives us something concrete to inspect. Human adjudication must establish what that sample should teach us.\n\nJev agreed with ORCA’s final verdict on thirteen of sixteen cases, or 81.25%. Agreement with the raw classifier output was twelve of sixteen, or 75%. Every Jev verdict stayed unchanged across all three repetitions.\n\nThe following table counts distinct cases, using ORCA’s final automatic verdict as the comparator:\n\nThe most informative disagreement involved added detail. A reference documented a result secured by a firm. The claim repeated that result and added circumstances the reference did not establish. Jev selected supported, with 0.95 probability on that option and a returned confidence of 0.93. ORCA’s final verdict was unsupported.\n\nThe plausible core of the claim appears to have carried its unverified detail along with it. That is my interpretation of the observed decision, not a demonstrated account of Jev’s internal reasoning. The case needs human review, and the claim might still be true elsewhere. Against this supplied reference, however, the added specificity was the issue.\n\n*Figure 3. An anonymized adaptation of the pilot’s central disagreement: a documented result does not establish every circumstance added to a claim about it.*\n\nThe other two final-verdict disagreements concerned the boundary between unsupported content and statements outside the rubric. We should adjudicate that boundary before counting either system’s answer as an error. A further review case showed why raw and final verdicts differ: ORCA’s initial contradiction label became unsupported after application checks; Jev also selected unsupported.\n\nThe operational observations were encouraging. Successful calls had a median latency of approximately 0.446 seconds and a p95 of 0.629 seconds. Reported cost for the 48 valid responses was about $0.00124. One HTTP 520 took roughly 40.7 seconds and is excluded from those successful-call percentiles; its billed cost was unknown. We measured no corresponding baseline cost or latency.\n\nThe interface worked. The next question was what happened when decisions had to become usable application objects.\n\nThe extended automatic session produced 701 valid decisions in 705 attempts across separate diagnostics. These were repeated judgments over selected cases and variants, not hundreds of independently labeled claims.\n\nFirst, we tested alias ordering in brand–attribute judgments. Four previously sensitive pairs were evaluated under five list-order variants, with ten draws per configuration. For one pair, attribution appeared in one of ten base draws and nine of ten after reversing only the target attribute’s aliases. Membership and task stayed the same.\n\nThat is enough to require an invariance test. It does not identify a neural mechanism or estimate typical performance: the cases were selected development examples, and the schedule did not balance order and timing. A stable label under one presentation can become a different stable label under another.\n\nEvidence selection exposed a more consequential distinction. Seven diagnostic pairs contained 81 candidate passages. Three repetitions produced 243 passage-membership decisions. Agreement with the automatic baseline was 223/243, or 91.77%.\n\nThat attractive number was dominated by 201 agreements on passages the baseline did not select. Of 42 baseline-selected passage events, Jev selected 22. Neither figure is accuracy or recall against human gold.\n\nWe then assembled the passages into 21 evidence sets: seven pairs, three repetitions. Only six passed the existing authenticity and literal-binding checks. Five were empty; ten more lacked the required literal binding. Passing those checks establishes particular properties of a selected passage, not that it adequately supports the entire interpretation.\n\nThere was also an integration defect. Our experimental adapter omitted separately required identity information, so none of the 21 sets satisfied the complete application contract. That is an adapter limitation. Reporting it as 21 demonstrated semantic failures by Jev would misdescribe the experiment.\n\nConsider two authentic excerpts: one names a brand, while another describes rapid setup for a different product. Both can be relevant to the surrounding comparison. Together, they still do not establish that the named brand offers rapid setup. This is why a passage classifier’s agreement cannot substitute for evaluating the relationship the final collection is meant to support. We need to test the model’s selection and the adapter’s ability to preserve it as separate responsibilities.\n\nThe Objections replay tested composition from another direction. We asked separately about suitability and the consequence of an existing objection. Twenty of 72 repeated compositions failed the preserved cross-field or evidence requirements and remained unresolved. Individual typed answers can fit their own schemas while their combination makes no sense.\n\nAn answer that says a product was not evaluated cannot casually coexist with fields describing an established recommendation about its suitability. Negative wording does not, by itself, establish a qualifying objection. A reported objection can also contain a false premise: identifying its role in a recommendation and verifying its truth remain separate jobs.\n\nA small reference-applicability diagnostic produced stable decisions on 23 of 24 pairs across three draws. It covered only two reference scopes and had no human labels. Approved source metadata still does not prove that a particular premise matches that source’s proposition and scope.\n\nEach test pushes evaluation one step outward: label, passage, collection, composed outcome. That is where the application starts to matter.\n\nThe final session tested Score alongside Choice. Across 304 successful requests, Jev returned 1,552 question answers: 598 Scores and 954 Choices. We observed no HTTP errors, missing answers, or retries. Reported total cost was approximately $0.0291. The caller was serial; this was not a concurrency or end-to-end throughput benchmark.\n\nThe live Score contract worked: scores, probability distributions, legends, and confidence were present. That is a useful technical result. A number on a sentiment rubric still needs an identifiable speaker, target, and passage that actually expresses the judgment.\n\nWe compared a local mention window with the full answer for 24 real brand/answer units, repeating each condition three times. Orientation changed in 39 of the 72 paired comparisons. Ninety of 144 real projections passed the application checks, giving 62.5% usable coverage. That percentage measures usable outputs under those checks, not sentiment accuracy.\n\nSome local windows had a preprocessing defect: sentence splitting cut URLs at periods, fragmenting the text around the first mention. That behavior existed before this experiment. Changing to the full answer changed both context and exposure to that defect. We cannot assign the whole difference to model quality.\n\nFull answers introduce their own problems. Consider an assistant reporting that customers call a tool difficult, then explaining why that criticism does not apply to the buyer’s use case. Who owns the negative view? Does the assistant adopt it? What does the recommendation conclude? Those are distinct judgments.\n\nAspect-level analysis narrowed the usable material further. Only two of sixteen real brand–aspect pairs yielded coherent adopted evaluations under the tested setup. A passage around the first brand mention is not a general-purpose evidence selector for every aspect. The relevant evaluation may occur later, or never occur.\n\nThe synthetic absent-aspect fixtures behaved usefully: the application suppressed a raw score when the aspect was not applicable. We should preserve that distinction in the product. Missing or insufficient evidence must not silently become neutral sentiment or a zero score.\n\nThe clearest illustration came from objections. In synthetic contrasts, emphatic criticism with no exclusion consequence scored approximately 2.98–2.99 on our criticism rubric. A politely expressed requirement-based exclusion scored about 1.73–1.78. The stronger tone was not the stronger constraint on the choice.\n\nOn the real objection occurrences, consequence agreement with the saved automatic comparator was 21/36, or 58.33%. That is an adjudication queue, not an established accuracy score or an improvement over a different earlier sample.\n\nBatching was promising in a tiny technical test: median HTTP latency was about 0.410 seconds for one question and 0.438 seconds for ten. Only two synthetic texts were used, question mix varied, and cache conditions were uncontrolled. The middle batch sizes were not monotonic. We have evidence of feasibility, not a general scaling law.\n\nWhat I take from this session is concrete: choose the evidence boundary before arguing about the classifier.\n\nThis brings me to NobleSEO, a project I work on with [Helium SEO](https://helium-seo.com/), and to ORCA’s Content Studio/Content Factory.\n\nThe [LangChain article on building a Jev harness](https://www.langchain.com/blog/building-a-harness-with-jev) shows how a decision model fits into an existing workflow. LangGraph can continue managing state and transitions while Jev supplies a bounded semantic judgment.\n\nIn NobleSEO’s article-writing graph, an optional model-based ContentAnalyzer evaluates a draft before refinement. The SEOReviewer performs deterministic Python checks. Calling both “review agents” would obscure where inference cost actually exists.\n\nA proposed Jev node would sit after drafting and before the ContentAnalyzer, where that analyzer is enabled. It could ask separately whether the draft addresses the requested intent, covers the brief’s essential comparisons, and follows the required structure.\n\nPicture a comparison article with all the right headings and keywords, but no explanation of the trade-off the buyer came to understand. A bounded decision might detect that omission before a full analysis. To make the route useful, the application would need a predefined repair action or a richer reviewer capable of explaining what to change.\n\nORCA’s Content Factory has a different seam: an enhanced draft enters a model-based ReviewAgent, with a bounded enhancement loop. Here, Jev could first detect a narrow class of obvious defects and trigger an early repair. Full review would remain part of that proposed path.\n\nA reassuring decision is not publication approval. If NobleSEO’s analyzer is disabled, adding Jev adds work rather than avoiding that analyzer’s cost. Quality must also be assessed after refinement and downstream assembly, when the reader’s actual article exists.\n\nWe checked historical material, but did not establish the matching draft, brief, source, and review packets needed for a controlled replay. No editorial gate was run, no reviewer was bypassed, and no quality gain or cost saving was measured. The first implementation should collect shadow decisions while the current workflow continues.\n\nThe next step is to turn the frozen pilot cases into a small human-adjudicated gold set, then evaluate on a separate held-out corpus. These sixteen cases are development material now. We have looked at them too closely to pretend otherwise.\n\nFor claim verification, two independent reviewers should assess source suitability, support for every factual detail, genuine contradiction, and statements outside the rubric, then adjudicate disagreements. We need contradiction examples and broader reference scopes. Frozen provenance makes a judgment auditable; reviewers establish whether the judgment is justified.\n\nReviewers should not see either model’s prediction before labeling the material. An adjudicated disagreement may reveal a problem in the historical comparator, in Jev, or in the task definition. All three are useful findings. We should report results by class and reference readiness, rather than allowing easy cases to dominate one reassuring percentage.\n\nFor brand associations, evidence selection, sentiment, and objections, the annotation target changes. Does the answer attribute an aspect to this brand? Which passage establishes the relationship? Is criticism adopted, reported, or rejected? Does it qualify a recommendation or exclude the product? A shared packet can contain several labels, each with its own definition.\n\nThe next comparison should give competing components identical evidence and rubrics. Context preparation deserves its own experiment. Alias-order variants should use a balanced schedule and fresh cases. Related answers, article versions, and session variants must stay in the same data partition, so familiar material cannot masquerade as a new test case.\n\nFor sentiment, that means verified, intact passages local to the relevant aspect, supplied consistently to each compared component. Calibration and routing thresholds belong on separate development data. The holdout should measure the resulting policy, including how often it withholds judgment and whether the accepted decisions are reliable enough for their intended use.\n\nSBI deserves a place here too. Its conversational Truth Analysis Index, cTAI, concerns factual alignment inside buyer–seller exchanges. A useful starting task would be a bounded product-specification claim or a plan limitation, followed by a test of whether a later answer resolves the same reservation. We prepared five structural conversation packets; we did not run that temporal experiment or establish that the paired passages addressed an identical requirement. That binding must come first.\n\nFor editorial triage, I would compare the current workflow, Jev in shadow mode, and a narrowly defined routing policy. Reviewers should judge the final articles blind to the route. The measurement should include missed defects, unnecessary rewrites, total inference cost, total latency, failure tails, and review effort. Our roughly forty-second HTTP failures belong in that evaluation alongside the attractive medians.\n\nThe operational decision is how much authority each tested task has earned: continue, gather evidence, repair, or escalate. False support for an invented detail and a missed exclusion condition deserve explicit attention. Coverage and accuracy should be reported together, keeping unresolved outcomes visible.\n\nJev gives semantic judgment a software interface I want to keep exploring. Our early runs confirm that the interface is practical and expose concrete places where it might help. The work ahead is to connect each decision to suitable evidence, an adjudicated expectation, and a consequence we are prepared to accept.\n\nThat is how “building prod” becomes more than a good line in a manifesto.\n\n[Jev Beyond the Demos: Testing System One Models in ORCA Signal](https://pub.towardsai.net/jev-beyond-the-demos-testing-system-one-models-in-orca-signal-0a343ce3341a) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/jev-beyond-the-demos-testing-system-one-models-in-orca-signal", "canonical_source": "https://pub.towardsai.net/jev-beyond-the-demos-testing-system-one-models-in-orca-signal-0a343ce3341a?source=rss----98111c9905da---4", "published_at": "2026-09-20 00:01:02+00:00", "updated_at": "2026-09-20 00:22:41.432280+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-research", "ai-products", "ai-startups"], "entities": ["TypeSafe", "Jev", "Diogo Almeida", "OpenAI", "ORCA Signal", "System One", "RLCD", "InstructGPT"], "alternates": {"html": "https://wpnews.pro/news/jev-beyond-the-demos-testing-system-one-models-in-orca-signal", "markdown": "https://wpnews.pro/news/jev-beyond-the-demos-testing-system-one-models-in-orca-signal.md", "text": "https://wpnews.pro/news/jev-beyond-the-demos-testing-system-one-models-in-orca-signal.txt", "jsonld": "https://wpnews.pro/news/jev-beyond-the-demos-testing-system-one-models-in-orca-signal.jsonld"}}