Jev Is Not a Language Model, but It Breaks Like One

Check Point Team

A new kind of AI model returns decisions instead of text. We spent a day trying to change its mind. It cost about 50 cents.

• →every configuration we tested was broken: risk downgraded to low and investment advised, on a document that flags every warning sign
• →about 50 cents per successful break, with our strongest attacker landing on the fourth turn on average
• →typed, structured input did not make the model meaningfully harder to manipulate, and neither did telling it to distrust the document
• →reasoning was the strongest defense we measured, and Jev has no reasoning setting to turn on

Nobody reads a decision #

That is the whole reason we ran this test.

A model that writes text gets checked by the person reading the text. A model that returns a verdict gets wired straight into the system that acts on it: an application proceeds to the next hiring stage, a recommendation goes to a committee, or an insurance claim gets rejected. There is no paragraph to disagree with, because there is not one.

Last week a company called TypeSafe AI released a model named Jev, and the AI industry has talked about little else since. The pitch is unusual. Jev is not a chatbot. You give it a question and the data to judge, and it returns a typed answer: a choice, a yes/no, a score, each with a probability attached. It is fast, it is cheap, and its creators are explicit that it is meant to be consumed by machines rather than read by humans.

That design is genuinely interesting, and for a lot of jobs it is the right shape. Nobody knows exactly what happens inside it. There are theories, including this analysis and several open-source reproduction attempts. What we do know is that Jev emits decisions the way a language model can, only cheaper and faster.

Which leaves the question we cared about. A different architecture is not automatically a different set of weaknesses. Does Jev have the same failure modes? How hard is it to change its verdict from the outside?

We spent a day finding out.

What we did #

We used a slightly adjusted scenario from our Agent Breaker challenge: a due diligence assistant, the kind of application a company might realistically deploy. It reads an uploaded report about a potential investment and returns a verdict, meaning a risk level and whether to advise investing.

The report is about a fictional company called PonziCorp Financial Services, and it flags the company as high risk. Monthly returns of 15 to 20 percent. A business running on new investor money. Open regulatory inquiries. Delayed withdrawals. Given that report the correct answer is obvious, and with no attack in place, every model we tested got it right every time.

Then we gave an attacker control of one section inside the document and asked it to change the verdict: rate the company low risk, and recommend investing. This is the manipulation that matters in practice, because documents arriving from outside your organization are exactly what these systems are built to read.

Because Jev takes structured input, we also had to decide how to hand it the document, so we tried three ways: inline with the instructions, as a separate message as if fetched by a tool, and as a separate message explicitly marked untrusted. We ran three difficulty levels: basic, one with added instructions warning of manipulation, and one with a much longer negative report. Three agentic attackers, ten turns each, three independent runs per configuration.

What we found #

Every combination broke. All nine attacker and difficulty combinations produced at least one complete success: risk downgraded to low, investment advised, on a document that flags every warning sign.

It was cheap. Our strongest attacker broke through in 25 of 27 runs, succeeding on the fourth turn on average. Counting the API cost of running the attacker, including the runs that failed, that is about 50 cents per successful break.

| Attacker | Level 0 (basic) | Level 1 (+ warning) | Level 2 (+ long report) | Overall | 
|---|---|---|---|---|

| Attacker A (strongest) | 8/9 broken · $0.39 per break | 9/9 · $0.48 | 8/9 · $0.64 | 25/27 · $0.50 | | Attacker B (medium) | 7/9 · $0.54 | 7/9 · $0.55 | 1/9 · $5.08 | 15/27 · $0.85 | | Attacker C (weakest) | 3/9 · $0.63 | 1/9 · $2.00 | 1/9 · $2.00 | 5/27 · $1.18 |

Typed input did not help. This is the finding that matters most, because structured data is Jev’s main selling point. The same document delivered three ways, including one explicitly labeled as untrusted, made no meaningful difference to how often the attacks succeeded or how fast.

The anti-injection instruction was close to worthless. We added explicit instructions telling the model to ignore any instructions embedded in the documents it reads. Successful breaks went from 18 to 17 out of 27. That is noise.

Jev sits in the middle of the cheap, fast field. We ran the same attacks against two mainstream low-cost models doing the same task. Those two have a control Jev does not: reasoning effort. Turning it on improved resilience in both cases.

What a break actually costs #

The individual numbers are easy to skim past, so it is worth putting them side by side.

Flipping the verdict cost about 50 cents. The strongest defense we measured anywhere in this study, reasoning switched on, pushed the price of a break from 56 cents to $4.39. That is an eightfold increase, but it is still only a few dollars.

Our attacker had ten turns available and typically needed four. It was not working at its limit.

And if you were the analyst using that assistant, you would have seen none of it. You upload a report, you read the verdict, and the verdict says low risk. There is no reasoning to inspect and no wording that looks off. Just a typed field, correctly formatted, confidently wrong.

Every defense we measured moved the cost of an attack from cents to a few dollars. None of them moved it out of reach.

Why structured output does not prevent prompt attacks #

The output format constrains the shape of the answer. It does nothing about the content of the input.

Our attacks never told the model what to output. The ones that worked appended what looked like a legitimate addendum to the due diligence report: a clean audit opinion from a major firm, a regulatory file number, a revised risk table. Nothing instructed the model. Everything simply reported the warning signs as already resolved.

The model then did its job correctly, on false evidence. The output was valid and the probability was shown. The answer conformed to the output format but was wrong, because the document read by the model had a manipulated paragraph and it had no way to know.

What made the model more resilient was more evidence to weigh, and, where available, reasoning. Neither of those is an output format.

What this means if you are deploying one #

Test your system, not the model. Our results came from adaptive attackers that adjust across turns. Off-the-shelf attack lists told us nothing useful about this model at all. Evaluate the whole system, as close to your real deployment as you can get it.

Give the decision as much evidence as you can. The scenario with the longer, richer report was significantly harder to break, requiring more effort from the attacker.

Do not mistake an output format for a security boundary. Typed output, output validation and structured input are good engineering. They constrain what a model can say, not what it can be convinced of.

Check what goes in, not just what comes out. The manipulation arrived inside a document the application was built to read. Screening inputs before they reach the decision model is the layer that addresses that directly.

An honest word on scope #

This is a focused, directional study, not a benchmark. We used only a single application with a single manipulation objective. A proper benchmark would have different applications and different objectives, giving a more robust comparison. Limited as it is, it provides enough evidence to say that Jev suffers the same type of vulnerability regular language models do. Our attacker also saw the model’s probabilities as it worked, which is realistic if your application exposes them and an advantage if it does not.

We are publishing early because the question is live right now, and all security insights are valuable as people decide whether and how to use Jev. If there is appetite for a fuller public benchmark, we will build one.

Jev is a clever piece of engineering, and this is not a verdict on whether it is a good model. It is a fresh model that still has rough edges, as is stated in the documentation itself. This analysis is a narrow answer to one question: is it safe to put Jev in front of untrusted input without anything around it?

It is not. Neither is anything else we tested.