Testing Jev for IAM Access Decisions

Trustle evaluated TypeSafe AI's Jev decision model for IAM least-privilege role selection and found it picked the correct 3-permission role (roles/secretmanager.secretAccessor) with 0.99 confidence where Trustle's existing search ranking had put the 2,563-permission roles/iam.securityReviewer first. Trustle said Jev underperformed on broad questions such as "How risky is this role assignment?" and scored 62.6% when asked "is this phishing?", but excelled on narrow, focused signals combined programmatically. The test followed Trustle's measurements that Gemini 3.8-flash cost about $3.08 per 1,000 role-selection decisions, took 3.4 seconds at the median and 11.2 seconds at the 95th percentile per decision, and changed its answer on about 6% of repeated requests.

At Trustle, we manage IAM access for people and agents across cloud and SaaS platforms to achieve least privilege, giving users only the access they need. While LLMs help us improve security insights and navigate IAM complexity, they also bring significant drawbacks. - They’re expensive at scale . In our role-selection evals, Gemini 3.8-flash cost about $3.08 per 1,000 decisions. That is small for one request, but it adds up when you make thousands of decisions a day. - They’re slow . The same evals took 3.4 seconds per decision at the median, and 11.2 seconds at the 95th percentile. This creates a frustrating user experience when users have to wait for an answer. - They can be inconsistent , and they can hallucinate . Asked the same question three times, Gemini changed its answer on about 6% of requests. Our assistant's eval suite even tracks a dedicated failure mode for invented role names. While we have mitigated these drawbacks where possible, limits remain. Consequently, we were excited by TypeSafe AI's launch of Jev <https://en.wikipedia.org/wiki/Jev AI model , a fast, low-cost "System One" decision model. We evaluated whether Jev delivers the accuracy needed for our access decisions. Testing Jev for least privilege analysis For an LLM, you pass it a prompt, and it generates text as an output. You can hold LLMs to a JSON schema, but the answer is still generated text, and any confidence the model gives you is something it wrote, not something actually measured. For example, a prompt might look something like: You help users request cloud access with least privilege. Pick the role from the candidates below that lets the user do exactly what they asked, with the least access beyond that. Answer with JSON: {"role": "<role name or none "}. Request: I need to read the value of the DB PASSWORD secret in project creds-stg. Candidates: - roles/secretmanager.secretAccessor: Reads secret values 3 permissions . - roles/secretmanager.viewer: Sees secret metadata, not values 11 permissions . - roles/iam.securityReviewer: Reads IAM policies everywhere 2,563 permissions . Unlike LLMs, Jev evaluates a given state against defined, typed questions and limits choices to preset options to prevent role invention. Because each answer includes probabilities, you can configure decision thresholds programmatically. The equivalent Jev request is as follows: { "model": "jev-1.14.0", "state": { "request": "I need to read the value of the DB PASSWORD secret in project creds-stg." }, "questions": { "role": { "type": "choice", "instructions": "Pick the role that lets the user do exactly what request asks, with the least extra access.", "criteria": { "roles/secretmanager.secretAccessor": "Reads secret values 3 permissions .", "roles/secretmanager.viewer": "Sees secret metadata, not values 11 permissions .", "roles/iam.securityReviewer": "Reads IAM policies everywhere 2,563 permissions .", "none of these": "No listed role lets the user do what they asked." } } } } With the response being: { "model": "jev-1.14.0", "answers": { "role": { "type": "choice", "choice": "roles/secretmanager.secretAccessor", "confidence": 0.99, "probabilities": { "roles/secretmanager.secretAccessor": 0.99, "roles/secretmanager.viewer": 0.0, "roles/iam.securityReviewer": 0.0, "none of these": 0.01 } } }, "usage": { "input tokens": 433, "output tokens": 79 } } Jev picked the 3-permission role with 0.99 confidence. For a similar request, our existing search ranking had put the 2,563-permission iam.securityReviewer first. During testing, Jev performed worse on broad questions like "How risky is this role assignment?" but excelled when evaluating narrow, focused signals combined programmatically. Similarly, Jev scored 62.6% when asked "is this phishing?", but reached 95.0% when split into five targeted signals and processed via logistic regression. Beri: "TypeSafe's Jev scores 62.6% asked once and 95% split five ways" https://www.beri.net/article/typesafe-jev-typed-decision-model-calibration-decomposition-shadow-eval . Our risk scoring went through a similar arc: | Version | What we changed | What happened | |---|---|---| | v1 | One broad question: "how risky is this role assignment, 0–4?" | 89% of assignments landed on level 3, and nothing scored 0. The scores told us almost nothing. | | v2 | Four narrow questions about the role only: what it can do, whether it can escalate privileges, whether it can read secrets, and what kind of role it is. Scope and usage moved into code. | The scores spread across the whole scale, and the payload no longer contained anything about the user. | | v3 | Real permission lists, but we only sent each role's 25 riskiest permissions. | We overcorrected: 84% of assignments landed on level 4. Showing only the worst permissions makes every role look dangerous. | | v4 | Sent the role's overall access mix and a sample of typical permissions, plus only its 10 riskiest. Escalation and secrets now set a minimum level instead of adding points. | Sensible levels: roles/viewer 2, roles/owner 4, AmazonEC2ReadOnlyAccess 0. | | v5 | Fitted the weights that combine the answers, with cross-validation against an independent reference. | Agreement with the reference rose from 0.75 to 0.86 weighted agreement, on roles held out from the fit . The fit also removed two rules we had added by instinct. | v1–v4 ran on about 4,300 role assignments in our internal staging org. v5 ran on 110 public Google Cloud roles. We also integrated Jev to suggest roles when users ask our agent for help. With proper context on tasks and permissions, Jev quickly identifies the ideal role. When requests become ambiguous, Jev signals low confidence, allowing us to fall back to an LLM effectively. We tested this on 66 access requests with answer keys we wrote and froze before running anything. Requests included vague queries needing follow-up, dual-role needs, and misleading candidate roles. Evaluated on identical role lists across three runs per model: | | Today's ranking top pick | Gemini 3.8-flash | Jev | Jev first, Gemini if unsure | |---|---|---|---|---| | Accuracy | 30% | 98.0% | 91.4% | 98.5% | | Picked an over-broad role | 27% | 1.0% | 1.5% | 0.5% | | Asked a follow-up on vague requests | 14% | 98% | 100% | 100% | | Same answer on all three runs | 100% | 94% | 97% | – | | Decisions sent to Gemini | – | 100% | 0% | 44% | | Median latency | – | 3.4 s | 0.1 s | 0.1 s for most | | Cost per 1,000 decisions | – | $3.08 | $0.10 | $1.45 | While Jev alone was less accurate than Gemini, its confidence scoring was reliable: at ≥0.9 confidence, Jev was 96% accurate. Using Jev for high-confidence decisions and falling back to Gemini for others matched Gemini's overall accuracy, reduced over-broad role selection, cut Gemini requests by over half, and delivered sub-second responses for most queries. A pre-set 0.9 threshold achieved 97.9% accuracy on held-out test cases, outperforming Gemini's 97.2%. Jev proved ineffective for: - Rule-based scenarios where models add no value. - Indirect privilege escalation e.g., Cloud Build Editor running code via privileged service accounts . - Counting tasks e.g., misjudging if 21 permissions exceeded 20 . - Multi-role requests. Jev's performance relies heavily on precise prompting: framing narrow questions, adding a "none of these" option, and replacing numbers with labels significantly improved results. When properly configured, decision models have strong potential for security applications.