How Auto Review works in Bionic Bionic introduced Auto Review, a new shell command approval mode that uses a deterministic Shell Judge to automatically approve up to 82% of commands without LLM review, falling back to a Shell Reviewer subagent for uncertain cases. The feature aims to reduce the burden of manually approving every command an agent runs. How Auto Review works in Bionic Choose Auto Review from Bionic's command approval menu. Are you tired of sitting in front of the computer and approving every command the agent wants to run without really reading its contents? You're in luck. Today we're introducing a new shell command approval mode called "Auto Review" in Bionic. Read on to learn what AST parsing, capability extraction, and command matching have to do with it. Auto Review Pipeline In Bionic's new Auto Review, each command the agent wants to run is passed through the "auto review pipeline". At a high level, the command is first sent to the Shell Judge : a purpose-built, deterministic shell command analyzer with an extensive list of known safe command-and-argument combinations. This part does not involve an LLM yet. If the Shell Judge determines that a command is definitely safe, the command is allowed to execute immediately. But if not, the command is passed to the second stage, where a separate reviewer subagent, the Shell Reviewer , examines the transcript of the main session and determines whether the command is allowed. This pipeline is illustrated below: The Auto Review pipeline parses supported shells, extracts capabilities, and applies safe rules before falling back to the reviewer agent. I. Shell Judge Using an LLM to review every command can get expensive quickly. To counter that, we introduce the Shell Judge: a subsystem whose job is to accept as many "safe" commands as possible without having to send them for LLM review. It does this by parsing the command into an AST, extracting its capabilities, and matching it against a set of known safe commands. Safe or Unsafe? For humans, reviewing commands is an arduous and time consuming task, made worse by the fact that it's now common for frontier models to use complex commands in order to save tokens/requests. Here are some examples we captured from our own real usage: git status --short --branch && git diff --check main...HEAD && base=$ git merge-base HEAD main && echo "merge-base=$base" && test "$base" = "$ git rev-parse main " && if git diff "$base"..HEAD -- . | grep -i -E 'try in chat|try-skill|onTryInChat'; then echo 'Unexpected Try in Chat diff found.'; exit 1; else echo 'Try in Chat diff audit: clean'; fi And another example: set +e output=$ npx tsc -p tsconfig.json --noEmit --incremental --tsBuildInfoFile .scratchpad/tsc.tsbuildinfo 2 &1 exit code=$? error count=$ printf '%s\n' "$output" | grep -c 'error TS' changed file errors=$ printf '%s\n' "$output" | grep 'benchmarks/bionic-agent/cases/rent-screenshots-to-xlsx' || true printf 'exit=%s errors=%s\n' "$exit code" "$error count" if -n "$changed file errors" ; then printf '%s\n' "$changed file errors"; else printf 'changed-file-errors=0\n'; fi exit 0 And here is a PowerShell example: php $files = Get-ChildItem electron/src -Recurse -File -Include .ts, .tsx; $matches = $files | Select-String -Pattern 'getCommittedChat\ '; $tests = @ $matches | Where-Object { $ .Path -match '\.test\.tsx?$' } ; $prod = @ $matches | Where-Object { $ .Path -notmatch '\.test\.tsx?$' } ; "all=$ $matches.Count prod=$ $prod.Count tests=$ $tests.Count test files=$ @ $tests.Path | Sort-Object -Unique .Count "; $prod | ForEach-Object { "$ $ .Path :$ $ .LineNumber :$ $ .Line.Trim " } As a reminder, Bionic is ZDR Zero Data Retention by default. The above examples were captured on our own internal test devices. We neither collect nor analyze users' data. As crazy as these commands look, they are actually safe to run. I am happy to report that, as of today, all of these commands can be automatically approved by the Shell Judge using mechanical analysis without using an LLM. In fact, as of its current version, the Shell Judge can automatically approve up to 82% of all the commands my agent runs - anecdotal, but a signal nonetheless. But how do we mechnically determine that a command is "safe" to run? It should be clear that we can't simply have a giant set of strings representing all the "safe commands" and try to match those. Also, since those shell commands require at least a context-free grammar, regular expressions won't be enough either. In fact, even the same command can be safe or unsafe depending on its exact usage. For example: This is safe target="notes.txt"; echo "done" $target This is not safe target="/etc/passwd"; echo "done" $target We need something more sophisticated. This means we must parse the command and analyze the behavior of the script. So we came up with a three-step process to analyze shell commands: - AST Parsing - Capability Extraction - Command Matching With the Shell Judge supporting the following shells: sh bash zsh PowerShell In case you didn't know, on macOS, Bionic prefers zsh . On Windows, it prefers Git Bash if it is installed. Otherwise, we fall back to PowerShell and then cmd . Shell Judge, Step 1: AST Parsing First, we parse the command into an AST https://en.wikipedia.org/wiki/Abstract syntax tree , a data structure that allows much easier inspection of a program. Well, building a parser and keeping it updated is a daunting task. We need to use some third-party libraries. For sh , bash , and zsh , we use mvdan/sh https://github.com/mvdan/sh . For PowerShell , we use PowerShell itself https://learn.microsoft.com/en-us/dotnet/api/system.management.automation.language.parser?view=powershellsdk-1.1.0 to parse commands into an AST. No, I did not know this was possible before working on this. Shell Judge, Step 2: Capability Extraction Once we have obtained the AST, we use a dedicated extractor to create a common representation we call ShellCapability , which essentially answers the question, "In the worst-case scenario, what can this shell command do?" Since we have access to the AST, this step is easier, though "easier" does not mean "easy." This step has a lot of details, so I won't list them all. Some examples: - This is strictly an "allowlist." For example, if we see any kind of AST structure we don't recognize, we immediately report an "unknown" and reject the command. Thus, we only allow commands we fully understand. - If a shell command exports an environment variable or sets an environment variable for a command, we immediately reject it. This is because many environment variables can fundamentally change the behavior of a command and enable arbitrary command execution GIT EXTERNAL DIFF='touch /tmp/pwned ' git diff . While we could have an allowlist, according to our team's data, few safe environment variables have ever been set by the agent. Thus, we decided to reject all environment variables for now. - Even local variable assignments may be dangerous. If a regular assignment assigns a value to an existing environment variable e.g., PATH=test , even if there is no export , that assignment will change the environment variable and must therefore be rejected. Consequently, the judgment of a command also depends on the current environment variables. - We use an internal concept called "finite alternatives." For each variable, we collect all the values it can possibly have. If it is repeatedly assigned in a loop, we give up and turn it into "full dynamic," meaning we don't know what it is. However, to prevent an exponential explosion, we limit the number of alternatives we track to 1,000. - Commands like echo $unknown must be rejected if we cannot fully evaluate all the possible values of unknown . This is because unknown could be "/some/secrets/ " , which could perform pathname expansion and reveal files inside the secret directory. For example, given the command: base=$ git merge-base HEAD main git diff $base changes.patch The shell capability extracted is the following: { "hasUnmodeledCwdChange": false, "potentialEnvironmentVariablesAssigned": , "potentialCommands": { "command": { "type": "literal", "text": "git" }, "args": { "type": "literal", "text": "merge-base" }, { "type": "literal", "text": "HEAD" }, { "type": "literal", "text": "main" } , "id": 0 }, { "command": { "type": "literal", "text": "git" }, "args": { "type": "literal", "text": "diff" }, { "type": "dynamic", "valueAlternatives": { "type": "literal", "text": "" }, { "type": "commandOutput", "commandId": 0 } } } , "writeTargets": { "type": "literal", "text": "changes.patch" } , "readTargets": , "unknowns": } Some observations: writeTargets and readTargets are additional read/write targets that the shell itself uses, in addition to those accessed through a command.- Commands in interpolations are also captured as "potential commands." This includes any kind of "nested" command. We can guarantee that we don't miss any "hidden commands" because we walk the AST. - When tracking finite alternatives, an alternative can be "the output of a command." For example, the value after git diff is tracked as the output of the command git merge-base HEAD main matched via commandId . - You may be curious why there is a literal alternative of "" . This is because if git merge-base HEAD main fails, $base will be empty, in which case we must also guarantee that git diff