Sentinel-IR: Stop Making Your Agents Read Human Code. Give Them a Fact Layer

Live-benchmarked against gpt-6-astra. Raw scorecard and log included.

Every coding agent today does the same wasteful thing: to answer a simple question like "does this merge request touch the network?", it reads the entire source file — hundreds of lines of human-oriented code — and burns thousands of tokens on it.

Source code is written for humans. Comments, formatting, naming style — all of it is noise to an agent that only needs to know what the code does. So we built Sentinel-IR: a compact, machine-oriented intermediate representation that keeps the meaning and drops the noise.

Sentinel-IR is not a new programming language. Nobody writes in it. It's a fact layer: a deterministic extraction of the security-relevant things in a file — the HTTP routes it exposes, the environment variables it reads, the files it writes, the processes it spawns, the surface it exports.

Think of it as the difference between handing your agent a 500-page novel and a one-page factual brief written by a parser that never gets tired and never guesses.

Key properties, in plain language:

content.includes("axios"). A call is a call_expression; a route is a call on app/ router/ server with a string path; an env read is a member access on process.env. That's how a false positive on regexp.exec got eliminated — name matching alone can't tell it from child_process.exec. We ran the benchmark for real: 12 files, 87 questions, 267 actual LLM calls against gpt-6-astra. Same files, same questions, same ground truth for every variant.

The headline: IR with fallback saves 71.3% of input tokens and is more accurate than reading the raw source — 100% vs 96.6%.

Two findings worth more than the headline:

git-probe, token-signer, report-worker), the model answered the IR has near-constant size, so savings scale with file size. Fitted break-even: ~303 source tokens (~34 lines).

Below ~34 lines, IR costs more than the source. We print that in our own output rather than hiding it — the per-file table is part of the scorecard.

The full live run: 263 requests, 395,847 input / 9,239 output tokens, $4.93 total on the org account. Two honest notes:

chars/4 token estimate predicted ~418k input tokens; reality was 396k (

The gap is expression-level: Orbit's graph knows file structure, but not "this line spawns a child process" or "this MR adds a POST route reading an env secret". That is exactly the layer IR fills. (Orbit Remote is unmeasured — it needs a Premium group and a Knowledge Graph: Read token; we don't claim it.)

JavaScript
    ↓
tree-sitter parser
    ↓
AstFacts    — routes / exports / imports / env / calls / risk
    ↓
Sentinel-IR — compact, flat, self-describing projection
    ↓
LLM (your agent)   [fallback: raw source on unresolved]
    ↓
Validator → Simulation → Commit

Everything below the parser is deterministic and local: no network, no LLM, no I/O. libs/core/ast-facts.js walks the AST and classifies real nodes — exec/ fork only count as process-spawning when the callee resolves to child_process/ execa/ zx.

compressFacts in libs/core/sentinel-ir.js — deliberately boring:

function compressFacts(facts) {
    if (!facts?.ast) {
        return { ast: false, reason: facts?.error || "source did not parse; IR fell back to text heuristics" };
    }

    const compressed = { ast: true };
    const put = (key, value) => {
        if (Array.isArray(value) && value.length > 0) compressed[key] = value;
    };

    put("routes", facts.routes);
    put("exports", facts.exports);
    put("imports", facts.imports);
    put("env", facts.env);
    put("operations", Object.entries(facts.operations || {})
        .filter(([, enabled]) => enabled)
        .map(([name]) => name));

    const calls = {};
    for (const [bucket, entries] of Object.entries(facts.calls || {})) {
        if (Array.isArray(entries) && entries.length > 0) calls[bucket] = entries;
    }
    if (Object.keys(calls).length > 0) compressed.calls = calls;

    put("dangerous", facts.dangerous);
    put("riskSignals", (facts.riskSignals || []).map(s => `${s.signal}:${s.evidence}@${s.line}`));

    return compressed;
}

Three design choices worth stealing:

signal:evidence@line, traceable back to the syntax that produced it. ast: true is a completeness contractast: false with a reason, never silently wrong data. The known gap this creates is the interesting part: explicitly-empty categories are omitted, so "are there env vars?" currently resolves to unresolved → escalate rather than provenly no. The fix — emitting explicit empty facts when ast: true — is the single change that would have turned 5 of our live misses into correct answers without touching the fallback. It's on the list.

The full compressed shape, field-for-field faithful to SentinelIR.compress():

{
  "mission":     { "target": "libs/api/server.js", "objective": "network_stability" },
  "world":       { "pressure": 0.5, "confidence": 0.78, "budget": 0.015, "risk": "high" },
  "constraints": ["preserve_api", "avoid_breaking_changes"],
  "forbidden":   ["eval", "child_process"],
  "summary":     { "size": 14203, "lines": 389, "hasCrypto": false,
                   "hasFilesystem": true, "hasNetwork": true },
  "facts": {
    "ast": true,
    "routes":  ["GET /health", "POST /orders"],
    "env":     ["DATABASE_URL", "STRIPE_SECRET_KEY"],
    "operations": ["inboundHttp", "diskWrite"],
    "calls":   { "process": ["spawn(node:child_process)@214"] },
    "riskSignals": ["process_spawn:spawn@214"]
  }
}

(Illustrative values; schema is exact.)

libs/ir-benchmark/runner.js defines "savings at retained accuracy": the best variant that is at least as accurate as reading raw source. If none is, the honest answer is 0% — not a smaller lie.

const candidates = [
    { variant: "ir", stats: ir },
    { variant: "ir+raw", stats: hybrid }
].filter(c => c.stats.correct >= raw.correct);

const best = candidates.sort((a, b) => a.stats.inputTokens - b.stats.inputTokens)[0] || null;
// → savingsAtRetainedAccuracyPct: best ? savings(best.stats) : 0

This run: ir+raw was the only variant at ≥ raw accuracy, so the claimed figure is 71.3% — not the prettier 79.1% that lost accuracy.

A diff shows what changed in text. IR answers what the change does: routes added/removed, env values newly read, fs/network/process operations appeared, exported surface changed, risk taxonomy movement. We dogfood it as a per-MR CI report across our own 44 merged MRs: 35 touched JS, 31 produced facts, median 13 facts/MR, 18 raised a file's risk level — and the job gates: an MR pushing a file to critical fails until acknowledged in .sentinel-gate.json.

npm run ir-benchmark        # offline: info content, upper bound
node scripts/ir-benchmark.js --live --model gpt-6-astra   # what we ran: 267 calls, ~$4.9
npm run orbit-ab            # IR vs GitLab Orbit Local
npm run ir-pipeline-ab      # 0 invariant drift across input modes
npm run mr-report           # per-MR fact report over your own history

All of it is libs/core/sentinel-ir.js + libs/core/ast-facts.js + libs/ir-benchmark/. Tree-sitter is the only runtime dependency. No source leaves your runner.

ir-benchmark-live.json

212.26 kb

Download here:

https://app.devin.ai/attachments/b44c2c3c-9619-4914-a509-02a4e6c59a27/ir-benchmark-live.json