{"slug": "i-have-an-issue-with-what-agets-affirm-do-you-have-too", "title": "I have an issue with what agets affirm, do you have too?", "summary": "Elliot 0.3, an open-source MCP tool from developer EmanueleCell4, ran unattended inside Claude Code on 9 popular open-source AI coding-agent tools in October 2026 and produced 13 findings, of which an independent reviewer confirmed 12 correct and 0 wrong, at 1 to 3 minutes per repository. Elliot checks what an AI agent claims and what a README promises against the actual code, requiring every quote to exist at the cited line and commit, and it caught a tool whose README promised \"No Telemetry\" while shipping default-on analytics with consent pre-set to \"yes\" and another that re-enabled telemetry at login and identified users by email. On the same 9 repos Elliot found 6 of the 10 serious issues the author had found by hand, missing most often a missing authorization check, which the tool does not yet look for.", "body_md": "**Independent verification for code written by AI agents.**\n\n  Elliot checks what your agent claims, and what your README promises, against the code. Every finding comes with proof.\n\n[Quick start](#quick-start) ·\n  [What it found](#what-it-found) ·\n  [How it works](#how-it-works) ·\n  [Commands](#commands) ·\n  [Configuration](#configuration) ·\n  [FAQ](#faq)\n\nIn the repository you want to check:\n\n```\nnpx -y github:EmanueleCell4/elliot-mcp init\n```\n\nRestart your agent, then type:\n\n```\n/elliot-audit\n```\n\nA minute or three later the report is in `elliot-report/<date>-<commit>/` at the root of the repo: `report.html`, `report.md` and `claims.json`. There is nothing to open, no dashboard and no account.\n\nRequirements: Node 20 or newer, git, and Claude Code, Cursor or Codex.\n\nCoding agents say \"Done ✅\" when the work isn't done. They delete failing tests, skip checks, describe security that doesn't exist, and write READMEs that promise what the code never does. CI stays green because tests check the code the agent wrote, not the promises the product makes.\n\nI found this out the hard way. My own legal-tech app had a security page promising role-based access enforced in the database. The data layer bypassed it by design, and every permission check was a function that ignored its input. The agent had told me it was done.\n\nThe usual fix is to ask a second AI to review the first one. That gives you another opinion from the same kind of system that made the mistake. In the first automatic audit I ran, the agent doing the checking cited line numbers that don't exist in 4 of 17 submissions.\n\nElliot works the other way around: **the agent proposes, Elliot verifies.** Every quote must really be in the file, at that line, at that commit. Every \"this doesn't exist\" is searched by Elliot itself. A fresh reviewer then tries to prove each finding wrong before it reaches the report.\n\nElliot 0.3 ran unattended inside Claude Code on 9 popular open-source tools from the AI coding-agent ecosystem (October 2026). A separate reviewer that had not seen the audits then checked every finding against the code.\n\n| Findings in the reports | Confirmed correct | Wrong | Time per repo | \n|---|---|---|---|\n| 13 | 12 | 0 | 1 to 3 minutes | \n\nSome of what it caught:\n\n- A tool whose README says **\"No Telemetry: no data collection or tracking\"** ships analytics that are on by default, with consent pre-set to \"yes\". ([public issue](https://github.com/winfunc/opcode/issues/481) )\n- An app where **logging in turns telemetry back on** after the user switched it off, then identifies the user by email. ([public issue](https://github.com/BloopAI/vibe-kanban/issues/3482) )\n- A security page that calls a script sandbox **\"read-only\"** while the code can give scripts file writes and MCP tools. Confirmed by the project's triage bot, with a docs fix sent upstream. ([issue](https://github.com/dyad-sh/dyad/issues/4879) ,[PR](https://github.com/dyad-sh/dyad/pull/4881) )\n- Docs saying an **API key is stored encrypted** while the code writes it as plain JSON.\n- A popular agent tool that sends users' **full AI prompts and responses to a third-party error tracker by default** , under an option named \"anonymousTelemetry\". Reported privately to the maintainers; details will be published after a fix.\n\nWhat it still misses: on the same 9 repos it found 6 of the 10 serious issues I had found by hand. The most common miss is a missing authorization check (for example, a download endpoint that doesn't check who is asking). Elliot doesn't look for those yet; see the [roadmap](#roadmap).\n\n1. **Risk sweep (Elliot).** Before the agent reads anything, Elliot scans the docs for risky promises (telemetry, privacy, encryption, access control, sandboxes, \"never\", \"by default\") and the code for signals that often contradict them: analytics SDKs, permission bypasses such as`--dangerously-skip-permissions` , privileged database clients, secrets written to disk. The audit cannot finish until every one is addressed or skipped with a reason.\n2. **Read and propose (your agent).** Your agent traces each claim into the code and proposes a verdict, quoting the docs and the code.\n3. **Verify the proof (Elliot).** Each quote is matched against the file at the pinned commit, within two lines of where the agent said it is. Each \"absent\" claim is searched by Elliot with`git grep` . Citations that don't check out are rejected, and the agent has to fix them.\n4. **Materiality test.** A FALSE or PARTIAL verdict must say what a user who trusts the docs would get wrong. Omissions, naming details and tiny delays become notes, not findings. Changelogs, roadmaps and internal developer docs are rejected as sources, because they are not promises to users.\n5. **Independent review.** A fresh subagent that never saw the audit tries to prove every finding wrong, and confirms, downgrades or retracts it. Elliot will not write the report until every finding has been reviewed.\n6. **Report.** HTML, Markdown and JSON in`elliot-report/` , plus a tamper-evident receipt for every check in`.elliot/receipts.jsonl` .\n\nIf you wrote code in the same session, the protocol tells the agent to run the audit in a fresh subagent, so the auditor is not the author.\n\nThe full report is in [`examples/lex-ai-report.html`](https://github.com/EmanueleCell4/elliot-mcp/blob/main/examples/lex-ai-report.html). It was produced by an earlier version, before the independent review step existed.\n\n```\nnpx -y github:EmanueleCell4/elliot-mcp init            # every agent it detects\nnpx -y github:EmanueleCell4/elliot-mcp init --claude   # only Claude Code\nnpx -y github:EmanueleCell4/elliot-mcp init --cursor   # only Cursor\nnpx -y github:EmanueleCell4/elliot-mcp init --codex    # only Codex\nnpx -y github:EmanueleCell4/elliot-mcp init --all      # all three\n```\n\n`init` is idempotent: running it twice changes nothing, and it never overwrites a command file you edited. It writes:\n\n| Agent | MCP server registered in | Command added | \n|---|---|---|\n| Claude Code | `.mcp.json` (project) | `.claude/commands/elliot-audit.md` →`/elliot-audit` | \n| Cursor | `.cursor/mcp.json` (project) | `.cursor/commands/elliot-audit.md` →`/elliot-audit` | \n| Codex | `~/.codex/config.toml` | `~/.codex/prompts/elliot-audit.md` →`/prompts:elliot-audit` | \n\nRun it in your agent from the repository root. In Claude Code and Codex, anything you type after the command is passed to the agent as extra instructions, for example `/elliot-audit focus on the security page`.\n\nThe same server also checks individual claims while the agent works. It ships instructions telling the agent to use them before it says \"done\".\n\n| The agent says | Tool | What Elliot looks at | \n|---|---|---|\n| \"I fixed / cleaned / refactored the code\" | `elliot_snapshot` , then`verify_changes` | the content hash of every file before and after the task | \n| \"It's deployed / live\" | `verify_url` | the HTTP status and the text really served at the URL | \n| \"PR merged\", \"CI is green\", \"pushed\" | `verify_github` | the GitHub API: merge state, check runs, whether the commit exists | \n| \"Published to npm\" | `verify_npm` | the npm registry: version exists, published after a given time | \n| \"What did my agent really do today?\" | `elliot_report` | your receipts: claims checked, how many held up | \n\nThese checks answer `VERIFIED`, `NOT_VERIFIED` or `INCONCLUSIVE`. Inconclusive means Elliot could not confirm, for example because of a GitHub rate limit. It never guesses.\n\n| Tool | Used by | What it does | \n|---|---|---|\n| `elliot_audit_start` | the agent | Pins the commit, lists the files that make claims, runs the risk sweep, returns the protocol. | \n| `elliot_record_claim` | the agent | Records one claim with a verdict. Checks the source quote, every evidence quote and every absence search. | \n| `elliot_record_behavior` | the agent | Records risky code behaviour the docs never mention, with checked code quotes. | \n| `elliot_review_packet` | the agent | Returns a self-contained prompt for a fresh reviewer subagent. | \n| `elliot_record_review` | the reviewer | Confirms, downgrades or retracts a finding. A retraction needs a code quote. | \n| `elliot_audit_finish` | the agent | Refuses to finish while risk-sweep items or reviews are missing, then writes the report. | \n| `elliot_snapshot` | the agent | Fingerprints the working tree before a task. | \n| `verify_changes` | the agent | Compares the tree with a snapshot, or with HEAD. | \n| `verify_url` | the agent | Checks status and expected text at a URL. | \n| `verify_github` | the agent | Checks `pr_merged` ,`checks_passing` or`commit_exists` . | \n| `verify_npm` | the agent | Checks a package version on the registry. | \n| `elliot_report` | you | Summarises recent receipts. | \n\n| Verdict | Meaning | \n|---|---|\n| `VERIFIED` | The code implements the claim. A small gap that misleads nobody is kept as a note. | \n| `PARTIAL` | Implemented, with a gap that would make a user who relies on it do something wrong. | \n| `FALSE` | The code contradicts the claim, or the feature, command, flag or option does not exist. | \n| `UNVERIFIABLE` | Depends on a deployment, a contract or a service outside the repo. | \n| `UNDISCLOSED` | Risky behaviour in the code that the docs never mention. | \n\nUse this if you'd rather not run `init`.\n\n**Claude Code**\n\n```\nclaude mcp add elliot -- npx -y github:EmanueleCell4/elliot-mcp\n```\n\n**Codex** (`~/.codex/config.toml`)\n\n```\n[mcp_servers.elliot]\ncommand = \"npx\"\nargs = [\"-y\", \"github:EmanueleCell4/elliot-mcp\"]\n```\n\n**Cursor** (`.cursor/mcp.json`) and any other MCP client\n\n```\n{ \"mcpServers\": { \"elliot\": { \"command\": \"npx\", \"args\": [\"-y\", \"github:EmanueleCell4/elliot-mcp\"] } } }\n```\n\nThen add the command yourself, or ask your agent to \"run an Elliot claim audit of this repository\". For agents that support skills, also add [`skills/elliot/SKILL.md`](https://github.com/EmanueleCell4/elliot-mcp/blob/main/skills/elliot/SKILL.md).\n\nAll settings are optional environment variables.\n\n| Variable | Default | What it does | \n|---|---|---|\n| `GITHUB_TOKEN` | unset | Used by `verify_github` for private repos and higher rate limits. | \n| `ELLIOT_DIR` | `.elliot/` in the working directory | Where receipts and audit state are stored. | \n| `ELLIOT_GITHUB_API` | `https://api.github.com` | GitHub API base URL, for GitHub Enterprise or tests. | \n| `ELLIOT_NPM_REGISTRY` | `https://registry.npmjs.org` | Registry used by `verify_npm` . | \n\n```\nyour-repo/\n├── elliot-report/\n│   └── 2026-10-22-dd84474/\n│       ├── report.html     # open in a browser\n│       ├── report.md       # paste into an issue or PR\n│       └── claims.json     # everything, for scripts and CI\n└── .elliot/\n    ├── receipts.jsonl      # one tamper-evident receipt per check\n    └── audits/             # audit state\n```\n\nAdd `.elliot/` to `.gitignore`. Commit `elliot-report/` if you want the report to live with the code.\n\nA receipt looks like this:\n\n```\n{\n  \"elliot\": \"0.3.0\",\n  \"id\": \"rcpt_3f9a1c2b7d10\",\n  \"check\": \"claim\",\n  \"claim\": \"Role-based access is enforced at the database layer through row-level security.\",\n  \"target\": \"lex-ai@dd84474\",\n  \"verdict\": \"FALSE\",\n  \"reason\": \"All data access uses the service-role client, which bypasses RLS.\",\n  \"evidence\": { \"source\": { \"file\": \"app/security/page.tsx\", \"line_start\": 14 }, \"evidence\": [ { \"file\": \"lib/supabase/index.ts\", \"line_start\": 10 } ] },\n  \"checked_at\": \"2026-10-21T19:42:11.000Z\",\n  \"checker\": \"local\",\n  \"digest\": \"sha256:…\"\n}\n```\n\nThe digest makes a receipt tamper-evident: change a field and the digest no longer matches. A local receipt is still produced on your own machine, so it proves nothing to a third party. Independently signed receipts are on the roadmap.\n\n**Isn't this just an LLM judging an LLM?**\nNo. The agent proposes verdicts, but every piece of evidence is checked deterministically: quotes against files at a pinned commit, absences with Elliot's own `git grep`. A second agent that never saw the audit reviews every finding, and a retraction has to be backed by a code quote that Elliot also checks.\n\n**Does it send my code anywhere?**\nElliot itself makes no network calls during an audit. It reads files and runs git locally. Your agent reads the code as it normally does, with the model you already use. The `verify_url`, `verify_github` and `verify_npm` checks call those services only when the agent uses them.\n\n**Does it change my code?**\nNo. It writes only `.elliot/` and `elliot-report/`, and `init` writes the config files listed above.\n\n**What does it cost?**\nNothing. Elliot is MIT-licensed and the audit runs on the agent and model you already pay for. An audit of a medium-sized repo took 1 to 3 minutes in our tests.\n\n**Can it audit private repos?**\nYes. It runs where your code is. Only `verify_github` needs `GITHUB_TOKEN` for private repos.\n\n**Can I trust a PASS?**\nTreat a clean report as \"nothing found among the claims checked\", not as a certificate. Elliot is a static reading of the code. It does not run your app yet, and it does not look for missing authorization checks yet.\n\n**I found a false positive.**\nPlease open an issue with the report's `claims.json` entry. False positives are the number Elliot is built to keep at zero.\n\n- File, URL, GitHub and npm checks with local receipts\n-  `/elliot-audit` : claim audit with checked citations, report in`elliot-report/`\n- Risk sweep, undisclosed behaviours, materiality test and independent review\n- Authorization checks: unauthenticated handlers, missing ownership checks\n- Verify the agent's own \"done\" at the end of every task, by running tests and the app\n- Hooks that run Elliot automatically when the agent stops (Claude Code, Codex)\n- A GitHub Action that audits the claims touched by each pull request\n- Independently signed receipts with a public verification link\n\n```\ngit clone https://github.com/EmanueleCell4/elliot-mcp && cd elliot-mcp\nnpm install\nnpm test     # 19 end-to-end tests: spawns the server over stdio and drives it with the MCP client\n```\n\nThe server is plain Node (ES modules) with two dependencies: the MCP SDK and zod. The main pieces:\n\n| File | What it does | \n|---|---|\n| `src/index.js` | MCP server, tool definitions, `init` entry point | \n| `src/audit.js` | Audit state, citation checks, review gate, verdict resolution | \n| `src/risk.js` | Risk sweep: risky doc promises and code signals | \n| `src/audit-protocol.js` | The protocol the agent follows | \n| `src/audit-report.js` | HTML and Markdown reports | \n| `src/init.js` | Registers Elliot in Claude Code, Cursor and Codex | \n| `src/checks/` | `verify_changes` ,`verify_url` ,`verify_github` ,`verify_npm` | \n\nIssues, false positives and stories of agents telling you they were done are very welcome.\n\n[MIT](https://github.com/EmanueleCell4/elliot-mcp/blob/main/LICENSE) © 2026 Emanuele Cella", "url": "https://wpnews.pro/news/i-have-an-issue-with-what-agets-affirm-do-you-have-too", "canonical_source": "https://github.com/EmanueleCell4/elliot-mcp", "published_at": "2026-10-22 03:05:05+00:00", "updated_at": "2026-10-22 03:41:30.634972+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "ai-safety", "agent-protocols"], "entities": ["Elliot", "EmanueleCell4", "Claude Code", "Cursor", "Codex", "opcode", "vibe-kanban", "dyad"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/i-have-an-issue-with-what-agets-affirm-do-you-have-too", "markdown": "https://wpnews.pro/news/i-have-an-issue-with-what-agets-affirm-do-you-have-too.md", "text": "https://wpnews.pro/news/i-have-an-issue-with-what-agets-affirm-do-you-have-too.txt", "jsonld": "https://wpnews.pro/news/i-have-an-issue-with-what-agets-affirm-do-you-have-too.jsonld"}}