{"slug": "comprehension-gate-for-ai-generated-code-what-sphinx-checks-before-you-ship", "title": "Comprehension gate for AI-generated code: what Sphinx checks before you ship", "summary": "Sphinx, an MIT-licensed Git hook by Parth Bhagat, holds `git push` to `main` or `master` until the developer correctly answers up to five LLM-generated questions about the committed diff, and it drew 252 points on Hacker News on 18 October 2026. At the default level a change of 80 lines and 3 files or fewer gets one question, a medium change two and a large change three, while the `thorough` level asks up to five; paths containing words like `auth`, `billing` or `migration` count as large, Git waits up to 25 minutes, and a pass is remembered for 24 hours. The tool's own README calls it \"practice, not a security gate,\" and in the author's evaluation on five synthetic repositories 33 of 39 grading probes matched the expected verdict in one round and 12 of 14 in the latest.", "body_md": "# Comprehension gate for AI-generated code: what Sphinx checks before you ship\n\nA comprehension gate is a step between \"the agent is done\" and \"the code leaves my machine\" that does not pass until the person shipping can predict what the change does. [Sphinx](https://stackness.dev/tools/sphinx-understand), an MIT-licensed Git hook shown on Hacker News on 18 October 2026 (252 points), is the most elaborate open-source one I found: it holds `git push` to `main` until you correctly answer up to five questions about the diff. It tests you, not the code, and its own README calls it \"practice, not a security gate\".\n\n## What is a comprehension gate?\n\nA comprehension gate for AI-generated code is a check on the person, placed before the code ships: explain or predict what the change does, or it does not go out. The term appears in a 2026 [study of 207 students](https://arxiv.org/html/2604.13277v1) at Karlstad University, where some rewrote AI code before committing it, which the author calls \"explicit imposition of a comprehension gate before committing GenAI code\".\n\nSphinx turned that habit into a tool; its first commit, on 14 October 2026, is titled \"Build Sphinx comprehension gate with adaptive question budgets\". The demand is visible in an [Ask HN thread](https://news.ycombinator.com/item?id=50151036) from 19 October on shipping AI-written code you do not understand, at 592 points and 709 comments two days later. It opened by quoting a LinkedIn post from a Meta engineer asking \"how much I need to understand before I'm comfortable moving forward.\"\n\n## Sphinx holds git push until every question is answered right\n\nSphinx installs as a global Git hook. After `sphinx install`, pushes to `main` or `master` in every repository on the machine wait for a check. A separate reviewer model, which \"does not see the conversation that produced the code\", reads the committed diff and writes questions with private answer keys. You answer each with a prediction and a short reason.\n\n```\nnpm install -g sphinx-understand\nsphinx install          # sets your global core.hooksPath\ncd your-repo\nsphinx init             # writes .sphinxconfig, .sphinxignore, .sphinxcontext.md\nsphinx doctor\n```\n\nAt the default level, a small change (80 lines and 3 files or fewer) gets one question at most, a medium one two, and a large one three; the `thorough` level asks up to five. Paths containing words like `auth`, `billing` or `migration` count as large. A wrong answer brings an explanation and a fresh scenario on the same concept. Git waits up to 25 minutes, and a pass is remembered for 24 hours.\n\n*A still frame from the Sphinx demo, version 0.1.7. Image: [parthh01/sphinx](https://github.com/parthh01/sphinx) by Parth Bhagat, [MIT](https://github.com/parthh01/sphinx/blob/main/LICENSE).*\n\nThe presets write questions with [Claude Haiku](https://stackness.dev/tools/claude-haiku) on your [Claude Code](https://stackness.dev/tools/claude-code) plan or GPT-6 Luna on your [OpenAI Codex](https://stackness.dev/tools/codex-cli) plan; [OpenRouter](https://stackness.dev/tools/openrouter) and any OpenAI-compatible endpoint also work. A typical check makes 5 to 8 model requests plus about one per answer.\n\n## Sphinx tests the person, not the diff\n\nSphinx checks whether you can predict how the code behaves, not whether that behaviour is right. It is not a linter, a test runner or a reviewer, and its questions cover \"behavior, failure modes, and invariants\" rather than file names. A change can pass the quiz with a bug nobody noticed. One commenter in the [Show HN thread](https://news.ycombinator.com/item?id=50137950) listed what lives in the diff instead: \"a leftover debug print, an assertion weakened to match the bug it was meant to catch, an import of a package that never existed\".\n\nThe grader is an LLM. In the author's [own evaluation](https://github.com/parthh01/sphinx/blob/main/eval/RESULTS.md) on five synthetic repositories, 33 of 39 grading probes matched the expected verdict in one round and 12 of 14 in the latest, with errors both ways: a correct answer rejected for not naming a literal `if`, an incomplete one accepted. The author's summary: \"not an all-green semantic evaluation\".\n\nThe gate is easy to step around. `git push --no-verify` skips any pre-push hook and nothing on the machine records it (a required Sphinx check on GitHub would show the commit unstamped), \"Skip this time\" and `sphinx yolo` turn checks off, and an agent could answer for you; Sphinx only tells it not to. The diff, related files and your answers go to the model provider you picked.\n\n## A comprehension gate sits after the agent and before the reviewer\n\nA comprehension gate sits between the agent finishing and a reviewer or CI seeing the change. Tests and CI check the code against an oracle, and review asks whether the code should exist at all. The gate checks only the author, which makes it the cheapest layer and the weakest evidence. It adds to the [six verification layers for agent-written pull requests](https://stackness.dev/blog/alternatives-to-line-by-line-code-review-six-verification-layers-for-agent-written-pull-requests-and) and replaces none.\n\n|  | Comprehension gate | Human code review | CI | \n|---|---|---|---|\n| Checks | The shipper can predict behaviour | The code, its intent and its fit | The code against tests, types and linters | \n| Judge | An LLM with a key written from the diff | A colleague | Deterministic tools | \n| Runs | Locally, before push | On the pull request | On the server, after push | \n| Bypass | `--no-verify` , skip, outside help | Rubber-stamping | Admin override | \n\nJohn Allspaw [wrote](https://www.adaptivecapacitylabs.com/2026/08/24/there-is-more-to-code-review-than-automatable-detection/) in August: \"When an experienced engineer reads a diff and says 'I don't understand this.', their confusion *is* the finding.\" A gate makes the author meet that confusion before a reviewer does.\n\nSphinx is not the first quiz gate. [PR Quiz](https://stackness.dev/tools/pr-quiz), a GitHub Action that quizzes the reviewer on a pull request, dates from June 2025, and I found more than a dozen smaller repositories doing some version of it, about half of them at the pull request. Sphinx can post a pass to GitHub as a commit status for a required check, though that status comes from the pusher's own machine.\n\n## The Brain Sandwich is the manual version of the gate\n\nThe Brain Sandwich is a working rule, not a tool: brain, then AI, then brain. Matheus Lima [described it](https://terriblesoftware.org/2026/10/02/the-brain-sandwich/) on 2 October 2026 and credits it to an engineer on his team. You read and understand the problem before prompting, delegate the build \"with no shame\", then come back to make the work reviewable before anyone else sees it.\n\nHis last slice ends on the rule a comprehension gate can check: \"if you can't explain what's changing and why, it's not ready for someone else to take a look.\" Sphinx automates that check. It cannot do the first slice, understanding the problem before the agent starts, nor judge whether the complexity was worth it.\n\n## How much of an agent-written diff do you need to understand?\n\nHow much of an agent-written diff you need to understand depends on the blast radius, and no study yet says how much a quiz helps. The best measurement of the problem is Anthropic's [January 2026 study](https://www.anthropic.com/research/AI-assistance-coding-skills): 52 mostly junior engineers learning a Python library, where the AI-assisted group averaged 50% on a comprehension quiz against 67% for those who coded by hand.\n\nThat 17-point gap was measured shortly after the task, and was widest on debugging questions. I found no study measuring whether a comprehension gate reduces reverts, incidents or escaped bugs.\n\nThe counterweight is the Claude Code team's [October post](https://claude.dev/blog/solving-code-review-at-anthropic/). Claude writes about 90% of Claude Code, and from June to September merged pull requests \"more than doubled while the share that had to be reverted fell by more than 40 percent\". On understanding: \"Nobody on the team keeps a complete mental model of Claude Code in their head, and we don't think anyone needs to.\" Before changing a module, they ask Claude for an explainer of it, the opposite timing to a pre-push quiz. The numbers are self-reported without counts, from a company that sells the review product; the [post on those numbers](https://stackness.dev/blog/how-much-human-code-review-ai-generated-code-needs-claude-code-numbers) covers the caveats.\n\nThe Ask HN thread split along blast radius: \"Some stateless frontend won't require much beyond visual inspection, but if you introduce some client-server architecture or require payment processing, you must consider the cost of a security intrusion when auditing code changes.\" Sphinx's question budget encodes the same rule.\n\n## A comprehension gate turns into friction for people who already read their diffs\n\nA comprehension gate turns into friction for someone who already designs first and reads every diff, because it asks what they know. One Show HN commenter: \"this hook would either ask me things I know, or it would ask me things I do not care about.\" Another named the workaround: \"Nothing prevents a lazy person from copy-pasting the question to the agent\". A third had built the same thing, \"immediately got frustrated with it and stopped using it.\"\n\nThe case for one is a team where the author is often the only human on a change. From the Ask HN thread: \"Without guardrails around the rate of code production, teams tend toward 0% code understanding.\" Payment, auth and migration paths, juniors and anyone new to a codebase gain the most. Side projects, people who already read every diff, and teams that need a real security control can skip it.\n\n## Where to start with a comprehension gate\n\nStart without a tool. Before you push an agent's change, write the pull request description yourself: why, what changed, and what happens on the failure path. If you cannot, it is not ready. Or open a fresh agent session with no history and ask it to quiz you on the branch. Both sit in front of the [pull request code review](https://stackness.dev/moves/pull-request-code-review) move, not in place of it.\n\nTo try Sphinx, set `level` and `branches` in `.sphinxconfig` for the one repository you care about, and remember that `sphinx install` changes your global Git hooks path; `sphinx uninstall` takes it out. Other [AI tools developers run beside their agents](https://stackness.dev/categories/ai-tools) are in the catalog.\n\n## Sphinx in brief\n\n- **Price:** free and MIT-licensed. The model calls bill your Claude Code or Codex plan, or OpenRouter credits.\n- **Runs on:** macOS, Linux and WSL with Node.js 22.19 or newer. Not native Windows.\n- **Works with:** Claude Code, Codex,[Cursor](https://stackness.dev/tools/cursor) 1.7+,[OpenCode](https://stackness.dev/tools/opencode) 1.x,[Pi](https://stackness.dev/tools/pi) 1.2+, or plain Git.\n- **Version:** 0.1.13 on npm, 21 October 2026, the thirteenth npm release since 0.1.0 on 16 October.\n\n## New posts by email\n\nAbout one email a week with what we published. No account needed.\n\n## Which of these do you use?\n\nNobody on Stackness runs Sphinx yet. Tick what you use and be the first.\n\n[Play Lonny Eagle Amateur Skater 4A pixel art skateboarding game. Signing up opens it straight away.Sign up to play](https://stackness.dev/games/lonny-eagle)", "url": "https://wpnews.pro/news/comprehension-gate-for-ai-generated-code-what-sphinx-checks-before-you-ship", "canonical_source": "https://stackness.dev/blog/comprehension-gate-ai-generated-code-before-shipping-sphinx", "published_at": "2026-10-21 09:47:25+00:00", "updated_at": "2026-10-21 11:28:04.727623+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "ai-agents", "large-language-models"], "entities": ["Sphinx", "Parth Bhagat", "Hacker News", "Karlstad University", "Claude Haiku", "Claude Code", "OpenAI Codex", "OpenRouter"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/comprehension-gate-for-ai-generated-code-what-sphinx-checks-before-you-ship", "markdown": "https://wpnews.pro/news/comprehension-gate-for-ai-generated-code-what-sphinx-checks-before-you-ship.md", "text": "https://wpnews.pro/news/comprehension-gate-for-ai-generated-code-what-sphinx-checks-before-you-ship.txt", "jsonld": "https://wpnews.pro/news/comprehension-gate-for-ai-generated-code-what-sphinx-checks-before-you-ship.jsonld"}}