{"slug": "intuition-probe-let-an-llm-hallucinate-the-api-ux-it-expects-then-conform-your", "title": "intuition-probe: let an LLM hallucinate the API/UX it expects, then conform your design to the guess. Companion to 'Complicated Thoughts On LLMs' (theocharis.dev)", "summary": "A developer introduced intuition-probe, a tool that uses an LLM to guess the expected shape of an API, config, CLI, or UI before revealing the real design, then recommends conforming the design to the guess. The tool follows the principle 'make the API whatever the LLM guesses' to surface familiar patterns and improve adoption.", "body_md": "| name | intuition-probe |\n|---|---|\n| description | Use when you want to know what shape a developer or user reaches for when they first meet your API, config format, CLI, or UI, so you can make your design that shape. Spawns a blind agent that commits the design it EXPECTS before seeing the real thing, then reports the shape they reached for and conform-first recommendations (make the API whatever the guess is). Triggers on \"is this intuitive\", \"would a developer guess this\", \"intuition probe\", \"blind-test this API/config/CLI\", \"test the DX/UX of\", \"what would someone expect here\", \"make the API whatever the LLM guesses\". |\n\nDiscover the shape a developer (or an LLM) reaches for when they meet your API, config, CLI, or UI for the first time, then make your design that shape. An agent guesses how to use the artifact blind, before it sees the real thing. The guess is not an error to grade; it is a design proposal. Where the guess diverges from your API, the default move is to conform the API to the guess, because the guess is what feels familiar, and familiar is what gets adopted.\n\nThis follows the Jazz principle \"make the API whatever the LLM guesses\": you only\nhear from the few users who push through an unfamiliar design, but an LLM shows you\nwhat people reach for cheaply and at volume. See `references/going-deeper.md`\n\nfor\nthe theory, the conform-first default, and when NOT to conform.\n\nIt surfaces what the model's prior reaches for, the best cheap proxy for what is\nfamiliar to a developer. **At the default N=1 it surfaces CANDIDATES, it does not\nCONFIRM:** a confident divergent guess is a candidate shape to conform to, but a\nsingle draw cannot prove it is *the* familiar default, and a single match does not\nprove your API is familiar. Dial N up; only convergence across agents confirms a\nshape worth conforming to.\n\nStep order matters: **freeze prompt (3) before reading the artifact (4) before\nspawning agents (5)**. Reversing 3 and 4 contaminates the blind test.\n\nAsk the user for: the **artifact under test** (a file path, package name, CLI\ncommand, or URL -- e.g. `plugins/foo/SKILL.md`\n\nor `git commit`\n\n), a **rough\ngoal**, an optional **read-set** (a README, a docs URL, or a snippet), and **N**\n(default 1). If the artifact is a path/package/command, note it; do not read it\nyet.\n\nRestate the rough goal as a pure outcome. Strip any leaked identifiers: method names, config keys, flags, exact labels. Show the user:\n\nSanitized goal: \"\" is that the task? (y / edit)\n\nWait for confirmation. The read-set is NEVER sanitized (a README is allowed to contain the answer; that's the doc-informed test).\n\nTake the template in `references/blind-agent-prompt.md`\n\nand fill its placeholders\n(sanitized goal, system name, read-set if any) into a frozen working prompt that\nyou hold in your own context. Do NOT edit the shipped template file. Also record N\nand the mode (cold / doc-informed) next to the frozen prompt so step 8 has them.\n**Freeze the prompt now and do not revise it after step 4**, so the real\nimplementation cannot leak back into it. If the read-set is a URL, you (the\norchestrator) fetch it now and inline its content into the frozen prompt; the\nblind agent is forbidden from opening URLs, so it must receive the doc text,\nnever a bare link.\n\nRead the real surface from the repo: source/signatures (API), schema/examples\n(config), `--help`\n\n/command defs (CLI), component/route source (UI). If you cannot\nread it, ask the user to paste the real surface. Keep it private; it never goes\ninto the blind prompt.\n\nSpawn N agents via the Agent tool, each with the frozen prompt text from step 3\nas its prompt. Use\n`subagent_type: general-purpose`\n\n. Collect each agent's `decision_points`\n\nJSON.\n(For N=1, one call. For N>1, issue the Agent calls in parallel.)\n\nFor each agent's guess, apply the `references/scorer-rubric.md`\n\nclassification\nagainst the answer key, either inline (you, the orchestrator) or by spawning a\nscorer Agent. Produce the `findings`\n\nJSON. Each finding names the familiar anchor\nthe agent reached for and a conform-first recommendation; the adoption signal is\nread from the agent's recorded confidence (and, at N>1, convergence). Do not invent\nsignal strength from hindsight. If you spawn a separate scorer Agent rather than\nscoring inline, pass it three things: the agent's `decision_points`\n\nJSON, the answer\nkey from step 4, and `references/scorer-rubric.md`\n\n.\n\nGroup findings by decision point across agents. Mark a finding **priority** only\nif at least 2 independent agents hit it. Single-agent findings are listed,\nde-prioritized.\n\nRender per `references/report-format.md`\n\n: lead with the shape the prior reached for\n(the candidate spec) and a conform-first recommendation per divergence. If N is 1,\ninclude the candidate banner and never report the design as a confirmed familiar\ndefault. Offer to save the report to a scratch/artifacts folder. **Never write to the\nrepo under test.**", "url": "https://wpnews.pro/news/intuition-probe-let-an-llm-hallucinate-the-api-ux-it-expects-then-conform-your", "canonical_source": "https://gist.github.com/JeremyTheocharis/83c76da5a10bcf495d4298c70fee91b4", "published_at": "2026-07-14 19:41:33+00:00", "updated_at": "2026-07-16 12:30:33.604980+00:00", "lang": "en", "topics": ["developer-tools", "artificial-intelligence", "large-language-models", "ai-agents"], "entities": ["intuition-probe", "theocharis.dev"], "alternates": {"html": "https://wpnews.pro/news/intuition-probe-let-an-llm-hallucinate-the-api-ux-it-expects-then-conform-your", "markdown": "https://wpnews.pro/news/intuition-probe-let-an-llm-hallucinate-the-api-ux-it-expects-then-conform-your.md", "text": "https://wpnews.pro/news/intuition-probe-let-an-llm-hallucinate-the-api-ux-it-expects-then-conform-your.txt", "jsonld": "https://wpnews.pro/news/intuition-probe-let-an-llm-hallucinate-the-api-ux-it-expects-then-conform-your.jsonld"}}