{"slug": "show-hn-catt-0-2-1-test-agents-services-against-a-muse-instinct-alike-in-ci", "title": "Show HN: CATT 0.2.1: Test agents/services against a Muse/Instinct-alike, in CI", "summary": "A team has open sourced CATT 0.2.1, a Consumer Agent Testing Tool that lets developers test their services against a surrogate Muse or Instinct agent, with runs executable in local builds or CI with or without LLM backing. CATT records full interactions including tool calls and responses so testers can diff what happened from what the agent reports, and it labels its assumptions about assistant-specific behavior as `documented`, `unofficial`, `assumed` or `unknown`; the tool is not affiliated with Muse or Instinct. The release targets reproducibility problems caused by agent memory and account statefulness when building APIs or MCP connectors for proprietary assistants.", "body_md": "Hey HN, we recently open sourced our Consumer Agent Testing Tool to help teams iterate against proprietary assistants like Muse and Instinct. We built this to reduce our own frustrations with building an MCP/API for these agents to consume.\n\nThe problem: testing against these tools is hard! Agent memory + account statefulness make reproducibility, in the ways required for repeatable/reliable outcomes, between agents and the APIs/MCPs they interact with very challenging.\n\nThe solution: CATT lets devs test their services against a surrogate Muse or Instinct agent (not affiliated with either of these companies). Testers define starting conditions, approvals, and success conditions for their service's state. CATT then records the full interaction (incl. tool calls, responses) so it can diff what happened from what the agent reports. Runs can be tested in local builds or via CI, with or without LLM backing.\n\nCATT is designed to emulate the important parts of personal assistant interactions. Its got:\n\n- Tool calls and command-line tool access - Simulated approvals for service actions - Outcomes tracking and comparison - Optional MCP profile based on an unofficial Muse template\n\nThere are limitations to how much we know about different assistant specific behaviors so the tool labels its assumptions as `documented`, `unofficial`, `assumed` or `unknown` for developer review. We'll do our best to keep the tool updated as new insights about the internal workings of these assistants come to light. It is an area of active research and interest, a good example is this recent report on how Instinct uses Exa for web search [0].\n\nAre you building a connector or MCP? If yes, we'd like to hear from you! Have you had assistants behave incoherently, use stale information, or procalim success without evidence? What else should we add tests for?\n\nThe best way to get started is...to point your coding assistant at the repo and ask it to integrate the tool with your CI or local testing pipeline!\n\nHope it's helpful!\n\n[0]: [https://www.sunbeam.ai/thoughts/instinct-is-using-exa-for-we...](https://www.sunbeam.ai/thoughts/instinct-is-using-exa-for-web-search)\n\nComments URL: [https://news.ycombinator.com/item?id=50201860](https://news.ycombinator.com/item?id=50201860)\n\nPoints: 1\n\n# Comments: 0", "url": "https://wpnews.pro/news/show-hn-catt-0-2-1-test-agents-services-against-a-muse-instinct-alike-in-ci", "canonical_source": "https://github.com/Coverage-Cat/consumer-agent-testing-tool", "published_at": "2026-10-22 22:18:40+00:00", "updated_at": "2026-10-22 22:41:05.720493+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "agent-protocols", "developer-tools", "ai-safety"], "entities": ["CATT", "Consumer Agent Testing Tool", "Muse", "Instinct", "Exa", "Sunbeam"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-catt-0-2-1-test-agents-services-against-a-muse-instinct-alike-in-ci", "markdown": "https://wpnews.pro/news/show-hn-catt-0-2-1-test-agents-services-against-a-muse-instinct-alike-in-ci.md", "text": "https://wpnews.pro/news/show-hn-catt-0-2-1-test-agents-services-against-a-muse-instinct-alike-in-ci.txt", "jsonld": "https://wpnews.pro/news/show-hn-catt-0-2-1-test-agents-services-against-a-muse-instinct-alike-in-ci.jsonld"}}