{"slug": "show-hn-experimentation-platform-for-simulating-agents-as-users", "title": "Show HN: Experimentation platform for simulating agents as users", "summary": "Fiveonefour launched AX Cloud, a managed experimentation platform for simulating coding agents as product users to optimize agent experience (AX). The platform runs unbiased experiments in secure sandboxes, supports agents like Claude Code, Codex, and Cursor, and captures rich data including transcripts, test outcomes, and custom metrics. It aims to help developers improve product surfaces such as CLIs, SDKs, APIs, and docs.", "body_md": "# Overview\n\nIntroduction to Fiveonefour and AX Cloud.\n\n[What is Fiveonefour](#what-is-fiveonefour)\n\nFiveonefour is an experimentation platform for understanding and optimizing your product's **agent experience (AX)**.\n\nUse Fiveonefour to run unbiased experiments that simulate coding agents as product users. Measure and improve how actual coding agents discover and operate product surfaces, including CLIs, SDKs, APIs, web apps, MCP servers, docs, etc:\n\n- Design\n[experiments](/0.5.0-rp/experiments)that test variants across prompts, agents, models, products, features, and environments. [Run](/0.5.0-rp/runs)those experiments with simulated coding agent users in secure sandboxes, on demand and at scale- Score agent runs with custom defined\n[tests](/0.5.0-rp/tests)that measure the agents' activities and sandbox output - Analyze the resulting\n[rich data captures](/0.5.0-rp/data-and-results)to extract insights and optimize your product, marketing, and documentation\n\n[AX Cloud](#ax-cloud)\n\nFiveonefour's [AX Cloud](https://app.514.ax/sign-up) offers managed infrastructure, agents, and data to enable your experimentation at scale:\n\n-\n**Managed infrastructure**: for each experiment[Run](/0.5.0-rp/runs), AX Cloud spins up a fresh sandbox (locally or in the cloud). Runs are isolated, with no conversation history, to prevent bias and context pollution. Runs in the cloud are dispatched in parallel so you can get large sample sizes in reasonable timeframes.[Secrets](/0.5.0-rp/secrets)are managed in the cloud and injected securely into each sandbox. After each run, AX Cloud runs the experiment's[Tests](/0.5.0-rp/tests)in a separate sandbox created from a snapshot, which ensures the agent cannot cheat by accessing the tests during the run. Sandbox end-state is retained so you can retest or inspect without re-doing the whole run (and paying for LLM tokens again). -\n**Managed agents**: write your[Experiment](/0.5.0-rp/experiments)once and replicate consistently across agents and models, for apples to apples comparisons. Claude Code, Codex, and Cursor are supported so far as first class citizens in the experiment YAML. Model calls go through Fiveonefour's managed gateway: the sandbox receives a short-lived, run-scoped token and never holds provider API keys. For billing LLM tokens, bring your own LLM provider keys, or use managed model access from Fiveonefour, metered at market prices. -\n**Managed data**: each run captures a[rich data set](/0.5.0-rp/data-and-results#captured-data)including agent transcripts / ACP streams, system activity, test outcomes, and custom metrics like tokens, cost, and wall-clock time. Data is automatically normalized across agent harnesses for consistent analysis, and stored securely in the cloud with managed access. All data is queryable via CLI and MCP, and ready for analysis by your agents with. Analysis worth saving becomes shareable`ax experiment query`\n\n[insights](/0.5.0-rp/data-and-results).\n\n[Common use cases](#common-use-cases)\n\n- Discovery and install: Where do agents hit bottlenecks when discovering or onboarding onto a product?\n[Guide: Test your product's CLI installation](/0.5.0-rp/cli-install-guide) - Core user flows: Where do agents succeed or fail at important jobs-to-be-done?\n- Product optimization: Which variation of an API, CLI command, tool description, or interface helps agents succeed faster?\n- Interface comparison: Do agents perform better with docs, skills, MCP tools, APIs, or another interface?\n- Competitive analysis: What alternatives do agents reach for, and what can you learn from their agent experience? When the mention rate is low, plan for larger n (see Scale the experiment).\n- Product marketing: Can you generate evidence that agents are more effective with your product?\n\n[Where to start](#where-to-start)\n\n[Get started](/0.5.0-rp/getting-started) by installing the CLI and running your first experiment.\n\nExplore best practices for [experiment design](/0.5.0-rp/experiment-design).\n\nStart with a common use case guide: [Test your product's CLI installation](/0.5.0-rp/cli-install-guide)", "url": "https://wpnews.pro/news/show-hn-experimentation-platform-for-simulating-agents-as-users", "canonical_source": "https://docs.514.ax/0.5.0-rp", "published_at": "2026-08-11 17:53:53+00:00", "updated_at": "2026-08-11 18:14:07.681227+00:00", "lang": "en", "topics": ["ai-agents", "ai-products", "ai-tools", "ai-infrastructure"], "entities": ["Fiveonefour", "AX Cloud", "Claude Code", "Codex", "Cursor"], "alternates": {"html": "https://wpnews.pro/news/show-hn-experimentation-platform-for-simulating-agents-as-users", "markdown": "https://wpnews.pro/news/show-hn-experimentation-platform-for-simulating-agents-as-users.md", "text": "https://wpnews.pro/news/show-hn-experimentation-platform-for-simulating-agents-as-users.txt", "jsonld": "https://wpnews.pro/news/show-hn-experimentation-platform-for-simulating-agents-as-users.jsonld"}}