Show HN: Experimentation platform for simulating agents as users

Fiveonefour launched AX Cloud, a managed experimentation platform for simulating coding agents as product users to optimize agent experience (AX). The platform runs unbiased experiments in secure sandboxes, supports agents like Claude Code, Codex, and Cursor, and captures rich data including transcripts, test outcomes, and custom metrics. It aims to help developers improve product surfaces such as CLIs, SDKs, APIs, and docs.

Overview Introduction to Fiveonefour and AX Cloud. What is Fiveonefour what-is-fiveonefour Fiveonefour is an experimentation platform for understanding and optimizing your product's agent experience AX . Use Fiveonefour to run unbiased experiments that simulate coding agents as product users. Measure and improve how actual coding agents discover and operate product surfaces, including CLIs, SDKs, APIs, web apps, MCP servers, docs, etc: - Design experiments /0.5.0-rp/experiments that test variants across prompts, agents, models, products, features, and environments. Run /0.5.0-rp/runs those experiments with simulated coding agent users in secure sandboxes, on demand and at scale- Score agent runs with custom defined tests /0.5.0-rp/tests that measure the agents' activities and sandbox output - Analyze the resulting rich data captures /0.5.0-rp/data-and-results to extract insights and optimize your product, marketing, and documentation AX Cloud ax-cloud Fiveonefour's AX Cloud https://app.514.ax/sign-up offers managed infrastructure, agents, and data to enable your experimentation at scale: - Managed infrastructure : for each experiment Run /0.5.0-rp/runs , AX Cloud spins up a fresh sandbox locally or in the cloud . Runs are isolated, with no conversation history, to prevent bias and context pollution. Runs in the cloud are dispatched in parallel so you can get large sample sizes in reasonable timeframes. Secrets /0.5.0-rp/secrets are managed in the cloud and injected securely into each sandbox. After each run, AX Cloud runs the experiment's Tests /0.5.0-rp/tests in a separate sandbox created from a snapshot, which ensures the agent cannot cheat by accessing the tests during the run. Sandbox end-state is retained so you can retest or inspect without re-doing the whole run and paying for LLM tokens again . - Managed agents : write your Experiment /0.5.0-rp/experiments once and replicate consistently across agents and models, for apples to apples comparisons. Claude Code, Codex, and Cursor are supported so far as first class citizens in the experiment YAML. Model calls go through Fiveonefour's managed gateway: the sandbox receives a short-lived, run-scoped token and never holds provider API keys. For billing LLM tokens, bring your own LLM provider keys, or use managed model access from Fiveonefour, metered at market prices. - Managed data : each run captures a rich data set /0.5.0-rp/data-and-results captured-data including agent transcripts / ACP streams, system activity, test outcomes, and custom metrics like tokens, cost, and wall-clock time. Data is automatically normalized across agent harnesses for consistent analysis, and stored securely in the cloud with managed access. All data is queryable via CLI and MCP, and ready for analysis by your agents with. Analysis worth saving becomes shareable ax experiment query insights /0.5.0-rp/data-and-results . Common use cases common-use-cases - Discovery and install: Where do agents hit bottlenecks when discovering or onboarding onto a product? Guide: Test your product's CLI installation /0.5.0-rp/cli-install-guide - Core user flows: Where do agents succeed or fail at important jobs-to-be-done? - Product optimization: Which variation of an API, CLI command, tool description, or interface helps agents succeed faster? - Interface comparison: Do agents perform better with docs, skills, MCP tools, APIs, or another interface? - Competitive analysis: What alternatives do agents reach for, and what can you learn from their agent experience? When the mention rate is low, plan for larger n see Scale the experiment . - Product marketing: Can you generate evidence that agents are more effective with your product? Where to start where-to-start Get started /0.5.0-rp/getting-started by installing the CLI and running your first experiment. Explore best practices for experiment design /0.5.0-rp/experiment-design . Start with a common use case guide: Test your product's CLI installation /0.5.0-rp/cli-install-guide