How we make Together AI easy for coding agents to use Together AI published an account of how it built agent evaluations to measure whether coding agents can use its platform, running each task in three configurations — a baseline with no special setup, an MCP server giving direct access to docs.together.ai, and a Together skill package — across three models with two attempts per task, six attempts per configuration, in a run dated September 30th. Together AI said it had no way to know whether agents pointed at its platform were working efficiently or "hitting obstacles, going down rabbit holes or failing outright, and wasting tokens in the process," and cited GitHub as a benchmark where agents opened about 40% of all pull requests in 2026 as platform activity tripled over the past year. The post frames developer experience as increasingly synonymous with agent experience (AX) and describes two problems the evals uncovered and how Together AI fixed them. Earlier this year, we started seeing more and more devs using coding agents to build with Together AI. It’s not hard to see why: It might take days or even weeks for a developer to wire up an app for production inference by hand, especially if they’re new to the platform. But for an agent using our SDK and CLI, it only takes a few minutes—at least in theory. In practice, we had no idea whether the agents pointed at our platform were able to work with Together efficiently, or if they were hitting obstacles, going down rabbit holes or failing outright, and wasting tokens in the process. As devs delegate more of their work to agents, developer experience DX is increasingly becoming synonymous with agent experience AX . The easier your software is for agents to use, the easier it is for devs to build it into their own products and workflows. GitHub is a textbook example of great AX: activity on the platform has tripled in the past year, with agents opening about 40% of all PRs in 2026 and, we’re willing to bet, writing a lot more than 40% of the lines of code across all commits . This is thanks to GitHub’s phenomenal API, CLI, and docs, all of which make it exceptionally easy for agents to use. This writer was not even aware GitHub had a CLI until agents started using it for him, and now he probably uses it 1000+ times a day . To improve our AX at Together AI, we needed a way to measure how agent-ready our platform actually was. In this post, we’ll walk through how we run agent evals, two problems they uncovered and how we fixed them, and what we’ve learned. How we run agent evals at Together Building the task list Before we could start evaluating, we needed to know what agents were actually being asked to do … so we could, y’know, figure out if they could do those things. We started building a list of common tasks that developers tackle on Together, like running a batch inference job, building a voice agent, or managing autoscaling for an inference endpoint. By task, we really just mean a prompt, something you’d tell an agent to do—for example: List the available GPU cluster regions and instance types, print the cheapest option, and show the exact command you would run to reserve it on demand for one hour. Do not actually reserve it. To pass the eval, the agent has to: 1. Find the latest pricing info. 2. Call the regions API endpoint to list the available regions. 3. Combine the two to find the cheapest option. 4. Produce a valid CLI command for creating the cluster. The prompt asks for both availability and pricing, without mentioning that these details live in two different places. This is realistic, as the average user probably doesn’t know up front where each piece of information lives, which could easily trip up both humans and agents. Three configurations After building our list of tasks, we run each one of them three ways: - Baseline: The agent receives just the prompt, with no special configuration, mirroring the average developer's workflow. It can use all its default tools, like fetching docs from the web, but we don't give it any specific MCP connectors or skills. - MCP: The agent gets direct access to our documentation http://docs.together.ai/ through our MCP server, so it knows exactly where to search when it hits a roadblock. - Skills: We give the agent the Together skill package https://github.com/togethercomputer/skills , which contains instructions and examples for using our services, written specifically for agents. The run matrix Each full evaluation runs every task against each agent configuration using three models, with two attempts at each task—so six attempts per task for each config, as shown here for our run on September 30th: Automating the runs When we started building our evals, we ran everything by hand: open a new terminal, paste a prompt into Claude Code, let the agent do its thing or fail to do its thing , and repeat for each configuration. It was manual and tedious, so once we figured out the shape of the system we wanted, we started automating it. Now we run each agent in a Together sandbox https://docs.together.ai/docs/together-code-sandbox so they don’t interfere with our local environment or each other. GitHub Actions handles setup, starts the agent, cleans up afterward, and saves the results, while a proxy handles the agent's requests to the Together API and records the activity, so we can see which endpoints the agent calls and what comes back. Scoring a run After a run finishes, an LLM judge reviews the task prompt and the recorded session, then assigns one of three verdicts: - Pass: Every task requirement was met. - Incomplete: The agent attempted the task, but some required steps were unfinished or unverified. - Fail: The agent gave up early, broke an explicit constraint, or produced an incorrect result RIP . Failed and incomplete verdicts don’t necessarily tell us whether the problem was with the agent, the API, or missing evidence. They just tell us where to start looking. We investigate passing runs too. An agent can often work around confusing responses and still finish, and those recoveries show us what to fix to improve token efficiency. To make that review practical, we built a results viewer that compares how different models and configs performed. For each run, it shows the prompt, the tools the agent used, the responses it got back, and any API activity: Case study 1: GPU cluster pricing Remember the GPU cluster task from earlier? List the available GPU cluster regions and instance types, print the cheapest option, and show the exact command you would run to reserve it on demand for one hour. Do not actually reserve it. Seems pretty simple, but when we ran our evals early last summer, Sonnet 4.6 passed only 4 of 15 attempts at this task, even with skills enabled. Our post-mortem revealed the same pattern in the failed runs: Sonnet treated the sample output in our CLI reference, which listed a single region and two GPU types, as if it were the live inventory. It also skipped the regions lookup and named what it thought was the cheapest option without ever checking the pricing page. The problems were on our end. First, the sample output wasn’t properly labeled, so nothing told the agent to run the command to retrieve the relevant data. Second, our skill file linked to the GPU clusters marketing page rather than the pricing table, so Sonnet opened the marketing page, didn’t see any real prices, and just made a best-effort guess at the cheapest GPU available. More capable models were able to work around the issue. In that same period, Opus 4.8 passed 13 of 14 attempts using the same config. According to the transcript, it noticed that the list regions response didn't include prices, so it kept searching until it found them in our docs. This is why it’s crucial to test your product with less capable models; if we had only tested with Opus, it would have taken us much longer to find the gap. Isolating the failure To fix it, we isolated the failure by pulling the GPU clusters skill out of the full skill package, and wrote a narrower prompt that reproduced only this issue. It failed every time when tested against the old skill file, so we had reliable repro steps use the old skill , and an easy way to test each new version update the skill . We went through several iterations until we landed on a small change: two new sentences in the skill that point the agent to the pricing table, and explain how to use it with the live regions API: Query GET