Cline-Bench: Faithful and Fair Evaluation of Agents on Real User Requests Cline-Bench, a new benchmark built from real user requests, found that Gemini 4 Argon led 11 evaluated models with a 51.1% success rate (pass@1) across 47 agentic coding tasks, each run three times per model. Opus 5.5 and Gemini 4 Carbon tied at 48.9%, while Nemotron 3.5 Super VL ranked last at 9.2% pass@1 and 0.0% pass^3. The benchmark's authors report they repair task instructions and environments for faithfulness and validate tests with deliberately incorrect and tampered solutions for fairness, restoring protected tests in a clean environment and counting missing or skipped required tests as failures. Cline-Bench: Faithful and Fair Evaluation of Agents on Real User Requests A benchmark built from real user requests, with results on execution performance, consistency, cost, and time. Cline-Bench evaluates how well AI agents carry out real user requests. We begin with coding tasks in existing software projects and plan to extend the benchmark to multi-turn interactions and non-coding use cases. The current benchmark contains 47 agentic coding tasks. Each task comprises a user request, a runnable starting repository, and tests for the requested changes and relevant existing behavior. We invested substantial engineering and review effort in constructing, repairing, and validating these task packages. We run each model three times per task to measure success rate and consistency across repeated trials. Faithfulness : we review and repair task instructions and environments to preserve the original request, its constraints, and the context available to the agent. Fairness : we review tests and graders against the required behavior, repair unsupported assumptions and scoring defects, and validate the changes with correct and deliberately incorrect implementations. We evaluate agent solutions in a separate clean environment, restore protected tests, and count missing or skipped required tests as failures. We also test the grader with deliberately tampered solutions. These safeguards help prevent agents from receiving credit by weakening tests, skipping checks, or fabricating passing output. The current evaluation focuses on single-turn coding tasks and builds on the Cline-Bench initiative https://cline.bot/blog/cline-bench-initiative?ref=cline.ghost.io . Extensions to multi-turn tasks, in which users provide feedback or additional instructions, and to non-coding tasks remain future work. Benchmark results The chart reports success rate pass@1 , estimated from three trials per task. The table adds pass@3 and pass^3 to distinguish tasks solved at least once from tasks solved in every trial. | Eleven model rows covering 47 tasks, with three trials per task. Ranked by success rate, then pass^3, both descending. | | | | | |---|---|---|---|---| | Rank | Model | Success rate pass@1 | pass@3 | pass^3 | |---|---|---|---|---| | 1 | Gemini 4 Argon | 51.1% | 63.8% | 40.4% | | 2 | Opus 5.5 | 48.9% | 59.6% | 42.6% | | 3 | Gemini 4 Carbon | 48.9% | 66.0% | 29.8% | | 4 | Sonnet 5.5 | 48.2% | 61.7% | 38.3% | | 5 | GPT 6 Astra | 44.7% | 53.2% | 36.2% | | 6 | DeepSeek V4.1 Flash 1017 | 44.7% | 66.0% | 21.3% | | 7 | GPT 6.1 Sol | 44.0% | 53.2% | 34.0% | | 8 | Kimi K3.1 Pro | 43.3% | 57.4% | 25.5% | | 9 | GLM 5.3 | 34.0% | 48.9% | 19.1% | | 10 | Mistral Large 4 | 29.1% | 46.8% | 12.8% | | 11 | Nemotron 3.5 Super VL | 9.2% | 19.1% | 0.0% | - Success rate pass@1 : the mean success rate across tasks, estimated from three trials per task. With the same number of trials for every task, it equals successful trials divided by total trials. - pass@3: the percentage of tasks with at least one successful trial out of three. - pass^3: the percentage of tasks with all three trials successful. Model inference costs We report estimated mean model inference cost per trial, including both successful and failed trials with recorded usage. Runtime and interaction efficiency We report mean and median solver wall-clock time and assistant turns. Solver time excludes environment setup and grading. An assistant turn is one recorded assistant message, including messages containing tool calls; tool-result messages are excluded. These statistics use available records from both successful and failed trials. | Mean and median solver time and assistant turns, alongside overall success rate. | | | | | | |---|---|---|---|---|---| | Model | Solver time | | Assistant turns | | Success rate | |---|---|---|---|---|---| | | Mean | Median | Mean | Median | | | Gemini 4 Argon | 27.8 min | 21.2 min | 80.5 | 69.0 | 51.1% | | Opus 5.5 | 46.6 min | 32.6 min | 69.8 | 62.0 | 48.9% | | Gemini 4 Carbon | 37.8 min | 29.3 min | 82.9 | 69.0 | 48.9% | | Sonnet 5.5 | 25.1 min | 15.2 min | 49.7 | 43.0 | 48.2% | | GPT 6 Astra | 14.2 min | 13.1 min | 59.9 | 51.5 | 44.7% | | DeepSeek V4.1 Flash 1017 | 14.4 min | 13.4 min | 120.3 | 95.0 | 44.7% | | GPT 6.1 Sol | 15.3 min | 14.6 min | 49.7 | 43.5 | 44.0% | | Kimi K3.1 Pro | 16.5 min | 13.8 min | 72.7 | 67.5 | 43.3% | | GLM 5.3 | 18.9 min | 16.6 min | 102.2 | 93.0 | 34.0% | | Mistral Large 4 | 17.1 min | 16.8 min | 88.4 | 82.0 | 29.1% | | Nemotron 3.5 Super VL | 6.9 min | 6.2 min | 90.7 | 74.5 | 9.2% | Performance over time The chart measures how success accumulates with elapsed solver time, at 15-minute intervals. At time t , cumulative success rate is the number of successful trials completed by t divided by all 141 trials for that model. Trials still running at t contribute zero to that rate. Task example A SkillShare task asks an agent to connect an existing application’s connection-request and messaging features. When a recipient accepts a request, the backend must create a persistent connection and a conversation shared by both users. The request specifies FastAPI and PostgreSQL, requires reuse of the existing authentication and data models where available, and rules out browser storage as the source of truth. Behavioral checks For a request authenticated as User B with a forged sender id: User A , the grader accepts either rejection with status 400 or 422 and no stored message, or acceptance with User B stored as the sender. An unrelated User C must not be able to open, read or send to the conversation. | Selected requirements and grading checks | | |---|---| | User requirement | Behavioral check | |---|---| | Both users access the same conversation after acceptance. | Authenticate as each participant and verify that both receive the same conversation ID. | | Acceptance is atomic and does not create duplicate relationships. | Inject a database insertion failure and verify rollback; repeat a successful acceptance and verify that no rows are added. | | Conversations remain private to their participants. | Attempt to open, read, and post as an unrelated user; verify refusal and unchanged message history. | | Message identity comes from authentication. | Submit forged sender fields; require either rejection without a write or a message attributed to the authenticated sender. | The tests verify that acceptance creates a persistent connection and shared conversation, that both participants can access it, and that a failed transaction leaves no partial state. The backend tests exercise these requirements through authenticated API requests and database-state checks. Regression tests also verify that account creation, login, and access to the authenticated user’s profile continue to work. The sender-identity test illustrates how grading accommodates valid alternatives. Identity comes from authentication: the implementation can reject an unexpected sender field without storing a message, or accept the message while recording the authenticated user as its sender. The test was revised to accept both behaviors and reject forged attribution. Sandbox and grading The environment uses established evaluation mechanisms: repository images and patch-based testing, as in SWE-bench https://www.swebench.com/SWE-bench/faq/?ref=cline.ghost.io how-does-the-evaluation-work , and Harbor’s separate verifier https://github.com/harbor-framework/harbor/blob/main/examples/tasks/separate-verifier-environment/task.toml?ref=cline.ghost.io . In this example, local PostgreSQL supports checks of persistence, access control and transaction rollback. Sandbox and grading walkthrough SkillShare starts from a fixed repository snapshot in an isolated container with a pinned Python runtime and dependencies, plus local PostgreSQL. Dependencies are installed during image construction; the task configures application execution and grading without external network access. The held-out tests and reference solution are kept out of the agent’s image. After the agent submits its Git patch, a separate verifier applies it to a clean copy of the starting environment. The verifier restores protected test files and applies the held-out tests before execution. Those tests call the FastAPI routes and inspect a real PostgreSQL database, with database state reset between tests. The solver’s running processes and installed tools are not reused for verification. The grader requires all 19 feature tests and three regression tests to pass. Missing or skipped required tests count as failures. We check both directions: the unchanged starting implementation scores 0, while the reference solution scores 1. The base and reference controls check that the grader distinguishes the starting implementation from the reference solution; restoring protected files limits direct changes to graded tests. These measures do not establish complete resistance to report tampering: this version converts pytest’s text output into the report used for scoring. The walkthrough illustrates selected backend requirements; the full task also includes frontend integration. Task formulation Each task is a versioned bundle containing a user request and a starting repository. The agent can inspect and modify files, run available project tests, and iterate before submitting a patch. A single-turn task supplies one user instruction while allowing multiple tool calls and code revisions. 1. User request 2. Starting workspace 3. Submitted change 4. Separate verification Held-out tests and the reference solution are excluded from the agent's workspace. A separate environment evaluates the submitted patch. We retain the agent trace, patch, and verification output for outcome analysis. Task package contents Our coding tasks use a Harbor-compatible layout. This simplified tree shows the request, runtime, verification inputs, and reference implementation. task/ ├── instruction.md ├── task.toml ├── environment/ │ ├── Dockerfile │ └── base repo git.tar.gz ├── tests/ │ ├── test.sh │ ├── test.patch │ └── config.json └── solution/ ├── solve.sh └── solution.patch - The request: instruction.md describes what the user wants, the constraints, and the expected deliverable. - The run settings: task.toml connects the package to the harness, including environment resources, execution limits, and how to collect the submission. Evaluation settings are recorded alongside the results. - The starting workspace: environment/ contains the repository snapshot and a Dockerfile defining its runtime and dependencies. - The checks: tests/ contains the verification entrypoint, test changes, and grading configuration. It checks both the requested behavior and existing behavior that must keep working. - A reference solution: solution/ holds an implementation used to validate the task. The agent is judged on behavior, not on whether its code matches this patch. Task taxonomy Each task is assigned a primary engineering category based on the requested change. Individual tasks can involve several areas. Reference implementations span JavaScript, TypeScript, Python, Java, Go, C , and PHP, often combining frontend, backend, and database changes. Task validity and evaluation fairness We trace task instructions and grading criteria to the original user request and the starting repository's existing contracts. Independent review checks that task preparation preserves the requested functionality, constraints, and context available to the agent. This correspondence defines faithfulness. We inspect source code, data fixtures, archives, and Git history for personal information and credentials. We remove unnecessary sensitive data and replace values needed by the task with synthetic data, preserving the data structures and relationships used by the application and its tests. We then validate affected functionality and retain required open-source license and attribution notices. We assess fairness by linking tests for newly requested functionality to user requirements or established repository contracts. Independent reviewers check for unsupported implementation assumptions. To detect reward hacking, where an agent passes tests without satisfying the request, we evaluate deliberately incorrect variants and alternative valid implementations. We check that incorrect variants fail for the intended behavioral reason and valid alternatives pass without reproducing the reference solution. Regression tests check that existing functionality continues to work. We validate graders by checking that the starting repository fails the new-feature tests and the reference solution passes the required checks. Additional integrity checks target manipulated tests and fabricated results. These checks provide evidence of task quality without ruling out every possible defect. A forthcoming arXiv paper will describe task construction, validation, and remaining limitations. Failure classification We inspect failed trials through their conversations, tool calls, submitted patches, and verification output to identify unmet requirements, incomplete implementations, regressions, and timeouts. For example, a SkillShare solution could leave partial database state after a transaction error or allow unauthorized conversation access. These are illustrative failure modes, not findings about particular trials. We distinguish confirmed infrastructure interruptions from failures attributable to the agent. When the cause cannot be established, we retain an unknown classification. Experimental setup We conducted the evaluation using Cline CLI 3.0.74, a four-hour solver limit, and isolated hourly grading checkpoints. We used high reasoning effort for all models that support an effort setting. Nemotron 3.5 Super VL used its native reasoning setting. These labels specify the requested configuration; effort settings are not necessarily equivalent across providers. A model reaching its time limit counts as a failure. A run interrupted by a confirmed technical failure may be repeated, with only the replacement included in the results. Cost, time, and turn statistics use available records; missing values are excluded. Estimated costs cover model inference, excluding test environments and replaced runs. We follow the pass@k convention used in HumanEval https://arxiv.org/abs/2107.03374?ref=cline.ghost.io and the pass^k reliability metric introduced in τ-bench https://arxiv.org/abs/2406.12045?ref=cline.ghost.io . With three trials per task, pass@1 is the total number of successful trials divided by 141. We compute pass@3 and pass^3 from each task's observed outcomes, rather than from powers of the aggregate success rate. Statistical uncertainty We constructed 10,000 bootstrap samples. Each sample contains 47 tasks drawn with replacement, with all three trials retained for every sampled task. We use the same task sample for every model to preserve the paired comparison. The error bars mark the 2.5th and 97.5th percentiles of the resulting success rates; the random seed is 20261113. These intervals quantify variation under resampling of the selected tasks. They are conditional on this retrospectively selected subset and its recorded trials. They do not correct task-selection bias or configuration differences between batches, and do not establish statistically significant differences between adjacent models. Future work The current evaluation provides one user request per trial. Real workflows also involve feedback on intermediate results, clarification of requirements, and changes in direction. Evaluating these interactions requires a multi-turn setting. We plan to extend Cline-Bench to measure how agents incorporate feedback while preserving earlier requirements. The evaluation will consider clarification requests, revisions in response to user feedback, and completion of the evolving task. A second planned extension covers research, data analysis, and artifact creation. These tasks require domain-specific evaluation: evidence for research claims, correct calculations and interpretations for analysis, and deliverables that satisfy the user's stated requirements. These extensions are not included in the reported results. Acknowledgments We thank SFData, Micro1, Arcee, and NVIDIA for their feedback on our tasks.