# Testing AI Coding Agents Beyond Code Generation: A Real-World Benchmark > Source: > Published: 2026-07-30 03:20:56+00:00 Most public demonstrations of AI coding agents begin with an empty directory and end when a visible feature works. That is useful, but it measures only the first and easiest part of software engineering: generation. Real projects are stateful. They contain existing behavior, data that must survive, security boundaries, partially completed work, and verification steps that cannot be replaced by a convincing demo. This benchmark was designed as a small, practical record of that harder problem. It documents two Codex runs on the same full-stack task-manager project. Phase 1 was a greenfield build. Phase 2 returned to the existing application for an authentication and database-migration refactor. The second run also included a long, user-initiated interruption, making recovery part of the observed workflow. The narrow result is encouraging: the initial application was recorded as complete in approximately 34 minutes, and the later refactor took approximately 42 minutes of active execution, excluding the user-initiated pause. The Phase 2 run reported 16 backend tests, 1 legacy-migration test, and 25 frontend tests passing, along with lint, type, build, dependency, and browser-acceptance checks. Those numbers need boundaries. This is a single-environment case study, not an official benchmark, not a controlled comparison between providers, and not a production guarantee. The current documentation set does not retain the complete source tree, exact prompts, raw terminal output, migration artifact, database fixture, or browser trace. Results are therefore presented as supplied run records rather than independently reproduced evidence. A prompt such as "build a task app" primarily tests synthesis. The agent chooses a stack, creates files, connects components, and produces a working path. It may reveal speed and basic tool use, but it says little about how the agent behaves when constraints collide. Maintenance work introduces those collisions. An authentication refactor changes data models, API contracts, frontend state, routing, test fixtures, and migration behavior at the same time. Adding user ownership is not complete when an `owner_id` column exists; every read and write path must enforce the boundary. Adding a migration is not complete when a fresh database works; legacy rows must still be valid after upgrade. A frontend that compiles can still fail because of CORS, token storage, routing, or a mismatched response shape. The important unit is therefore not generated code. It is a verified change to a stateful system. A useful agent evaluation should ask at least five questions: These questions force the benchmark to include tests, builds, migration checks, and browser acceptance. They also make failure evidence valuable. A slow run that exposes a migration bug is more informative than a fast run that never tests migration. Phase 1 started from a greenfield project. The recorded stack was React and TypeScript on the frontend, FastAPI on the backend, and SQLite for persistence. The requested product was a task manager with create, read, update, and delete flows and a responsive interface. The run was recorded as completing in approximately 34 minutes end to end. The completion record includes the application implementation, automated tests, a production build, security checks, and browser acceptance of the CRUD workflow. One HTTP 429 occurred. The workflow waited approximately two to three minutes, reduced extra agent and skill calls, and then continued to completion. Phase 1 demonstrated that an agent could coordinate several layers of a modest full-stack application quickly. More importantly, it established a project state that could be changed later. That made it possible to move beyond the familiar greenfield demo and test a real maintenance problem. The evidence boundary is significant. Exact test counts, dependency versions, request totals, token totals, and raw output were not retained for this phase. The 34-minute figure describes this recorded run, not expected performance for another application or environment. Phase 2 returned to the existing task manager instead of rebuilding it. The requested scope added registration and login with JWT authentication, Argon2 password hashing, backend-enforced task ownership, tags, archive and restore behavior, and Alembic-managed schema evolution for the existing SQLite database. This changed the engineering problem. Authentication crossed frontend, backend, and persistence boundaries. Per-user isolation required authorization checks on every relevant server path. The existing task rows predated users, so the migration needed an explicit strategy for legacy ownership. The supplied run record says those rows were preserved through a placeholder-user approach rather than deleting or recreating the database. The refactor took approximately 42 minutes of active execution. Its wall-clock span was approximately 11 hours and 49 minutes because the user paused the task for a long period. The longer wall-clock value must not be described as continuous agent work. The reported security baseline included JWT authentication, Argon2 password hashing, server-side `owner_id` isolation, and a CORS allowlist. The acceptance record says one user could not read, modify, or delete another user's tasks. This is a meaningful boundary check, but it is not a complete security assessment. Token storage in `localStorage` , missing reported rate limiting for registration and login, unspecified JWT rotation and revocation behavior, and SQLite concurrency limits remain documented risks. The interruption in Phase 2 was initiated by the user. It was not recorded as a model crash, service outage, or rate limit. When work resumed, the agent continued from the existing project state, recovered the remaining scope, and completed implementation and verification. The record reports no wholesale restart, destructive project replacement, database recreation, or legacy-data loss. This is useful because interruption recovery is part of reliability. Long-running agent work should not depend on an uninterrupted chat transcript. Durable state belongs in the repository: changed files, migration revisions, test commands, acceptance criteria, and explicit notes about what remains. A recovery process should re-establish the current state before making more changes, then rerun the relevant gates instead of assuming earlier results are still valid. The benchmark does not prove a particular internal memory mechanism. A timestamped recovery transcript and checkpoint manifest were not retained. What it does record is the external behavior that matters to an operator: the project state survived, the agent resumed the requested change, and the validation sequence was completed. The Phase 2 supplied run record reports the following results: | Validation gate | Reported result | |---|---| | Backend tests | 16 passed | | Legacy migration tests | 1 passed | | Frontend tests | 25 passed | | ESLint | Passed | | TypeScript | Passed | | Production build | Passed | `pip check` | Passed | `npm audit` | 0 vulnerabilities reported | | Browser console | 0 warnings and 0 errors observed | | Backend logs during acceptance | No 5xx responses observed | The automated suites total 42 reported passing tests: 16 backend, 1 migration, and 25 frontend. That total is coincidentally the same number as the approximate active execution time in minutes. They are unrelated measurements. The migration test is especially important because it targets the riskiest transition: upgrading a legacy database rather than initializing a new one. The browser check matters for a different reason. Authentication, CORS, token handling, route protection, API wiring, and rendering only meet in the integrated application. Unit tests and a successful build cannot cover that entire boundary by themselves. Raw command output, coverage, skipped-test counts, test names, browser traces, and the migration revision are not present in the current workspace. The table should therefore be read as a structured report of the run, not as independently auditable CI evidence. First, separating the benchmark into greenfield and refactor phases exposed different capabilities. Phase 1 measured coordinated generation. Phase 2 measured change under constraints. Combining both views was more informative than repeating another from-scratch application. Second, explicit validation gates kept the definition of "done" wider than a working screen. Backend tests, frontend tests, migration tests, static checks, dependency checks, production build, and browser acceptance each covered a different failure surface. Third, durable project state supported recovery. The long pause did not require a restart from zero. This suggests that repositories, fixtures, checklists, and repeatable commands are part of the agent harness, not just administrative overhead. Fourth, the migration scope forced data preservation into the benchmark. That is a better proxy for real maintenance than generating a new schema on an empty database. The placeholder-user strategy also made the compatibility decision explicit instead of silently discarding old rows. Finally, the run surfaced risks rather than treating passing tests as proof of production readiness. Local token storage, absent reported auth rate limiting, incomplete JWT lifecycle evidence, SQLite write-concurrency limits, and missing raw artifacts remain visible in the record. The largest failure was evidence retention. The current repository does not include the complete application source, exact prompts, raw test logs, dependency lockfiles, migration file, database snapshots, browser trace, or machine-readable timing. That prevents an independent rerun and makes every result dependent on the supplied record. The benchmark also lacks repeated trials and failed runs. Two successful phases cannot establish a reliability rate. There is no controlled baseline against another agent, another model, or a human implementation. Environment details such as operating-system build, runtime versions, browser version, and hardware are incomplete. Phase 1's 429 event was recovered in practice, but the response headers, retry metadata, timestamps, and request trace were not retained. It therefore supports a narrow operational lesson, not a general rate-limit claim. Phase 2's security checks were useful but incomplete. A passing `npm audit` is dependent on the lockfile and advisory database at that moment. `pip check` validates installed dependency compatibility, not vulnerabilities. No penetration test, load test, rollback transcript, or exhaustive authorization matrix was retained. This is a single environment and a small application. It is not an official benchmark, and it was not run or endorsed by OpenAI, Anthropic, or an independent standards body. It is not a production guarantee. Timing is approximate. Active time and wall-clock time are different, especially in Phase 2. The test results are reported outcomes without raw logs in the current documentation workspace. Model routing identity, request counts, token usage, and cost were not independently verified. The benchmark cannot establish typical latency, uptime, repeatability, provider superiority, privacy guarantees, or performance on a larger codebase. It also cannot prove lossless migration for every legacy database shape or complete resistance to authorization, XSS, authentication-abuse, and concurrency failures. The next useful experiment is repetition, not a larger marketing claim. A stronger protocol would freeze one starting commit, one exact prompt set, one environment manifest, and one hashed legacy-database fixture. The same refactor should run at least three times, with successful, incomplete, and failed outcomes all retained. Future runs should publish raw test and build output, exact commands, dependency versions, coverage and skipped-test counts, the Alembic revision, backup checksum, pre- and post-migration row counts, rollback output, browser traces, and machine-readable active and wall-clock timing. Request, token, and nominal credit records should be captured when available. The security plan should add exhaustive cross-user endpoint checks, registration and login rate-limit tests, an HttpOnly refresh-token design, migration interruption and restore scenarios, and SQLite lock-contention testing. A second agent or model could then be evaluated under the same frozen protocol, but only after the evidence pipeline is consistent enough to make comparison meaningful. The publisher promotes TianShu API and may receive a referral commission or other referral benefit from registrations or purchases associated with that service. The documented Codex workflow used the promoted service. This article is therefore not presented as an independent third-party review. TianShu API is a third-party service. No official partnership, authorization, endorsement, or representative relationship with OpenAI, Anthropic, Codex, Claude, or their respective companies is claimed. Pricing, privacy, logging, retention, model availability, and service terms were not independently verified as part of this benchmark and may change. The purpose of publishing the benchmark is to make the observed workflow, result boundaries, failures, and future protocol visible. Readers should evaluate the evidence and limitations independently.