How to Black-Box Test MonkeyCode Task Reliability

A developer created a black-box test plan for MonkeyCode, an AI task management system, to verify reliability under failures like client disconnects, worker crashes, and model timeouts. The plan defines invariants and fixtures to ensure deterministic verification without coupling to internal system details.

A long-running AI development task is a distributed workflow even when the interface looks like one conversation. The client can disconnect. A worker can die. A model request can time out after producing a response. A repository write can succeed before its acknowledgment is lost. Cancellation can race with completion. Reliability depends on how the system resolves those events, not on whether the happy-path demo finishes. This article develops a black-box test plan for MonkeyCode https://github.com/chaitin/MonkeyCode , whose public repository describes AI task management and managed development environments. The source reference is commit 1ac778f https://github.com/chaitin/MonkeyCode/tree/1ac778fdba1da1b353f7f5672d2e4550801cf46d . The plan has not been executed against a production deployment, so it defines claims that could be measured rather than reporting results. Do not couple the test model to undocumented endpoints. Create an adapter around the supported user or API surface: php submit logical request id, repository revision, task spec - task id observe task id, after cursor - ordered events answer task id, question id, value, expected version cancel task id, expected version artifacts task id - patch, checks, preview metadata The adapter may drive an official API, a test-only interface, or browser automation. Its job is to normalize observations without pretending to know the internal queue or database schema. Every test event should include: test run id, logical request id, task id, event id, task version, event type, observed at, actor, repository revision, artifact digest If the public surface does not expose an event ID or version, record that limitation. A test cannot assert ordered recovery from evidence the system never reveals. An invariant must remain true across retries and failures. Retrying a timed-out submit with the same logical request ID must return the existing task or an explicit duplicate relationship. It must not silently create two workers that can both write a patch. Once the authoritative task is succeeded , failed , or canceled , a delayed event must not make it running again. A task cannot be considered successful until required artifacts exist and their verification result is attached. “The model stopped” is not task success. The client must distinguish cancel requested from canceled . If completion wins the race, the final state and artifacts must explain that outcome. Every patch or repository write must map to one task identity, source revision, and actor policy. A retry must not duplicate a non-idempotent effect. Refreshing, polling, reconnecting, or opening the same task from a second device must not resubmit it. Use fixtures with deterministic verification even if model output varies. | Fixture | Task | Verifier | |---|---|---| | Small API | add division-by-zero validation | named unit test passes; bounded file set | | Frontend form | add required-field feedback | DOM test plus accessibility assertion | | Dependency update | update one pinned library | lockfile change plus full test command | | Broken build | repair one known configuration error | build exits 0 ; expected file changes | Pin the repository revision, toolchain, dependency cache policy, model route, task text, maximum duration, and acceptance command. The model response may differ, but the outcome remains machine-checkable. The most valuable failures occur after a side effect but before its acknowledgment. | Fault | Injection point | Required observation | |---|---|---| | Client disconnect | after submit response is lost | reconnect finds one task without resubmission | | Duplicate submit | same logical ID in parallel | one accepted task or explicit deduplication result | | Worker termination | during clone, edit, test, and artifact upload | bounded recovery or explicit terminal failure | | Model timeout | before and after response receipt | retry policy visible; no duplicated repository effect | | Delayed event | deliver older running event after terminal state | version does not regress | | Cancel race | cancel while final verification completes | one explainable terminal result | | Storage interruption | while persisting event or artifact metadata | no success without durable evidence | | Credential expiry | before repository read or write | bounded failure; no partial unauthorized retry | Use a disposable deployment or test environment. Killing shared production workers to prove a point is not a reliability test plan. Traditional uptime can hide the worst task outcome: neither the user nor the operator knows whether work happened. Track: task completion rate = verified terminal tasks / accepted tasks duplicate effect rate = duplicated repository effects / logical tasks orphan rate = non-terminal tasks with no live owner beyond deadline / accepted tasks ambiguous outcome rate = tasks whose effect and final state disagree / accepted tasks recovery time = authoritative recovery event time - injected fault time cancel latency = confirmed terminal event time - cancel acceptance time Report distributions, not only averages. A p95 recovery time and the raw worst cases matter more than a fast median when a developer waits for a lost task. A system can remain available while returning stale or contradictory task state. Score each run on four axes: | Axis | Question | |---|---| | Availability | Could the user submit, observe, and act? | | Safety | Were duplicate or unauthorized side effects prevented? | | Liveness | Did the task reach a bounded terminal or recovery state? | | Explainability | Could the user and operator reconstruct the outcome from evidence? | Do not combine the axes into one “reliability score” until the raw failure categories are visible. Executed against a versioned deployment, this plan can provide evidence about observed task semantics under declared faults. It cannot prove all internal failure modes, security isolation, model quality across arbitrary tasks, or behavior under a topology that was not tested. The official MonkeyCode README documents the product category and deployment options needed to make the plan relevant. It does not establish that each proposed adapter field or invariant is already exposed. Missing observability is itself a test finding, but it should not be rewritten as an implementation claim. Disclosure: I contribute to the MonkeyCode project. This is a proposed black-box evaluation design based on public documentation, not an independent reliability benchmark. Teams interested in executing the matrix can join the MonkeyCode Discord https://discord.gg/2pPmuyr4pP to clarify supported test surfaces and deployment topology. The team can also confirm current availability, eligibility, and limits for free model credits used during an evaluation. The strongest reliability claim is not “our agent usually finishes.” It is a versioned test result showing what happens when acknowledgments disappear, workers die, events arrive late, and cancellation loses a race.