{"slug": "how-to-black-box-test-monkeycode-task-reliability", "title": "How to Black-Box Test MonkeyCode Task Reliability", "summary": "A developer created a black-box test plan for MonkeyCode, an AI task management system, to verify reliability under failures like client disconnects, worker crashes, and model timeouts. The plan defines invariants and fixtures to ensure deterministic verification without coupling to internal system details.", "body_md": "A long-running AI development task is a distributed workflow even when the interface looks like one conversation.\n\nThe client can disconnect. A worker can die. A model request can time out after producing a response. A repository write can succeed before its acknowledgment is lost. Cancellation can race with completion. Reliability depends on how the system resolves those events, not on whether the happy-path demo finishes.\n\nThis article develops a black-box test plan for [MonkeyCode](https://github.com/chaitin/MonkeyCode), whose public repository describes AI task management and managed development environments. The source reference is commit [ 1ac778f](https://github.com/chaitin/MonkeyCode/tree/1ac778fdba1da1b353f7f5672d2e4550801cf46d). The plan has not been executed against a production deployment, so it defines claims that could be measured rather than reporting results.\n\nDo not couple the test model to undocumented endpoints. Create an adapter around the supported user or API surface:\n\n``` php\nsubmit(logical_request_id, repository_revision, task_spec) -> task_id\nobserve(task_id, after_cursor) -> ordered_events\nanswer(task_id, question_id, value, expected_version)\ncancel(task_id, expected_version)\nartifacts(task_id) -> patch, checks, preview metadata\n```\n\nThe adapter may drive an official API, a test-only interface, or browser automation. Its job is to normalize observations without pretending to know the internal queue or database schema.\n\nEvery test event should include:\n\n```\ntest_run_id, logical_request_id, task_id, event_id, task_version,\nevent_type, observed_at, actor, repository_revision, artifact_digest\n```\n\nIf the public surface does not expose an event ID or version, record that limitation. A test cannot assert ordered recovery from evidence the system never reveals.\n\nAn invariant must remain true across retries and failures.\n\nRetrying a timed-out submit with the same logical request ID must return the existing task or an explicit duplicate relationship. It must not silently create two workers that can both write a patch.\n\nOnce the authoritative task is `succeeded`\n\n, `failed`\n\n, or `canceled`\n\n, a delayed event must not make it `running`\n\nagain.\n\nA task cannot be considered successful until required artifacts exist and their verification result is attached. “The model stopped” is not task success.\n\nThe client must distinguish `cancel requested`\n\nfrom `canceled`\n\n. If completion wins the race, the final state and artifacts must explain that outcome.\n\nEvery patch or repository write must map to one task identity, source revision, and actor policy. A retry must not duplicate a non-idempotent effect.\n\nRefreshing, polling, reconnecting, or opening the same task from a second device must not resubmit it.\n\nUse fixtures with deterministic verification even if model output varies.\n\n| Fixture | Task | Verifier |\n|---|---|---|\n| Small API | add division-by-zero validation | named unit test passes; bounded file set |\n| Frontend form | add required-field feedback | DOM test plus accessibility assertion |\n| Dependency update | update one pinned library | lockfile change plus full test command |\n| Broken build | repair one known configuration error | build exits `0` ; expected file changes |\n\nPin the repository revision, toolchain, dependency cache policy, model route, task text, maximum duration, and acceptance command. The model response may differ, but the outcome remains machine-checkable.\n\nThe most valuable failures occur after a side effect but before its acknowledgment.\n\n| Fault | Injection point | Required observation |\n|---|---|---|\n| Client disconnect | after submit response is lost | reconnect finds one task without resubmission |\n| Duplicate submit | same logical ID in parallel | one accepted task or explicit deduplication result |\n| Worker termination | during clone, edit, test, and artifact upload | bounded recovery or explicit terminal failure |\n| Model timeout | before and after response receipt | retry policy visible; no duplicated repository effect |\n| Delayed event | deliver older running event after terminal state | version does not regress |\n| Cancel race | cancel while final verification completes | one explainable terminal result |\n| Storage interruption | while persisting event or artifact metadata | no success without durable evidence |\n| Credential expiry | before repository read or write | bounded failure; no partial unauthorized retry |\n\nUse a disposable deployment or test environment. Killing shared production workers to prove a point is not a reliability test plan.\n\nTraditional uptime can hide the worst task outcome: neither the user nor the operator knows whether work happened.\n\nTrack:\n\n```\ntask_completion_rate = verified terminal tasks / accepted tasks\nduplicate_effect_rate = duplicated repository effects / logical tasks\norphan_rate = non-terminal tasks with no live owner beyond deadline / accepted tasks\nambiguous_outcome_rate = tasks whose effect and final state disagree / accepted tasks\nrecovery_time = authoritative recovery event time - injected fault time\ncancel_latency = confirmed terminal event time - cancel acceptance time\n```\n\nReport distributions, not only averages. A p95 recovery time and the raw worst cases matter more than a fast median when a developer waits for a lost task.\n\nA system can remain available while returning stale or contradictory task state. Score each run on four axes:\n\n| Axis | Question |\n|---|---|\n| Availability | Could the user submit, observe, and act? |\n| Safety | Were duplicate or unauthorized side effects prevented? |\n| Liveness | Did the task reach a bounded terminal or recovery state? |\n| Explainability | Could the user and operator reconstruct the outcome from evidence? |\n\nDo not combine the axes into one “reliability score” until the raw failure categories are visible.\n\nExecuted against a versioned deployment, this plan can provide evidence about observed task semantics under declared faults. It cannot prove all internal failure modes, security isolation, model quality across arbitrary tasks, or behavior under a topology that was not tested.\n\nThe official MonkeyCode README documents the product category and deployment options needed to make the plan relevant. It does not establish that each proposed adapter field or invariant is already exposed. Missing observability is itself a test finding, but it should not be rewritten as an implementation claim.\n\nDisclosure: I contribute to the MonkeyCode project. This is a proposed black-box evaluation design based on public documentation, not an independent reliability benchmark.\n\nTeams interested in executing the matrix can join the [MonkeyCode Discord](https://discord.gg/2pPmuyr4pP) to clarify supported test surfaces and deployment topology. The team can also confirm current availability, eligibility, and limits for free model credits used during an evaluation.\n\nThe strongest reliability claim is not “our agent usually finishes.” It is a versioned test result showing what happens when acknowledgments disappear, workers die, events arrive late, and cancellation loses a race.", "url": "https://wpnews.pro/news/how-to-black-box-test-monkeycode-task-reliability", "canonical_source": "https://dev.to/robinzzz/how-to-black-box-test-monkeycode-task-reliability-4681", "published_at": "2026-07-10 10:09:15+00:00", "updated_at": "2026-07-10 10:45:23.525015+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "ai-agents", "machine-learning"], "entities": ["MonkeyCode", "Chaitin"], "alternates": {"html": "https://wpnews.pro/news/how-to-black-box-test-monkeycode-task-reliability", "markdown": "https://wpnews.pro/news/how-to-black-box-test-monkeycode-task-reliability.md", "text": "https://wpnews.pro/news/how-to-black-box-test-monkeycode-task-reliability.txt", "jsonld": "https://wpnews.pro/news/how-to-black-box-test-monkeycode-task-reliability.jsonld"}}