{"slug": "big-pickle-on-swe-atlas-codebase-qna", "title": "Big Pickle on SWE Atlas – Codebase QnA", "summary": "OpenCode Zen's free stealth model big-pickle resolved 50.8% (63/124) of Scale AI's SWE Atlas Codebase QnA benchmark tasks on 2026-08-11, outperforming all official Mini-SWE-Agent scaffold entries and both Codex-scaffold GPT models, trailing only Opus 5 (63.17%) and Opus 4.8 (57.26%) on the official leaderboard. The run used the official harness, task data, and judge model, with zero command timeouts or OOM kills, but Scale did not verify the result and the model's identity is unconfirmed.", "body_md": "**Task Resolve Rate: 50.8% (63/124)** — [ big-pickle](https://opencode.ai/docs/zen/), the free stealth model on OpenCode Zen, evaluated on\n\n[Scale AI's SWE Atlas](https://github.com/scaleapi/SWE-Atlas)Codebase QnA benchmark using the mini-swe-agent scaffold.\n\nRun on 2026-08-11 with the official open-source harness, task data, and judge model.\n\nAgainst the [official SWE Atlas QnA leaderboard](https://labs.scale.com/leaderboard/sweatlas-qna) (updated 2026-07-28):\n\n| Model (scaffold) | Task Resolve Rate |\n|---|---|\n| Opus 5 (Claude Code, xHigh) | 63.17 |\n| Opus 4.8 (Claude Code, xHigh) | 57.26 |\nbig-pickle (Mini-SWE-Agent) — this run |\n50.81 |\n| GLM 5.2 (Mini-SWE-Agent) | 48.12 |\n| GPT-5.6-Sol (Codex, xHigh) | 46.00 |\n| GPT 5.5 (Codex, xHigh) | 45.43 |\n\nWithin the Mini-SWE-Agent scaffold class — the apples-to-apples comparison — this run **outscores every entry on the official leaderboard**, and it also tops the Codex-scaffold GPT entries. Only the two Claude models running on their native Claude Code scaffold score higher. Note the caveats below before treating this as a leaderboard-equivalent number.\n\n| Language | Resolved | Rate |\n|---|---|---|\n| TypeScript | 18/31 | 58.1% |\n| Python | 16/29 | 55.2% |\n| Go | 19/38 | 50.0% |\n| C | 10/26 | 38.5% |\n\n| Category | Resolved | Rate |\n|---|---|---|\n| Code Onboarding | 17/28 | 60.7% |\n| Architecture & system design | 23/44 | 52.3% |\n| Root-cause analysis | 17/37 | 45.9% |\n| Security | 5/11 | 45.5% |\n| API & library usage / integration | 1/4 | 25.0% |\n\nEverything follows Scale's published protocol as closely as budget allowed:\n\n**Tasks:** all 124 Codebase QnA tasks from[scaleapi/SWE-Atlas](https://github.com/scaleapi/SWE-Atlas)(Apache-2.0), unmodified — including Scale's shipped`mswea_qa_config.yaml`\n\nagent configuration (system/instance templates,`step_limit: 250`\n\n).**Harness:**[Harbor](https://github.com/laude-institute/harbor)v0.18.0 with Modal sandboxes, per the SWE-Atlas README.** Scaffold:**[mini-swe-agent](https://github.com/SWE-agent/mini-swe-agent)pinned to 2.4.6 — the same minimal bash-only scaffold Scale uses for non-first-party models on the leaderboard.**Model:**`big-pickle`\n\nvia OpenCode Zen's OpenAI-compatible endpoint (`https://opencode.ai/zen/v1`\n\n), litellm route`openai/big-pickle`\n\n. Total consumption:**674M input / 4.3M output tokens, at $0**(the model is free during its stealth period).** Judge:**`claude-opus-4-5-20251101`\n\n— the exact judge model Scale specifies — accessed through Anthropic's OpenAI-compatible endpoint (`https://api.anthropic.com/v1`\n\n) with`EVAL_MODEL`\n\noverridden to the bare Anthropic model ID.**Scoring:** the benchmark's own rubric-based verifier, unmodified. A task resolves only if every scored must-have rubric passes.\n\nRead these before quoting the number:\n\n**Single trial per task**(`-k 1`\n\n). The official protocol runs 3 trials and reports the mean. At n=124, the single-trial standard error is ≈ ±4.5 points — comparable to the leaderboard's own reported error bars (±5).**Reduced sandbox resources.** Tasks declare 16 CPU / 16 GB; this run used 4 CPU / 8 GB to fit a personal budget. Slower command execution can only depress an agent's score (via command timeouts or OOM kills), not inflate it. Empirically it appears to have had no effect here: a scan of all 124 agent trajectories found**zero command timeouts and zero exit-137 kills**— no command ever hit the 900s ceiling or the memory limit.** Self-reported.**Scale did not run or verify this evaluation. The full per-task verifier logs in this repo allow independent auditing, and the run is reproducible from the configs here plus the public SWE-Atlas repo.**Model identity unknown.** big-pickle is officially unconfirmed; leaked provider errors and API response signatures suggest it is currently served by DeepSeek infrastructure. The underlying model may change without notice, so this result is a snapshot of whatever was behind the alias on 2026-08-11.**Data exposure.** OpenCode states that prompts to big-pickle during its free period may be used to improve the model. The benchmark's task content (already public, canary-marked by Scale) was necessarily sent to that endpoint.**Two resolved tasks had unscored rubrics.** On`task-...ba9ad`\n\n(5 of 11 rubrics) and`task-...baa1d`\n\n(1 rubric), the judge returned unparseable output through all 8 retries; the benchmark's verifier excludes unscored rubrics from the pass computation by design. Treating unscored-as-fail instead gives a strict-lower-bound of**61/124 = 49.2%**— still above every Mini-SWE-Agent leaderboard entry. All verifier logs are included so you can apply either convention.\n\n```\ngit clone https://github.com/scaleapi/SWE-Atlas && cd SWE-Atlas\ngit clone --branch v0.18.0 --depth 1 https://github.com/laude-institute/harbor.git\nuv tool install ./harbor --with modal && uv tool install modal && modal setup\n\n# from this repo: copy run_config/qa, run_config/tw, run_config/rf into\n# SWE-Atlas/run_config/ (preserving the subdirectories — the scripts resolve\n# .env and Scale's mswea_*_config.yaml relative to their own location),\n# copy preflight.sh and .env.example into the SWE-Atlas root,\n# create .env from .env.example, then:\n./preflight.sh\nbash run_config/qa/big-pickle_smoke.sh    # 3-task smoke test first\nbash run_config/qa/big-pickle_miniswe.sh  # full 124-task run\n```\n\nHard-won gotchas the configs already handle:\n\n**Do not pass**— Harbor silently switches mini-swe-agent to the OpenAI Responses API, which chat-completions-only endpoints like Zen don't serve.`--ak reasoning_effort`\n\nwith an`openai/`\n\n-prefixed model**Keep agent and judge credentials separate.** The judge reads host`OPENAI_API_KEY`\n\n/`OPENAI_API_BASE`\n\n(via each task's`[verifier.env]`\n\n); the agent's Zen credentials go through`--ae`\n\nper-agent overrides.**Pass secrets to** Harbor redacts literal secrets to`--ae`\n\nas`${VAR}`\n\ntemplates, not literals.`****`\n\nwhen persisting job state, which breaks`harbor job resume`\n\nwith instant 401s. Templates round-trip and re-resolve from the host env.- Expect a few % of trials to die to Modal\n`Failed to read exec stdio stream`\n\nerrors;`harbor job resume -f <ErrorType> ...`\n\nre-runs them cleanly.\n\nApproximate cost for the full QnA run: ~$70 of Modal compute (at reduced sandbox resources; roughly 2–3× that at the declared 16 CPU/16 GB), ~$25 of Anthropic API for judging, $0 for the model.\n\n`results/per_task_results.csv`\n\n— task ID, category, language, resolved, aggregate rubric score, rubrics passed/total`results/summary.json`\n\n— headline numbers and breakdowns`results/verifier_logs/`\n\n— the judge's full per-rubric output for every task (audit trail). Notes like`(flipped from raw=0)`\n\nare the benchmark's own shipped verifier logic (`evaluate_answer.py`\n\ninverts rubrics marked negative-polarity), not post-hoc re-scoring.`run_config/`\n\n— the exact Harbor run scripts used (QnA smoke + full, plus untested Test Writing / Refactoring variants)`preflight.sh`\n\n— endpoint/auth checks for both the model and the judge\n\n[SWE Atlas](https://github.com/scaleapi/SWE-Atlas)benchmark © Scale AI, Apache-2.0 — paper:[arXiv:2605.08366](https://arxiv.org/abs/2605.08366). Per the authors' request, please treat SWE Atlas as a held-out signal of progress rather than a training target.[Harbor](https://github.com/laude-institute/harbor)(Laude Institute) and[mini-swe-agent](https://github.com/SWE-agent/mini-swe-agent)(SWE-agent team).- big-pickle is served by\n[OpenCode Zen](https://opencode.ai/docs/zen/).\n\nEvaluation configs and results in this repo are MIT-licensed.", "url": "https://wpnews.pro/news/big-pickle-on-swe-atlas-codebase-qna", "canonical_source": "https://github.com/PhillipChaffee/big-pickle-swe-atlas", "published_at": "2026-08-16 00:10:48+00:00", "updated_at": "2026-08-16 00:40:34.220729+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-products", "ai-tools"], "entities": ["OpenCode Zen", "big-pickle", "Scale AI", "SWE Atlas", "Mini-SWE-Agent", "Harbor", "Anthropic", "DeepSeek"], "alternates": {"html": "https://wpnews.pro/news/big-pickle-on-swe-atlas-codebase-qna", "markdown": "https://wpnews.pro/news/big-pickle-on-swe-atlas-codebase-qna.md", "text": "https://wpnews.pro/news/big-pickle-on-swe-atlas-codebase-qna.txt", "jsonld": "https://wpnews.pro/news/big-pickle-on-swe-atlas-codebase-qna.jsonld"}}