Qwen3.8-Flash passed 17/17 checks on a DeepSWE v1.1 task in our agent harness Qwen3.8-Flash, running inside the closed-source host-enforced AIC software engineering runtime, passed all 17/17 canonical checks on the DeepSWE v1.1 updo-policy-alerting task at an operator-reported model/API cost of $0.31 and an observed time of 56m 55s, according to an evidence bundle published in the AIC-Evals repository. The same-task comparison shows GPT-6 Astra at 14/20 and GPT-5.6 Sol at 12/20 under the official mini-swe-agent harness, while Claude Opus 5 scored 8/20 and Gemini 3.8 Flash, Claude Fable 5, Claude Sonnet 5, and Claude Opus 4.8 each scored 0 passes. The repository states the figures are task-difficulty context only and not a controlled leaderboard, because the AIC run used a different harness and operator protocol and a single post-fix attempt cannot estimate a stable pass rate. Verified benchmark artifacts, model patches, and integrity receipts for AIC, a closed-source, host-enforced AI software engineering runtime. This repository publishes inspectable evidence from selected AIC evaluation runs. It does not contain the AIC product, source code, internal system prompts, private role artifacts, held-out tests, or credentials. Qwen3.8-Flash + AIC passed all 17/17 canonical checks on the DeepSWE v1.1 updo-policy-alerting task for an operator-reported model/API cost of $0.31. The comparison above establishes task-difficulty context. AIC and the official model rows used different harnesses, operator protocols, and sampling designs, so it is not a controlled leaderboard. Open the evidence bundle https://github.com/taiheqi718-art/AIC-Evals/blob/main/runs/deepswe-v1.1/updo-policy-alerting/qwen3.8-flash/attempt-01-post-fix · Inspect the frozen patch https://github.com/taiheqi718-art/AIC-Evals/blob/main/runs/deepswe-v1.1/updo-policy-alerting/qwen3.8-flash/attempt-01-post-fix/model.patch · Review official-result context https://github.com/taiheqi718-art/AIC-Evals/blob/main/comparisons/deepswe-v1.1/updo-policy-alerting · Read rubric notes https://github.com/taiheqi718-art/AIC-Evals/blob/main/comparisons/deepswe-v1.1/updo-policy-alerting/RUBRIC-NOTES.md · Read the FAQ https://github.com/taiheqi718-art/AIC-Evals/blob/main/FAQ.md | Benchmark | Task | Model | Attempt | F2P | P2P | Reward | Model cost | Observed time | Effective time | |---|---|---|---|---|---|---|---|---|---| | DeepSWE v1.1 | updo-policy-alerting https://github.com/taiheqi718-art/AIC-Evals/blob/main/runs/deepswe-v1.1/updo-policy-alerting/qwen3.8-flash/attempt-01-post-fix | Qwen3.8-Flash | 1 post-fix | 17/17 | 123/123 | 1.0 | $0.31 | 56m 55s | < 56m 55s | Timing note: The observed AIC time includes six non-passing acceptance-probe executions—three probe error and three behavior failed outcomes—and the associated corrective turns. These included probe-construction errors and assertions superseded by corrected evidence, not canonical-verifier failures. The exact retry-adjusted duration cannot be isolated reliably, so effective time is reported only as a conservative upper bound: < 56m 55s . The official DeepSWE v1.1 data includes repeated mini-swe-agent trials for this task. Most of the models below have four trials at each of five reasoning-effort levels: | Model / system | Harness | Same-task scored passes | |---|---|---| | Qwen3.8-Flash + AIC | AIC | 1/1 post-fix attempt | | GPT-6 Astra | Official mini-swe-agent | 14/20 | | GPT-5.6 Sol | Official mini-swe-agent | 12/20 | | Claude Opus 5 | Official mini-swe-agent | 8/20 | | Gemini 3.8 Flash | Official mini-swe-agent | 0/8 across two published effort levels | | Claude Fable 5 | Official mini-swe-agent | 0/20 | | Claude Opus 4.8 | Official mini-swe-agent | 0/19 scored; 1 provider error excluded | | Claude Sonnet 5 | Official mini-swe-agent | 0/20 | These numbers provide task-difficulty context only. They are not an apples-to-apples ranking: the AIC run used a different harness and operator protocol, and one AIC attempt cannot estimate a stable pass rate. See the effort-by-effort comparison, official source links, snapshot hashes, and derivation https://github.com/taiheqi718-art/AIC-Evals/blob/main/comparisons/deepswe-v1.1/updo-policy-alerting . Two verifier-selected event-ordering details are more specific than the public prose. They are documented neutrally in the task's rubric notes https://github.com/taiheqi718-art/AIC-Evals/blob/main/comparisons/deepswe-v1.1/updo-policy-alerting/RUBRIC-NOTES.md ; the delivered candidate implements both and passes 17/17. - the exact public task instruction given to the runtime; - the frozen candidate patch produced by the run; - the verifier's aggregate reward.json ; - public-safe result metadata and integrity hashes; - a scorecard and its deterministic local renderer; - the upstream license applicable to the patched project. Raw test reports, test names, held-out test source, private rubrics, model transcripts, internal AIC role artifacts, and machine-local configuration are intentionally excluded. Cost and timing are reported with an explicit scope. Model cost is operator-reported model/API spend, not total infrastructure cost. Observed AIC time runs from prompt acceptance to the delivered terminal state; end-to-end time additionally includes the post-delivery canonical verifier. Effective time excludes known non-passing probe and correction overhead, but is shown only as a strict upper bound because an exact counterfactual duration is not recoverable. | Layer | Public artifact | What it establishes | |---|---|---| | Task | task.md https://github.com/taiheqi718-art/AIC-Evals/blob/main/runs/deepswe-v1.1/updo-policy-alerting/qwen3.8-flash/attempt-01-post-fix/task.md | Exact public instruction supplied to AIC | | Candidate | model.patch https://github.com/taiheqi718-art/AIC-Evals/blob/main/runs/deepswe-v1.1/updo-policy-alerting/qwen3.8-flash/attempt-01-post-fix/model.patch | Frozen delivered implementation | | Result | reward.json https://github.com/taiheqi718-art/AIC-Evals/blob/main/runs/deepswe-v1.1/updo-policy-alerting/qwen3.8-flash/attempt-01-post-fix/reward.json | Aggregate canonical verifier outcome | | Provenance | evidence.json https://github.com/taiheqi718-art/AIC-Evals/blob/main/runs/deepswe-v1.1/updo-policy-alerting/qwen3.8-flash/attempt-01-post-fix/evidence.json | Candidate, verifier, cost, timing, and retained-evidence identities | | Integrity | manifest.json https://github.com/taiheqi718-art/AIC-Evals/blob/main/runs/deepswe-v1.1/updo-policy-alerting/qwen3.8-flash/attempt-01-post-fix/manifest.json | SHA-256 inventory of every published run file | | Context | official-results.json https://github.com/taiheqi718-art/AIC-Evals/blob/main/comparisons/deepswe-v1.1/updo-policy-alerting/official-results.json | Machine-readable same-task derivation and official source hashes | | Interpretation | RUBRIC-NOTES.md https://github.com/taiheqi718-art/AIC-Evals/blob/main/comparisons/deepswe-v1.1/updo-policy-alerting/RUBRIC-NOTES.md | Two verifier-selected behaviors more specific than the public prose | When the public AIC desktop and CLI clients are released, developers will be encouraged to evaluate models and tasks and submit public-safe evidence bundles through pull requests. Maintainer-published runs and community submissions will remain separate. Community results will disclose complete comparable attempt series and carry an evidence label—self-attested, artifact-checked, or, when supported by the clients, receipt-verified. A successful single attempt will not be presented as Pass@1 or a stable pass rate. The intended bundle, privacy rules, attempt-disclosure policy, and review process are described in CONTRIBUTING.md https://github.com/taiheqi718-art/AIC-Evals/blob/main/CONTRIBUTING.md . The clients should generate these bundles automatically so contributors do not need to expose AIC internals or protected verifier material. A published run shows that the frozen patch identified by its SHA-256 digest produced the recorded verifier result under the stated environment. AIC itself is proprietary and is not distributed here, so this repository is an artifact record—not a fully reproducible copy of the orchestration system. Attempt numbering is scoped to a materially stable AIC baseline. The first published result is post-fix attempt 1; earlier internal development and recovery rounds used materially different harness revisions and are not counted in this series. This repository does not yet represent a complete attempt ledger or a full-benchmark score. These results are independent publications and are not official leaderboard submissions. See METHODOLOGY.md https://github.com/taiheqi718-art/AIC-Evals/blob/main/METHODOLOGY.md for the evidence protocol, FAQ.md https://github.com/taiheqi718-art/AIC-Evals/blob/main/FAQ.md for common interpretation questions, and DISCLAIMER.md https://github.com/taiheqi718-art/AIC-Evals/blob/main/DISCLAIMER.md for scope and limits. For citation metadata, see CITATION.cff https://github.com/taiheqi718-art/AIC-Evals/blob/main/CITATION.cff . Cite the repository together with the immutable run directory used by your analysis. runs/