{"slug": "benchmarking-qwen3-coder-next-deepseek-r1-and-qwen3-5-with-dynamic-rsi", "title": "Benchmarking Qwen3-Coder-Next, DeepSeek-R1, and Qwen3.5 with Dynamic RSI", "summary": "A benchmark report compares two locally hosted models, qwen3-coder-next (80B total / 3B active MoE) and deepseek-r1:32b (32.8B dense reasoning model), on princeton-nlp/SWE-bench_Verified against published frontier-model scores, testing a \"Dynamic RSI Harness\" that combines just-in-time macro-runtime compilation with execution-gated micro-cognitive evolution. The harness's authors claim it differs from OPRO, TextGrad, DSPy (MIPROv2), ADAS, AFlow, standalone ACE, the Darwin Gödel Machine and Self-Harness by switching agent topologies per task and gating ACE playbook rules through a sandbox check with a cross-domain semantic threshold of 0.78, which they say prevents rule pollution. The report presents no numeric pass-rate results for either local model in the excerpt provided.", "body_md": "# Benchmark & Industry RSI Comparison Report: `qwen3-coder-next` & `deepseek-r1:32b` vs. Dynamic RSI Harness vs. Frontier Models\n\nComparing our live runs on `princeton-nlp/SWE-bench_Verified` across **two local models on our LAN server**—**`qwen3-coder-next:latest` (80B total / 3B active MoE)** and **`deepseek-r1:32b` (32.8B dense reasoning model)**—against published `SWE-bench Verified` scores of major frontier models:\n\n### Chart B: Progression Across Both Live Benchmarks & Both Models (`qwen3-coder-next` vs. `deepseek-r1:32b`)\n\nComparing **[A] Bare Model (Pass@1)** **[B1] Dynamic RSI (1st-Shot Multi-Critic + ACE Playbook)** **[B2] Dynamic RSI (Full Loop w/ Introspective Sandbox & ACE Self-Repair)** across both local models:\n\nMost self-improving agent frameworks in 2025–2026 optimize **only one layer** of the stack:\n\n- **Macro-Only Optimizers** (like**ADAS** and**AFlow** ) search for a better multi-agent graph or workflow topology offline, but keep the inner cognitive context/tools static during task execution.\n- **Micro-Only Optimizers** (like**ACE** ,**TextGrad** ,**OPRO** , or**LATM** ) refine prompt rules or synthesize a helper function inside a**fixed, hardcoded workflow loop** .\n\n**What makes our Dynamic RSI Harness unique is the unification of Two Orthogonal Architectures—JIT Macro-Runtime Compilation + Execution-Gated Micro-Cognitive Evolution—coupled through Pytest-Style Sandbox Introspection.**\n\n| Framework / Paradigm | Primary Optimization Target | Timescale of Adaptation | Handles Context Window Limits? | Synthesizes Verified Code Tools? | Prevents Prompt/Rule Collapse? | How Our Dynamic RSI Harness Differs | \n|---|---|---|---|---|---|---|\n| **1. OPRO / TextGrad***(DeepMind / Stanford)* | Prompt text strings (\"textual gradients\") | Offline batch optimization | ❌ No | ❌ No | ❌ No (Monolithic prompt rewrites) | TextGrad backpropagates natural-language critiques without deterministic LHS/RHS sandbox values or topology switching. | \n| **2. DSPy (MIPROv2)***(Stanford)* | Few-shot demonstrations & module instructions | Offline compilation on a training split | ❌ Static pipeline | ❌ No |  | DSPy requires an offline labeled training set to compile a fixed graph; our harness compiles **JIT per user prompt** and self-repairs online. | \n| **3. ADAS***(Hu, Lu, Clune et al.)* | Code-defined agent workflow graphs | Offline evolutionary meta-search |  | ❌ Fixed toolset | ❌ No structured rule memory | ADAS searches for *one* best agent architecture per benchmark offline; our`MetaArchitect` dynamically switches topologies**per task** . | \n| **4. AFlow***(ICLR 2025/2026)* | MCTS search over workflow nodes/edges | Offline MCTS on validation split | ❌ No | ❌ No | ❌ No | AFlow optimizes graph edges via MCTS offline; we combine **Macro topology selection** with**Micro ACE rule & DGM skill evolution** online. | \n| **5. ACE (Standalone)***(arXiv:2510.04618)* | Structured context playbooks (Generator-Reflector-Curator) | Online / Cross-task context updates | ❌ Fixed runtime | ❌ No tool synthesis | ✅ Yes (Delta rules) | Standalone ACE commits reflector rules without a **Darwinian Sandbox Gate** or**Cross-Domain Semantic Threshold** (`>= 0.78` ), which we proved can cause rule pollution. | \n| **6. Darwin Gödel Machine (DGM)***(Sakana AI, arXiv:2505.22954)* | The agent's own codebase & tools | Long-horizon evolutionary branching | ❌ No JIT context middleware | ✅ Yes | ❌ No structured playbook | DGM mutates source code across expensive evolutionary generations; we pair **JIT DGM skill synthesis** with fast**ACE Playbook delta rules** and**LHS/RHS assertion introspection** . | \n| **7. Self-Harness***(arXiv:2606.09498)* | Harness scaffolding & guardrails via trace mining | Offline / Batch iterative loop |  |  | ❌ No semantic vector playbook | Self-Harness patches a single global harness per model; we maintain **category-specific evolved templates** +**dense embedding rule retrieval** (`nomic-embed-text` ). | \n| **8. Our Dynamic RSI Harness** | **Dual-Architecture: Macro `HarnessConfig` + Micro (`PlaybookRule` + `SynthesizedSkill` + `Guardrails`)** | **3 Nested Loops:** JIT (Loop 0) + Intra-Task (Loop 1) + Cross-Task (Loop 2) | ✅ **Yes** (`Map-Reduce` &`Scratchpad` ) | ✅ **Yes** (Unit-tested in isolated scope) | ✅ **Yes** (Darwinian-gated ACE + Tag/Cosine filter) | **Unifies Macro Runtime Routing (ADAS/Self-Harness) + Micro Cognitive/Tool Evolution (ACE/DGM) in a single runtime.** | \n\nWhen we tested our harness on both a **3B-active-parameter MoE coding model (`qwen3-coder-next:latest`)** and a **32B dense reasoning model (` deepseek-r1:32b`)** against `SWE-bench Verified` and adversarial coding tasks, five deep engineering mechanisms separated our system from textbook implementations of ACE, ADAS, and DGM:\n\n```\nflowchart TD\n    subgraph Macro[\"Architecture 1: Macro-RSI (JIT Runtime Compiler — ADAS / Self-Harness)\"]\n        Task[\"Incoming User Task\"] --> Profiler[\"Task Bottleneck & Model Profiler\"]\n        Profiler -->|\"Context-Heavy\"| MapReduce[\"DIVIDE_AND_CONQUER +\\nHIERARCHICAL_MAP_REDUCE\"]\n        Profiler -->|\"Symbolic / Math\"| PlanVerify[\"PLAN_EXECUTE_VERIFY +\\nJIT DGM Skill Mounting\"]\n        Profiler -->|\"Complex Code / SWE-bench\"| MultiCritic[\"MULTI_CRITIC_DEBATE +\\nModel-Aware Prefill & Stop Guards\"]\n    end\n\n    subgraph Micro[\"Architecture 2: Micro-RSI (Cognitive & Tool Evolution — ACE + DGM)\"]\n        MultiCritic & PlanVerify & MapReduce --> Sandbox[\"Introspective Sandbox Gate\\n(Evaluates exact LHS vs. RHS & Rebinds Module Closures)\"]\n        Sandbox -->|\"Pass\"| DarwinGate[\"Darwinian Commit Gate\\n(Commits Rule/Skill ONLY if repair passes F2P & P2P)\"]\n        Sandbox -->|\"Fail: Exact LHS/RHS Trace\"| Reflector[\"Tri-Modal Reflector (Two-Block Format)\\n1. ACE Delta Rule (RULE_WHEN/DO/AVOID)\\n2. Standard Python Fenced Code Block\\n3. Post-Execution Guardrail\"]\n        Reflector --> Sandbox\n        DarwinGate --> VectorDB[(\"Hybrid Tag + Embedding Memory\\n(nomic-embed-text, cosine >= 0.78)\")]\n    end\n\n    VectorDB -.->|\"Zero-Shot Transfer to Next Task\"| Profiler\n```\n\n#### Deep Dive 1: Decoupling \"Wrong Runtime\" (Macro) from \"Cognitive Blindspot Inside the Right Runtime\" (Micro)\n\n- **The Industry Problem:** Systems like**AFlow** and**ADAS** assume that if an agent fails, you need a different workflow graph. Systems like**ACE** and**TextGrad** assume the workflow graph is fine and you only need better prompt instructions.\n- **Our Solution:** Both failure modes happen in reality, and they require different fixes:\n  - When `TASK-01-LONG-CONTEXT` failed because`3,910` tokens overflowed a`1,200` -token attention window, no prompt instruction could fix \"Lost in the Middle.\" Our**Macro-RSI** mutated the`HarnessConfig` to`DIVIDE_AND_CONQUER` +`HIERARCHICAL_MAP_REDUCE` , dropping peak context to`400` tokens.\n  - Conversely, on `BENCH-01-CUSTOM-DSL` and`pytest-dev__pytest-7324` , the`MULTI_CRITIC_DEBATE` runtime was**already optimal** , yet the model failed due to micro-level cognitive traps (confusing unary vs. binary`-` after`)` , or turning`\"False\"` into an AST constant instead of a prefixed identifier). Our**Micro-RSI** extracted surgical`WHEN -> DO -> AVOID` rules from the trace and solved both tasks without touching the topology.\n- When \n\n- **The Industry Problem:** Standard**Darwin Gödel Machine (DGM)** and code-repair loops run`exec()` or`pytest` and feed raw stack traces back to the LLM. In Python, a failed`assert f(x) == expected` raises an empty`AssertionError:` . When we ran`qwen3-coder-next:latest` on`rolling_xor_checksum` , the model got stuck in a loop because it couldn't tell*which* assertion line failed or what its function actually returned (`834` vs its mental-math guess`946` ).\n- **Our Solution:** In`SkillRegistry.register_and_verify` and`execute_swebench_patch` , we built an**AST/Expression Introspection Sandbox** that splits failed assertions and evaluates both`LHS` and`RHS` in the live namespace (`LHS evaluated to 834, expected RHS 946` ). This single diagnostic upgrade turned multi-attempt deadlocks into**Attempt-2 self-repairs** .\n\n- **The Industry Problem:** In the original**ACE** paper (*arXiv:2510.04618* ), the Reflector generates delta rules after every trajectory and the Curator merges them into the playbook. During our live run on`BENCH-02-TRANSFER-DSL` , when`qwen3-coder-next:latest` 's parser failed a test, the Reflector**blamed the specification** and wrote a poison rule:*\"The spec's precedence labels are reversed; swap `+` and `*` back to standard PEMDAS!\"*\n- **Our Solution:** We combined**Self-Harness's regression gate** with**ACE's curator** (`run_head_to_head_benchmark.py` &`run_swebench_verified.py` ): a candidate`PlaybookRule` is treated as an**untrusted hypothesis** and is**only committed to `PlaybookMemory` if the accompanying code repair passes the deterministic sandbox gate (`FAIL_TO_PASS` + `PASS_TO_PASS`)** . Speculative rules from failed repairs are automatically discarded.\n\n- \n**The Industry Problem:** Memory-based agents typically retrieve top-$K$ past reflections using raw cosine similarity from an embedding model (`nomic-embed-text` ). In our first`SWE-bench Verified` run, we discovered that any two Python bug reports share a baseline cosine similarity of`~0.60` , causing an Astropy`NDDataRef.mask` rule to mount into a Pytest AST compiler task and degrade first-shot accuracy.\n- \n**Our Solution:** In`PlaybookMemory.retrieve_for_task` , we implemented**Hybrid Domain-Gated Retrieval** : a rule is mounted only if it shares an explicit symbolic`domain_tag` with the target task**or** exceeds a strict cross-domain semantic threshold (`cosine_similarity >= 0.78` ), weighted by its empirical Bayes confidence (`times_succeeded / times_applied` ).\n\n- **The Industry Problem:** When we switched from`qwen3-coder-next:latest` to**`deepseek-r1:32b`** (a distilled reasoning model that wraps every response in`<think>...</think>` ), two severe model-specific failure modes emerged:\n  1. **Token Budget Starvation Inside `<think>`:** At` 3.4 tok/s` ,`deepseek-r1:32b` spent 2,000+ tokens ruminating inside`<think>` before closing`</think>` , returning an empty`message.content` (`SyntaxError: Empty code` ).\n  2. **JSON-Escaped Code Corruption:** When the Reflector asked`deepseek-r1:32b` to return`\"repaired_code\"` as a JSON-escaped string, the model mis-escaped`\\n` and`\\\"` inside multi-line Python docstrings, causing every self-repair attempt to fail JSON parsing or Python`compile()` .\n- **Our Self-Harness Solution:** Following the**Self-Harness** paradigm (*arXiv:2606.09498* ), we adapted`OpenAICompatibleClient.chat` and the Reflector interface:\n  - Added an **Assistant `<think>\\n</think>\\n` prefill + ```` \\n``` ````  stop-sequence guard** for R1 models, externalizing reasoning into our explicit**Root-Cause Critic + ACE Playbook** stages while cutting per-call latency by**4x** .\n  - Replaced JSON-embedded code strings with a **Two-Block Reflector Format** (`RULE_WHEN:` /`RULE_DO:` /`RULE_AVOID:` headers followed by a native```` ``` python ```` fenced code block), immediately unlocking` deepseek-r1:32b` 's**2x jump (`25.0% -> 50.0%`) on `SWE-bench Verified`** .\n- Added an \n\n## 3. Side-by-Side Comparison Table Across All 4 Tested Models (`deepseek-v4-pro-qwen3.5:9b`, `deepseek-r1:32b`, `qwen3-coder-next`, & `Gemini Flash`) & Frontier Models\n\n| Model / Agent System | Active Params / Token | Total Params | `SWE-bench Verified` (Focal Suite, 8 Issues) | Hard Algorithmic Suite (4 Tasks / 29 Asserts) | `HARD-FLASH` Legacy & Symbolic-Oracle Suite (4 Tasks / 27 Asserts) | Compute / Hosting Profile | \n|---|---|---|---|---|---|---|\n| **Raw Unadapted `deepseek-v4-pro-qwen3.5:9b`** | **9B** | 9B Dense ( `Q4_K_M` ) | **0.0%** (`0/8` ,`IndentationError` /`<think>` truncation) | **0.0%** (`0.0%` /`0/29` asserts) | — | Local GPU Server ( `5.4 GB` ,`13.8 tok/s` ) | \n| **Bare `deepseek-v4-pro-qwen3.5:9b` (w/ Tab Adapter, Pass@1)** | **9B** | 9B Dense ( `Q4_K_M` ) | **12.5%** (`1/8` ) | **0.0%** (`6.9%` /`2/29` asserts) | — | Local GPU Server ( `5.4 GB` ,`13.8 tok/s` ) | \n| **`deepseek-v4-pro-qwen3.5:9b` + Dynamic RSI (Full Loop)** | **9B** | 9B Dense ( `Q4_K_M` ) | **25.0%** (`2/8` ) | **0.0%** (`37.9%` /`11/29` asserts) | — | Local GPU Server ( **Matches Bare 32B/80B SWE-bench & 5.5x Algorithmic Asserts!** ) | \n| **Bare `deepseek-r1:32b` (Pass@1)** | **32.8B** | 32.8B Dense | **25.0%** (`2/8` ) | **0.0%** (`41.4%` /`12/29` asserts) | — | Local GPU Server ( `3.4 tok/s` , Released) | \n| **Bare `qwen3-coder-next:latest` (Pass@1)** | **3B** | 80B MoE | **25.0%** (`2/8` ) | **25.0%** (`69.0%` /`20/29` asserts) | — | Local GPU Server ( `~25 tok/s` ) | \n| **OpenAI GPT-4o (Baseline Scaffold)** | ~200B+ (est.) | Proprietary | **33.2%** | — | — | Cloud API | \n| **`deepseek-r1:32b` + Dynamic RSI (1st-Shot)** | **32.8B** | 32.8B Dense | **37.5%** (`3/8` ) | **0.0%** (`51.7%` /`15/29` asserts) | — | Local GPU Server (Critic + ACE Playbook) | \n| **`qwen3-coder-next` + Dynamic RSI (1st-Shot)** | **3B** | 80B MoE | **37.5%** (`3/8` ) | **50.0%** (`82.8%` /`24/29` asserts) | — | Local GPU Server (Critic + ACE Playbook) | \n| **`deepseek-r1:32b` + Dynamic RSI (Full Loop)** | **32.8B** | 32.8B Dense | **50.0%** (`4/8` ) | **0.0%** (`62.1%` /`18/29` asserts) | — | Local GPU Server ( **2x SWE-bench gain!** ) | \n| **`qwen3-coder-next` + Dynamic RSI (Full Loop)** | **3B** | 80B MoE | **62.5%** (`5/8` ) | **75.0%** (`93.1%` /`27/29` asserts) | — | Local GPU Server ( **2.5x SWE-bench gain!** ) | \n| **Qwen3-Coder-480B-A35B-Instruct** | **35B** | 480B MoE | **~69.0% – 73.0%** | — | — | Multi-Node H100 Cluster | \n| **Claude 3.5 Sonnet (New) / Sonnet 4** | Proprietary | Proprietary | **~76.0% – 79.0%** | — | — | Cloud API + Agentic Scaffold | \n| **DeepSeek-R1 / V3 (671B Full Cloud)** | **37B** | 671B MoE | **~76.0% – 79.4%** | — | — | Multi-Node H100 Cluster | \n| **Bare `Gemini Flash` (Pass@1)** | Frontier Flash | Cloud Flash | **100.0%** (`8/8` ) | **100.0%** (`29/29` asserts) | **25.0%** (`1/4` tasks,`24/27` asserts) | Cloud Flash Model (Single-shot) | \n| **`Gemini Flash` + Dynamic RSI (1st-Shot Transfer + DGM)** | Frontier Flash | Cloud Flash | **100.0%** (`8/8` ) | **100.0%** (`29/29` asserts) | **75.0%** (`3/4` tasks,`26/27` asserts) | Cloud Flash Model + ACE + DGM | \n| **`Gemini Flash` + Dynamic RSI (Full Loop)** | Frontier Flash | Cloud Flash | **100.0%** (`8/8` ) | **100.0%** (`29/29` asserts) | **100.0%** (`4/4` tasks,`27/27` asserts) | Cloud Flash Model ( **4x `HARD-FLASH` gain!** ) | \n\n### Deep Dive 6: How Our Dynamic RSI Harness Lifted `Gemini Flash` from `25.0%` $\\rightarrow$  `100.0%` on the `HARD-FLASH` Suite\n\nWhile `Gemini Flash` saturated the focal-window `SWE-bench Verified` (`8/8`) and `Algorithmic` (`4/4`) suites on Pass@1, testing it on our **4-Task Symptom-Only Legacy & Symbolic-Oracle Suite (`HARD-FLASH-01` to `HARD-FLASH-04`)** dropped Bare `Gemini Flash` Pass@1 to **`25.0%` (`1/4` tasks)** and proved that **even frontier reasoning models hit two fundamental walls without our Dynamic RSI Harness**:\n\n1. **Wall 1 — Coupled Invariant Blindspots in Symptom-Only Legacy Bug Fixing (`HARD-FLASH-01` & `HARD-FLASH-02`):**  - When asked to fix a regex engine's infinite loop on nullable quantified groups `(a*)*b` , Bare`Gemini Flash` blocked*all* zero-width iterations (`if next_pos == curr_pos: return None` ), breaking mandatory`min_count >= 1` iterations on`(a*)+b` matching`'b'` (`LHS -> None, expected RHS -> (True, {1: (0, 0)})` ).\n  - Feeding our **Introspective Sandbox LHS/RHS Trace** to the RSI Reflector extracted`RULE-HARD-FLASH-01-A1` (*allow zero-width iterations when `count < min_count` to satisfy `min_count`, blocking them only once `count >= min_count`* ), which fixed` HARD-FLASH-01` (`8/8` assertions)**and transferred zero-shot to solve `HARD-FLASH-02-REGEX-TRANSFER-LOOKAROUND` (`8/8` assertions) on the first shot!**\n2. When asked to fix a regex engine's infinite loop on nullable quantified groups \n3. **Wall 2 — System-2 Mental Math Drift on Long Bitwise Trajectories (`HARD-FLASH-04-SYMBOLIC-LFSR-ORACLE`):**  - When asked to embed the 16 round constants produced by 64 steps of a 16-bit Galois LFSR + affine byte-swap, Bare `Gemini Flash` simulated the bitwise math in its`<think>` chain accurately for the**first 20 steps (`i = 0..4`: `56421, 14441, 13412, 12680, 38700`)** , and then suffered a single bitwise carry/XOR slip at**Step 24 (`i = 5`: `127` instead of `4351`)** that corrupted all remaining 11 constants!\n  - Under our **DGM Verified Tool Synthesis (`PLAN_EXECUTE_VERIFY`)** ,` Gemini Flash` authored`synthesize_lfsr_constants()` , which our`SkillRegistry` verified and executed deterministically to inject the exact 16-entry tuple (`3/3` assertions passed).\n4. When asked to embed the 16 round constants produced by 64 steps of a 16-bit Galois LFSR + affine byte-swap, Bare \n\n### Deep Dive 7: Rescuing `deepseek-v4-pro-qwen3.5:9b` (`5.4 GB` Math/STEM Distill) from `0.0%` $\\rightarrow$  `25.0%` SWE-bench & `37.9%` Algorithmic Assertions\n\nWhen we imported `Jackrong/DeepSeek-V4-Pro-Qwen3.5-9B-MTP-GGUF` (`Q4_K_M`, `5.38 GB`, distilled purely on 250K Math/STEM samples with **zero coding SFT data**) onto our local GPU server (`13.8 tok/s`, **4x faster** than `deepseek-r1:32b`), running it raw scored **` 0.0%` (`0/8` on SWE-bench, `0/29` on Algorithmic Assertions)** due to three compounding failure modes that our **Dynamic RSI Harness** systematically diagnosed and overcame:\n\n1. \n**Failure Mode 1 — Ollama `0.34.2`'s `qwen35` BPE Space-Collapsing Detokenizer Bug (`\"    \"` `\" \"`):**$\\rightarrow$ \n  - Inspecting the raw byte output of `deepseek-v4-pro-qwen3.5:9b` (`general.architecture: qwen35` ,`tokenizer.ggml.pre: qwen35` ) revealed that Ollama`0.34.2` 's`qwen35` detokenizer normalizes consecutive ASCII spaces (`\"    \"`$\\rightarrow$ `\" \"` ), collapsing 4-space and 8-space Python indentation down to a single space (`'def foo(x):\\n if x > 0:\\n return x * 2'` ) and causing`IndentationError: expected an indented block` on 100% of multi-level Python functions!\n  - \n**Self-Harness Solution (`OpenAICompatibleClient.chat` & `sanitize_tab_indented_python`):** We discovered that` qwen35` preserves**TAB (`\\t`)** characters 100% intact. Our adapter automatically normalizes 4-space indentation in input focal code prompts to`\\t` , instructs the model to indent with`\\t` , converts`\\t` back to 4 spaces on extraction, and enforces Python's colon-block indentation invariant (`prev_indent + 4` after`:` ).\n2. Inspecting the raw byte output of \n3. \n**Failure Mode 2 — `<think>` Runaway vs. Prose-Preamble Truncation:**  - With `think=True` , the 9B model consumed all`2,048` tokens inside`<think>...</think>` (`len(content)=0` ). With`think=False` , it moved its math derivation into a 1,000-token prose preamble before```` ``` python ```` , truncating the code mid-function.\n  - \n**Self-Harness Solution:** Applying Qwen3.5's ChatML prefill (````<|im_start|>assistant\\n<think>\\n\\n</think>\\n\\n``` python\\n```` ) forced token-1 code emission and lifted Bare Pass@1 from` 0.0%` to**`12.5%` (`1/8`, resolving `sympy__sympy-23262`)** on` SWE-bench Verified` and**`6.9%` (`2/29` assertions)** on the Hard Algorithmic Suite.\n4. With \n5. \n**Failure Mode 3 — Sandbox-Grounded RSI vs. Ungrounded 9B Critic Speculation:**  - On a 9B model with zero coding SFT, running an ungrounded Critic *before* executing tests caused the 9B Critic to hallucinate a bad code snippet (`'(1),'` ) that talked the Generator out of its working Pass@1 fix on`sympy__sympy-23262` .\n  - Grounding the Critic/Reflector in the **actual Sandbox LHS/RHS failure trace** on Attempt 2 preserved`sympy__sympy-23262` (`Attempts=1` ) while repairing`psf__requests-1921` on Attempt 2 (`RULE-SWE-psf__requests-1921-A1` ), lifting`deepseek-v4-pro-qwen3.5:9b` to**`25.0%` (`2/8`) on `SWE-bench Verified`** (matching Bare` deepseek-r1:32b` and Bare`qwen3-coder-next:latest` in just`5.4 GB` VRAM!) and lifting its Hard Algorithmic assertion pass rate**5.5x (`6.9%` `37.9%`, including `6/7` = `85.7%` on `BENCH-03-TX-KV-STORE`)** .$\\rightarrow$\n6. On a 9B model with zero coding SFT, running an ungrounded Critic \n\n## 4. Official `SWE-bench Verified` Instance-by-Instance Results (`qwen3-coder-next`, `deepseek-r1:32b`, & `deepseek-v4-pro-qwen3.5:9b`)\n\n| SWE-bench Verified ID | Repository & Target File | Difficulty | `qwen3-coder` (Bare**RSI** ) | `deepseek-r1:32b` (Bare**RSI** ) | `deepseek-v4-qwen3.5:9b` (Raw**RSI** ) | Key Diagnostic / Learned ACE Rule | \n|---|---|---|---|---|---|---|\n| **`astropy__astropy-14995`** | `astropy/nddata/mixins/ndarithmetic.py` (`_arithmetic_mask` ) | `<15 min` | ❌ **RESOLVED** | ❌ **RESOLVED** | ❌ | `qwen3-coder` &`r1:32b` fixed`operand is None` vs`operand.mask is None` guard under RSI. | \n| **`psf__requests-1921`** | `requests/sessions.py` (`merge_setting` ) | `<15 min` | ✅ **RESOLVED** | ✅ **RESOLVED** | ❌ **RESOLVED** (Att. 2) | **9B model repaired on Attempt 2** after learning`RULE-SWE-psf__requests-1921-A1` from sandbox trace! | \n| **`psf__requests-6028`** | `requests/utils.py` (`prepend_scheme_if_needed` ) | `15m – 1h` | ✅ **RESOLVED** | ❌ **RESOLVED** | ❌ | RSI Critic guided `deepseek-r1:32b` to reconstruct`netloc = '@'.join([auth, netloc])` . | \n| **`sympy__sympy-23262`** | `sympy/utilities/lambdify.py` (`_recursive_to_string` ) | `15m – 1h` | ❌ **RESOLVED** | ❌ **RESOLVED** | ❌ **RESOLVED** (`1st-Shot` ) | **9B model + `qwen35` Tab Adapter** solved 1-element tuple trailing comma`\"(1,)\"` on Pass@1! | \n| **`pytest-dev__pytest-7324`** | `src/_pytest/mark/expression.py` (`expression` /`MatcherAdapter` ) | `15m – 1h` | ❌ **RESOLVED** | ❌ | ❌ | `qwen3-coder` + RSI learned`RULE-SWE-pytest-dev__pytest-7324-A1` to prefix`\"False\"` /`\"True\"` /`\"None\"` . | \n| **`astropy__astropy-12907`** | `astropy/modeling/separable.py` (`_cstack` ) | `15m – 1h` | ❌ | ✅ **RESOLVED** | ❌ | Bare `deepseek-r1:32b` spotted`cright[-right.shape[0]:, -right.shape[1]:] = right` on Pass@1. | \n| **`sympy__sympy-17630`** | `sympy/matrices/expressions/matexpr.py` (`get_postprocessor` ) | `1 – 4h` | ❌ | ❌ | ❌ | Hard `1–4h` SymPy core issue (`ZeroMatrix`` Zero` inside`_mul_postprocessor` ); solved by`Gemini Flash` . | \n| **`sympy__sympy-18698`** | `sympy/polys/polytools.py` (`_symbolic_factor_list` ) | `15m – 1h` | ❌ | ❌ | ❌ | Requires grouping factors of identical multiplicity `k` via`reduce(mul, ...)` ; solved by`Gemini Flash` . |", "url": "https://wpnews.pro/news/benchmarking-qwen3-coder-next-deepseek-r1-and-qwen3-5-with-dynamic-rsi", "canonical_source": "https://github.com/bibble/dynamic-rsi-harness/blob/main/BENCHMARK_REPORT.md", "published_at": "2026-11-14 16:49:34+00:00", "updated_at": "2026-11-14 17:23:44.966554+00:00", "lang": "en", "topics": ["ai-research", "large-language-models", "ai-agents", "ai-tools"], "entities": ["qwen3-coder-next", "deepseek-r1:32b", "SWE-bench Verified", "Dynamic RSI Harness", "ACE", "Darwin Gödel Machine", "DSPy", "AFlow"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/benchmarking-qwen3-coder-next-deepseek-r1-and-qwen3-5-with-dynamic-rsi", "markdown": "https://wpnews.pro/news/benchmarking-qwen3-coder-next-deepseek-r1-and-qwen3-5-with-dynamic-rsi.md", "text": "https://wpnews.pro/news/benchmarking-qwen3-coder-next-deepseek-r1-and-qwen3-5-with-dynamic-rsi.txt", "jsonld": "https://wpnews.pro/news/benchmarking-qwen3-coder-next-deepseek-r1-and-qwen3-5-with-dynamic-rsi.jsonld"}}