Benchmarking Qwen3-Coder-Next, DeepSeek-R1, and Qwen3.5 with Dynamic RSI

Comparing our live runs on princeton-nlp/SWE-bench_Verified across two local models on our LAN server—qwen3-coder-next:latest (80B total / 3B active MoE) and deepseek-r1:32b (32.8B dense reasoning model)—against published SWE-bench Verified scores of major frontier models:

Chart B: Progression Across Both Live Benchmarks & Both Models (qwen3-coder-next vs. deepseek-r1:32b)

Comparing [A] Bare Model (Pass@1) [B1] Dynamic RSI (1st-Shot Multi-Critic + ACE Playbook) [B2] Dynamic RSI (Full Loop w/ Introspective Sandbox & ACE Self-Repair) across both local models:

Most self-improving agent frameworks in 2025–2026 optimize only one layer of the stack:

• Macro-Only Optimizers (likeADAS andAFlow ) search for a better multi-agent graph or workflow topology offline, but keep the inner cognitive context/tools static during task execution.
• Micro-Only Optimizers (likeACE ,TextGrad ,OPRO , orLATM ) refine prompt rules or synthesize a helper function inside afixed, hardcoded workflow loop .

What makes our Dynamic RSI Harness unique is the unification of Two Orthogonal Architectures—JIT Macro-Runtime Compilation + Execution-Gated Micro-Cognitive Evolution—coupled through Pytest-Style Sandbox Introspection.

When we tested our harness on both a 3B-active-parameter MoE coding model (qwen3-coder-next:latest) and a 32B dense reasoning model ( deepseek-r1:32b) against SWE-bench Verified and adversarial coding tasks, five deep engineering mechanisms separated our system from textbook implementations of ACE, ADAS, and DGM:

flowchart TD
    subgraph Macro["Architecture 1: Macro-RSI (JIT Runtime Compiler — ADAS / Self-Harness)"]
        Task["Incoming User Task"] --> Profiler["Task Bottleneck & Model Profiler"]
        Profiler -->|"Context-Heavy"| MapReduce["DIVIDE_AND_CONQUER +\nHIERARCHICAL_MAP_REDUCE"]
        Profiler -->|"Symbolic / Math"| PlanVerify["PLAN_EXECUTE_VERIFY +\nJIT DGM Skill Mounting"]
        Profiler -->|"Complex Code / SWE-bench"| MultiCritic["MULTI_CRITIC_DEBATE +\nModel-Aware Prefill & Stop Guards"]
    end

    subgraph Micro["Architecture 2: Micro-RSI (Cognitive & Tool Evolution — ACE + DGM)"]
        MultiCritic & PlanVerify & MapReduce --> Sandbox["Introspective Sandbox Gate\n(Evaluates exact LHS vs. RHS & Rebinds Module Closures)"]
        Sandbox -->|"Pass"| DarwinGate["Darwinian Commit Gate\n(Commits Rule/Skill ONLY if repair passes F2P & P2P)"]
        Sandbox -->|"Fail: Exact LHS/RHS Trace"| Reflector["Tri-Modal Reflector (Two-Block Format)\n1. ACE Delta Rule (RULE_WHEN/DO/AVOID)\n2. Standard Python Fenced Code Block\n3. Post-Execution Guardrail"]
        Reflector --> Sandbox
        DarwinGate --> VectorDB[("Hybrid Tag + Embedding Memory\n(nomic-embed-text, cosine >= 0.78)")]
    end

    VectorDB -.->|"Zero-Shot Transfer to Next Task"| Profiler

Deep Dive 1: Decoupling "Wrong Runtime" (Macro) from "Cognitive Blindspot Inside the Right Runtime" (Micro)

• The Industry Problem: Systems likeAFlow andADAS assume that if an agent fails, you need a different workflow graph. Systems likeACE andTextGrad assume the workflow graph is fine and you only need better prompt instructions.

• Our Solution: Both failure modes happen in reality, and they require different fixes:

  • When TASK-01-LONG-CONTEXT failed because3,910 tokens overflowed a1,200 -token attention window, no prompt instruction could fix "Lost in the Middle." OurMacro-RSI mutated theHarnessConfig toDIVIDE_AND_CONQUER +HIERARCHICAL_MAP_REDUCE , dropping peak context to400 tokens.
  • Conversely, on BENCH-01-CUSTOM-DSL andpytest-dev__pytest-7324 , theMULTI_CRITIC_DEBATE runtime wasalready optimal , yet the model failed due to micro-level cognitive traps (confusing unary vs. binary- after) , or turning"False" into an AST constant instead of a prefixed identifier). OurMicro-RSI extracted surgicalWHEN -> DO -> AVOID rules from the trace and solved both tasks without touching the topology.

• When

• The Industry Problem: StandardDarwin Gödel Machine (DGM) and code-repair loops runexec() orpytest and feed raw stack traces back to the LLM. In Python, a failedassert f(x) == expected raises an emptyAssertionError: . When we ranqwen3-coder-next:latest onrolling_xor_checksum , the model got stuck in a loop because it couldn't tellwhich assertion line failed or what its function actually returned (834 vs its mental-math guess946 ).

• Our Solution: InSkillRegistry.register_and_verify andexecute_swebench_patch , we built anAST/Expression Introspection Sandbox that splits failed assertions and evaluates bothLHS andRHS in the live namespace (LHS evaluated to 834, expected RHS 946 ). This single diagnostic upgrade turned multi-attempt deadlocks intoAttempt-2 self-repairs .

• The Industry Problem: In the originalACE paper (arXiv:2510.04618 ), the Reflector generates delta rules after every trajectory and the Curator merges them into the playbook. During our live run onBENCH-02-TRANSFER-DSL , whenqwen3-coder-next:latest 's parser failed a test, the Reflectorblamed the specification and wrote a poison rule:"The spec's precedence labels are reversed; swap + and * back to standard PEMDAS!"

• Our Solution: We combinedSelf-Harness's regression gate withACE's curator (run_head_to_head_benchmark.py &run_swebench_verified.py ): a candidatePlaybookRule is treated as anuntrusted hypothesis and isonly committed to PlaybookMemory if the accompanying code repair passes the deterministic sandbox gate (FAIL_TO_PASS + PASS_TO_PASS) . Speculative rules from failed repairs are automatically discarded.

The Industry Problem: Memory-based agents typically retrieve top-$K$ past reflections using raw cosine similarity from an embedding model (nomic-embed-text ). In our firstSWE-bench Verified run, we discovered that any two Python bug reports share a baseline cosine similarity of~0.60 , causing an AstropyNDDataRef.mask rule to mount into a Pytest AST compiler task and degrade first-shot accuracy. #

Our Solution: InPlaybookMemory.retrieve_for_task , we implementedHybrid Domain-Gated Retrieval : a rule is mounted only if it shares an explicit symbolicdomain_tag with the target taskor exceeds a strict cross-domain semantic threshold (cosine_similarity >= 0.78 ), weighted by its empirical Bayes confidence (times_succeeded / times_applied ).

• The Industry Problem: When we switched fromqwen3-coder-next:latest to**deepseek-r1:32b** (a distilled reasoning model that wraps every response in<think>...</think> ), two severe model-specific failure modes emerged:
  1. Token Budget Starvation Inside <think>: At 3.4 tok/s ,deepseek-r1:32b spent 2,000+ tokens ruminating inside<think> before closing</think> , returning an emptymessage.content (SyntaxError: Empty code ).
  2. JSON-Escaped Code Corruption: When the Reflector askeddeepseek-r1:32b to return"repaired_code" as a JSON-escaped string, the model mis-escaped\n and\" inside multi-line Python docstrings, causing every self-repair attempt to fail JSON parsing or Pythoncompile() .
• Our Self-Harness Solution: Following theSelf-Harness paradigm (arXiv:2606.09498 ), we adaptedOpenAICompatibleClient.chat and the Reflector interface:
  • Added an Assistant <think>\n</think>\n prefill + \n``` stop-sequence guard for R1 models, externalizing reasoning into our explicitRoot-Cause Critic + ACE Playbook stages while cutting per-call latency by4x .
  • Replaced JSON-embedded code strings with a Two-Block Reflector Format (RULE_WHEN: /RULE_DO: /RULE_AVOID: headers followed by a native``` python fenced code block), immediately unlocking deepseek-r1:32b 's2x jump (25.0% -> 50.0%) on SWE-bench Verified .
• Added an

3. Side-by-Side Comparison Table Across All 4 Tested Models (deepseek-v4-pro-qwen3.5:9b, deepseek-r1:32b, qwen3-coder-next, & Gemini Flash) & Frontier Models #

Deep Dive 6: How Our Dynamic RSI Harness Lifted Gemini Flash from 25.0% $\rightarrow$ 100.0% on the HARD-FLASH Suite

While Gemini Flash saturated the focal-window SWE-bench Verified (8/8) and Algorithmic (4/4) suites on Pass@1, testing it on our 4-Task Symptom-Only Legacy & Symbolic-Oracle Suite (HARD-FLASH-01 to HARD-FLASH-04) dropped Bare Gemini Flash Pass@1 to 25.0% (1/4 tasks) and proved that even frontier reasoning models hit two fundamental walls without our Dynamic RSI Harness:

1. Wall 1 — Coupled Invariant Blindspots in Symptom-Only Legacy Bug Fixing (HARD-FLASH-01 & HARD-FLASH-02): - When asked to fix a regex engine's infinite loop on nullable quantified groups (a*)*b , BareGemini Flash blockedall zero-width iterations (if next_pos == curr_pos: return None ), breaking mandatorymin_count >= 1 iterations on(a*)+b matching'b' (LHS -> None, expected RHS -> (True, {1: (0, 0)}) ).

• Feeding our Introspective Sandbox LHS/RHS Trace to the RSI Reflector extractedRULE-HARD-FLASH-01-A1 (allow zero-width iterations when count < min_count to satisfy min_count, blocking them only once count >= min_count ), which fixed HARD-FLASH-01 (8/8 assertions)and transferred zero-shot to solve HARD-FLASH-02-REGEX-TRANSFER-LOOKAROUND (8/8 assertions) on the first shot!

2. When asked to fix a regex engine's infinite loop on nullable quantified groups
3. Wall 2 — System-2 Mental Math Drift on Long Bitwise Trajectories (HARD-FLASH-04-SYMBOLIC-LFSR-ORACLE): - When asked to embed the 16 round constants produced by 64 steps of a 16-bit Galois LFSR + affine byte-swap, Bare Gemini Flash simulated the bitwise math in its<think> chain accurately for thefirst 20 steps (i = 0..4: 56421, 14441, 13412, 12680, 38700) , and then suffered a single bitwise carry/XOR slip atStep 24 (i = 5: 127 instead of 4351) that corrupted all remaining 11 constants!

• Under our DGM Verified Tool Synthesis (PLAN_EXECUTE_VERIFY) , Gemini Flash authoredsynthesize_lfsr_constants() , which ourSkillRegistry verified and executed deterministically to inject the exact 16-entry tuple (3/3 assertions passed).

4. When asked to embed the 16 round constants produced by 64 steps of a 16-bit Galois LFSR + affine byte-swap, Bare

Deep Dive 7: Rescuing deepseek-v4-pro-qwen3.5:9b (5.4 GB Math/STEM Distill) from 0.0% $\rightarrow$ 25.0% SWE-bench & 37.9% Algorithmic Assertions

When we imported Jackrong/DeepSeek-V4-Pro-Qwen3.5-9B-MTP-GGUF (Q4_K_M, 5.38 GB, distilled purely on 250K Math/STEM samples with zero coding SFT data) onto our local GPU server (13.8 tok/s, 4x faster than deepseek-r1:32b), running it raw scored 0.0% (0/8 on SWE-bench, 0/29 on Algorithmic Assertions) due to three compounding failure modes that our Dynamic RSI Harness systematically diagnosed and overcame:

Failure Mode 1 — Ollama 0.34.2's qwen35 BPE Space-Collapsing Detokenizer Bug (" " " "):$\rightarrow$

• Inspecting the raw byte output of deepseek-v4-pro-qwen3.5:9b (general.architecture: qwen35 ,tokenizer.ggml.pre: qwen35 ) revealed that Ollama0.34.2 'sqwen35 detokenizer normalizes consecutive ASCII spaces (" "$\rightarrow$ " " ), collapsing 4-space and 8-space Python indentation down to a single space ('def foo(x):\n if x > 0:\n return x * 2' ) and causingIndentationError: expected an indented block on 100% of multi-level Python functions!

Self-Harness Solution (OpenAICompatibleClient.chat & sanitize_tab_indented_python): We discovered that qwen35 preservesTAB (\t) characters 100% intact. Our adapter automatically normalizes 4-space indentation in input focal code prompts to\t , instructs the model to indent with\t , converts\t back to 4 spaces on extraction, and enforces Python's colon-block indentation invariant (prev_indent + 4 after: ). 2. Inspecting the raw byte output of 3. Failure Mode 2 — <think> Runaway vs. Prose-Preamble Truncation: - With think=True , the 9B model consumed all2,048 tokens inside<think>...</think> (len(content)=0 ). Withthink=False , it moved its math derivation into a 1,000-token prose preamble before``` python , truncating the code mid-function. #

Self-Harness Solution: Applying Qwen3.5's ChatML prefill (<|im_start|>assistant\n<think>\n\n</think>\n\n``` python\n ) forced token-1 code emission and lifted Bare Pass@1 from 0.0% to**12.5% (1/8, resolving sympy__sympy-23262)** on SWE-bench Verified and**6.9% (2/29 assertions)** on the Hard Algorithmic Suite. 4. With 5. Failure Mode 3 — Sandbox-Grounded RSI vs. Ungrounded 9B Critic Speculation: - On a 9B model with zero coding SFT, running an ungrounded Critic before executing tests caused the 9B Critic to hallucinate a bad code snippet ('(1),' ) that talked the Generator out of its working Pass@1 fix onsympy__sympy-23262 .

• Grounding the Critic/Reflector in the actual Sandbox LHS/RHS failure trace on Attempt 2 preservedsympy__sympy-23262 (Attempts=1 ) while repairingpsf__requests-1921 on Attempt 2 (RULE-SWE-psf__requests-1921-A1 ), liftingdeepseek-v4-pro-qwen3.5:9b to**25.0% (2/8) on SWE-bench Verified** (matching Bare deepseek-r1:32b and Bareqwen3-coder-next:latest in just5.4 GB VRAM!) and lifting its Hard Algorithmic assertion pass rate5.5x (6.9% 37.9%, including 6/7 = 85.7% on BENCH-03-TX-KV-STORE) .$\rightarrow$

6. On a 9B model with zero coding SFT, running an ungrounded Critic

4. Official SWE-bench Verified Instance-by-Instance Results (qwen3-coder-next, deepseek-r1:32b, & deepseek-v4-pro-qwen3.5:9b) #