Benchmarking Qwen3-Coder-Next, DeepSeek-R1, and Qwen3.5 with Dynamic RSI A benchmark report compares two locally hosted models, qwen3-coder-next (80B total / 3B active MoE) and deepseek-r1:32b (32.8B dense reasoning model), on princeton-nlp/SWE-bench_Verified against published frontier-model scores, testing a "Dynamic RSI Harness" that combines just-in-time macro-runtime compilation with execution-gated micro-cognitive evolution. The harness's authors claim it differs from OPRO, TextGrad, DSPy (MIPROv2), ADAS, AFlow, standalone ACE, the Darwin Gödel Machine and Self-Harness by switching agent topologies per task and gating ACE playbook rules through a sandbox check with a cross-domain semantic threshold of 0.78, which they say prevents rule pollution. The report presents no numeric pass-rate results for either local model in the excerpt provided. Benchmark & Industry RSI Comparison Report: qwen3-coder-next & deepseek-r1:32b vs. Dynamic RSI Harness vs. Frontier Models Comparing our live runs on princeton-nlp/SWE-bench Verified across two local models on our LAN server — qwen3-coder-next:latest 80B total / 3B active MoE and deepseek-r1:32b 32.8B dense reasoning model —against published SWE-bench Verified scores of major frontier models: Chart B: Progression Across Both Live Benchmarks & Both Models qwen3-coder-next vs. deepseek-r1:32b Comparing A Bare Model Pass@1 B1 Dynamic RSI 1st-Shot Multi-Critic + ACE Playbook B2 Dynamic RSI Full Loop w/ Introspective Sandbox & ACE Self-Repair across both local models: Most self-improving agent frameworks in 2025–2026 optimize only one layer of the stack: - Macro-Only Optimizers like ADAS and AFlow search for a better multi-agent graph or workflow topology offline, but keep the inner cognitive context/tools static during task execution. - Micro-Only Optimizers like ACE , TextGrad , OPRO , or LATM refine prompt rules or synthesize a helper function inside a fixed, hardcoded workflow loop . What makes our Dynamic RSI Harness unique is the unification of Two Orthogonal Architectures—JIT Macro-Runtime Compilation + Execution-Gated Micro-Cognitive Evolution—coupled through Pytest-Style Sandbox Introspection. | Framework / Paradigm | Primary Optimization Target | Timescale of Adaptation | Handles Context Window Limits? | Synthesizes Verified Code Tools? | Prevents Prompt/Rule Collapse? | How Our Dynamic RSI Harness Differs | |---|---|---|---|---|---|---| | 1. OPRO / TextGrad DeepMind / Stanford | Prompt text strings "textual gradients" | Offline batch optimization | ❌ No | ❌ No | ❌ No Monolithic prompt rewrites | TextGrad backpropagates natural-language critiques without deterministic LHS/RHS sandbox values or topology switching. | | 2. DSPy MIPROv2 Stanford | Few-shot demonstrations & module instructions | Offline compilation on a training split | ❌ Static pipeline | ❌ No | | DSPy requires an offline labeled training set to compile a fixed graph; our harness compiles JIT per user prompt and self-repairs online. | | 3. ADAS Hu, Lu, Clune et al. | Code-defined agent workflow graphs | Offline evolutionary meta-search | | ❌ Fixed toolset | ❌ No structured rule memory | ADAS searches for one best agent architecture per benchmark offline; our MetaArchitect dynamically switches topologies per task . | | 4. AFlow ICLR 2025/2026 | MCTS search over workflow nodes/edges | Offline MCTS on validation split | ❌ No | ❌ No | ❌ No | AFlow optimizes graph edges via MCTS offline; we combine Macro topology selection with Micro ACE rule & DGM skill evolution online. | | 5. ACE Standalone arXiv:2510.04618 | Structured context playbooks Generator-Reflector-Curator | Online / Cross-task context updates | ❌ Fixed runtime | ❌ No tool synthesis | ✅ Yes Delta rules | Standalone ACE commits reflector rules without a Darwinian Sandbox Gate or Cross-Domain Semantic Threshold = 0.78 , which we proved can cause rule pollution. | | 6. Darwin Gödel Machine DGM Sakana AI, arXiv:2505.22954 | The agent's own codebase & tools | Long-horizon evolutionary branching | ❌ No JIT context middleware | ✅ Yes | ❌ No structured playbook | DGM mutates source code across expensive evolutionary generations; we pair JIT DGM skill synthesis with fast ACE Playbook delta rules and LHS/RHS assertion introspection . | | 7. Self-Harness arXiv:2606.09498 | Harness scaffolding & guardrails via trace mining | Offline / Batch iterative loop | | | ❌ No semantic vector playbook | Self-Harness patches a single global harness per model; we maintain category-specific evolved templates + dense embedding rule retrieval nomic-embed-text . | | 8. Our Dynamic RSI Harness | Dual-Architecture: Macro HarnessConfig + Micro PlaybookRule + SynthesizedSkill + Guardrails | 3 Nested Loops: JIT Loop 0 + Intra-Task Loop 1 + Cross-Task Loop 2 | ✅ Yes Map-Reduce & Scratchpad | ✅ Yes Unit-tested in isolated scope | ✅ Yes Darwinian-gated ACE + Tag/Cosine filter | Unifies Macro Runtime Routing ADAS/Self-Harness + Micro Cognitive/Tool Evolution ACE/DGM in a single runtime. | When we tested our harness on both a 3B-active-parameter MoE coding model qwen3-coder-next:latest and a 32B dense reasoning model deepseek-r1:32b against SWE-bench Verified and adversarial coding tasks, five deep engineering mechanisms separated our system from textbook implementations of ACE, ADAS, and DGM: flowchart TD subgraph Macro "Architecture 1: Macro-RSI JIT Runtime Compiler — ADAS / Self-Harness " Task "Incoming User Task" -- Profiler "Task Bottleneck & Model Profiler" Profiler -- |"Context-Heavy"| MapReduce "DIVIDE AND CONQUER +\nHIERARCHICAL MAP REDUCE" Profiler -- |"Symbolic / Math"| PlanVerify "PLAN EXECUTE VERIFY +\nJIT DGM Skill Mounting" Profiler -- |"Complex Code / SWE-bench"| MultiCritic "MULTI CRITIC DEBATE +\nModel-Aware Prefill & Stop Guards" end subgraph Micro "Architecture 2: Micro-RSI Cognitive & Tool Evolution — ACE + DGM " MultiCritic & PlanVerify & MapReduce -- Sandbox "Introspective Sandbox Gate\n Evaluates exact LHS vs. RHS & Rebinds Module Closures " Sandbox -- |"Pass"| DarwinGate "Darwinian Commit Gate\n Commits Rule/Skill ONLY if repair passes F2P & P2P " Sandbox -- |"Fail: Exact LHS/RHS Trace"| Reflector "Tri-Modal Reflector Two-Block Format \n1. ACE Delta Rule RULE WHEN/DO/AVOID \n2. Standard Python Fenced Code Block\n3. Post-Execution Guardrail" Reflector -- Sandbox DarwinGate -- VectorDB "Hybrid Tag + Embedding Memory\n nomic-embed-text, cosine = 0.78 " end VectorDB -.- |"Zero-Shot Transfer to Next Task"| Profiler Deep Dive 1: Decoupling "Wrong Runtime" Macro from "Cognitive Blindspot Inside the Right Runtime" Micro - The Industry Problem: Systems like AFlow and ADAS assume that if an agent fails, you need a different workflow graph. Systems like ACE and TextGrad assume the workflow graph is fine and you only need better prompt instructions. - Our Solution: Both failure modes happen in reality, and they require different fixes: - When TASK-01-LONG-CONTEXT failed because 3,910 tokens overflowed a 1,200 -token attention window, no prompt instruction could fix "Lost in the Middle." Our Macro-RSI mutated the HarnessConfig to DIVIDE AND CONQUER + HIERARCHICAL MAP REDUCE , dropping peak context to 400 tokens. - Conversely, on BENCH-01-CUSTOM-DSL and pytest-dev pytest-7324 , the MULTI CRITIC DEBATE runtime was already optimal , yet the model failed due to micro-level cognitive traps confusing unary vs. binary - after , or turning "False" into an AST constant instead of a prefixed identifier . Our Micro-RSI extracted surgical WHEN - DO - AVOID rules from the trace and solved both tasks without touching the topology. - When - The Industry Problem: Standard Darwin Gödel Machine DGM and code-repair loops run exec or pytest and feed raw stack traces back to the LLM. In Python, a failed assert f x == expected raises an empty AssertionError: . When we ran qwen3-coder-next:latest on rolling xor checksum , the model got stuck in a loop because it couldn't tell which assertion line failed or what its function actually returned 834 vs its mental-math guess 946 . - Our Solution: In SkillRegistry.register and verify and execute swebench patch , we built an AST/Expression Introspection Sandbox that splits failed assertions and evaluates both LHS and RHS in the live namespace LHS evaluated to 834, expected RHS 946 . This single diagnostic upgrade turned multi-attempt deadlocks into Attempt-2 self-repairs . - The Industry Problem: In the original ACE paper arXiv:2510.04618 , the Reflector generates delta rules after every trajectory and the Curator merges them into the playbook. During our live run on BENCH-02-TRANSFER-DSL , when qwen3-coder-next:latest 's parser failed a test, the Reflector blamed the specification and wrote a poison rule: "The spec's precedence labels are reversed; swap + and back to standard PEMDAS " - Our Solution: We combined Self-Harness's regression gate with ACE's curator run head to head benchmark.py & run swebench verified.py : a candidate PlaybookRule is treated as an untrusted hypothesis and is only committed to PlaybookMemory if the accompanying code repair passes the deterministic sandbox gate FAIL TO PASS + PASS TO PASS . Speculative rules from failed repairs are automatically discarded. - The Industry Problem: Memory-based agents typically retrieve top-$K$ past reflections using raw cosine similarity from an embedding model nomic-embed-text . In our first SWE-bench Verified run, we discovered that any two Python bug reports share a baseline cosine similarity of ~0.60 , causing an Astropy NDDataRef.mask rule to mount into a Pytest AST compiler task and degrade first-shot accuracy. - Our Solution: In PlaybookMemory.retrieve for task , we implemented Hybrid Domain-Gated Retrieval : a rule is mounted only if it shares an explicit symbolic domain tag with the target task or exceeds a strict cross-domain semantic threshold cosine similarity = 0.78 , weighted by its empirical Bayes confidence times succeeded / times applied . - The Industry Problem: When we switched from qwen3-coder-next:latest to deepseek-r1:32b a distilled reasoning model that wraps every response in