{"slug": "sandboxed-code-evaluation-for-ai-generated-outputs-how-i-built-safecode-arena", "title": "Sandboxed Code Evaluation for AI-Generated Outputs — How I Built SafeCode Arena", "summary": "A developer built SafeCode Arena, an automated verifier that evaluates AI-generated code candidates across five axes—correctness, security, performance, maintainability, and resource usage—and scores them to help developers make defensible choices. The tool compiles candidates to WebAssembly with hard limits on instructions and memory, and logs evaluations to SQLite to detect regressions and build an audit trail. It supports both Rust and Python, allowing cross-language comparisons on the same rubric.", "body_md": "You're using Cursor, Claude Code, or GitHub Copilot. The AI gives you three implementation options for the same feature.\n\n```\nAI: \"Here are three approaches:\n  A) Quick but uses unsafe\n  B) Slower but memory-safe\n  C) Balanced tradeoffs\"\n\nYou: \"Which one should I ship?\"\nAI: \"It depends...\"\n```\n\nThat **\"it depends\"** is where responsibility falls through the cracks.\n\nTests tell you if code compiles and passes specs. But they don't tell you about security, performance, maintainability, or resource limits — all at once. You end up making the call by gut feel.\n\n**This essay is about building a system that doesn't let that happen.**\n\nI built [SafeCode Arena](https://github.com/flipslidersand/safecode-arena) — an automated verifier that evaluates code candidates across five axes simultaneously, scores each, and surfaces the tradeoffs.\n\n| Axis | Weight | Computation |\n|---|---|---|\nCorrectness |\n50% | compile (40%) + tests (40%) + property tests (20%) |\nSecurity |\n20% | unsafe heuristics (50%) + clippy warnings (50%) |\nPerformance |\n15% | relative compile+test time across candidates |\nMaintainability |\n10% | function-length heuristics (60%) + clippy (40%) |\nResource Usage |\n5% | pass/fail of sandboxed Wasm execution |\n\n`unsafe`\n\ncompiles fine, but you need to detect it yourself\n\n```\nCandidate A: 85 points\n├─ correctness:     100 (all tests pass)\n├─ security:        60 (2 unsafe blocks flagged)\n├─ performance:     70 (10% slower than B)\n├─ maintainability: 85 (avg function 25 lines)\n└─ resource_usage:  80 (Wasm sandbox: 512MB, OK)\n\nCandidate B: 92 points ✓ Recommended\n├─ correctness:     95 (1 edge case warning)\n├─ security:        95 (no unsafe)\n├─ performance:     95 (fastest)\n├─ maintainability: 88 (avg function 20 lines)\n└─ resource_usage:  100 (Wasm sandbox: 200MB)\n```\n\nNow the choice is **defensible**. You can explain *why* B won.\n\nAI-generated code can hang, crash, or consume unbounded memory. Running it bare-metal is reckless.\n\n``` js\n// Compile the candidate → link to Wasm\nlet instance = linker.instantiate(&module)?;\nlet run_fn = instance.get_typed_func::<(), ()>(&mut store, \"run\")?;\n\n// Set fuel (instruction budget) and memory limit\nstore.set_fuel(100_000_000)?;  // ~100M instructions\nstore.set_memory_limit(1_000_000_000)?;  // 1GB max\n\n// Run with hard limits\nrun_fn.call(&mut store, ())?;  // Will trap if it exceeds limits\n```\n\n**Result**: The code runs in a box. If it tries to loop forever or allocate unbounded memory, it's killed cleanly with a trap. No crash, no OOM, just a resource_usage score of 0.\n\nComparing one candidate is useful. Comparing against *past* candidates is gold.\n\n```\n// Log every evaluation to SQLite\ndb.execute(\n    \"INSERT INTO evals\n     (spec_hash, code_hash, correctness, security, perf, maintain, resource, timestamp)\n     VALUES (?, ?, ?, ?, ?, ?, ?, ?)\",\n    params![\n        hash(spec), hash(code),\n        score.correctness, score.security, score.perf, score.maintain, score.resource,\n        now()\n    ],\n)?;\n\n// Detect regressions\nlet best = db.query_row(\n    \"SELECT score FROM evals WHERE spec_hash = ? ORDER BY timestamp DESC LIMIT 1\",\n    [spec_hash],\n    |row| row.get(0),\n)?;\n\nif new_score < best - threshold {\n    println!(\"⚠️  Regression detected: {} → {}\", best, new_score);\n}\n```\n\n**Why this matters**: You can build a changelog of \"every candidate for this spec, ranked\". Future devs can see what was tried and why something was chosen. That's audit trail.\n\nRust and Python coexist everywhere now. SafeCode Arena doesn't force you to rewrite everything in one language.\n\n``` js\nlet scorer = match candidate.extension() {\n    \"rs\" => RustScorer::new(),\n    \"py\" => PythonScorer::new(),\n    _ => return Err(\"Unsupported\"),\n};\n\nlet score = scorer.evaluate(candidate, tests)?;\n```\n\n**Benefit**: You can compare \"the Rust solution\" vs. \"the Python solution\" on the same rubric. Language is transparent to the ranking.\n\n```\n# .github/workflows/evaluate-candidate.yml\nname: Evaluate PR Candidate\n\non: pull_request\n\njobs:\n  safecode:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v4\n      - name: Evaluate candidates\n        run: |\n          safecode evaluate pr-candidate.rs \\\n            --tests tests/ \\\n            --format json \\\n            --db history.db > score.json\n\n      - name: Post score to PR\n        run: |\n          gh pr comment -b \"$(cat score.json | jq -r '.report')\"\n```\n\n**Result**: Every PR that proposes code gets a scored report. Developers see the scorecard before merge. No surprises in production.\n\nHere's what kept me thinking while building this.\n\n**Choosing a rubric is choosing what you trust.**\n\n```\nReviewer: \"This code feels unsafe.\"\n(Why? Personal experience, gut instinct, luck)\n\nNext reviewer: \"Is that still true?\"\n(No way to know. Depends on their experience.)\n```\n\nThis doesn't scale. It's not reproducible. It's not teachable.\n\n```\n\"Unsafe block found → -5 security points (non-negotiable)\"\n\"Property test passes → +10 correctness points\"\n...\n```\n\nNow:\n\nIn the age of AI-generated code, **an explicit rubric is how you take responsibility for trust.**\n\nYou're not saying \"AI is always right\" or \"AI is always wrong.\" You're saying: **\"Here's what I checked. Here's the score. Here's why I merged it.\"**\n\n```\n# 1. AI generates 3 candidates (you pick the most promising)\n# 2. Run the scorer\n$ safecode evaluate solution_a.rs solution_b.rs \\\n    --tests tests/ \\\n    --db evals.db\n\n# 3. Results\nSafeCode Arena Comparison Report\n─────────────────────────────────\nSolution A: 78 points (correctness 85, security 60, perf 70, maintain 80, resource 95)\nSolution B: 91 points (correctness 100, security 90, perf 95, maintain 85, resource 100) ← Winner\n\n# 4. Confidence\n(Run regression check: best_previous_score = 75, new = 91 → no regression, all good)\n\n# 5. Merge\nYou ship Solution B, confident you can explain your choice.\n```\n\nSafeCode Arena has completed phases 1–5:\n\nComing next:\n\nCursor, Claude Code, GitHub Copilot, and similar tools are shipping code daily. The industry collectively acts like \"the AI picked it, so it's fine,\" which is how you end up with security holes in production.\n\n**SafeCode Arena is my answer to that:** Not \"don't trust AI,\" but \"trust AI + verify systematically.\"\n\n**GitHub**: [https://github.com/flipslidersand/safecode-arena](https://github.com/flipslidersand/safecode-arena)\n\n**License**: MIT\n\n**Stack**: Rust, Wasm (wasmtime), SQLite, Python", "url": "https://wpnews.pro/news/sandboxed-code-evaluation-for-ai-generated-outputs-how-i-built-safecode-arena", "canonical_source": "https://dev.to/flipslidersand/sandboxed-code-evaluation-for-ai-generated-outputs-how-i-built-safecode-arena-1468", "published_at": "2026-08-19 15:31:13+00:00", "updated_at": "2026-08-19 15:43:09.241418+00:00", "lang": "en", "topics": ["developer-tools", "ai-tools", "ai-products", "ai-safety"], "entities": ["SafeCode Arena", "Cursor", "Claude Code", "GitHub Copilot", "Rust", "Python", "WebAssembly", "SQLite"], "alternates": {"html": "https://wpnews.pro/news/sandboxed-code-evaluation-for-ai-generated-outputs-how-i-built-safecode-arena", "markdown": "https://wpnews.pro/news/sandboxed-code-evaluation-for-ai-generated-outputs-how-i-built-safecode-arena.md", "text": "https://wpnews.pro/news/sandboxed-code-evaluation-for-ai-generated-outputs-how-i-built-safecode-arena.txt", "jsonld": "https://wpnews.pro/news/sandboxed-code-evaluation-for-ai-generated-outputs-how-i-built-safecode-arena.jsonld"}}