{"slug": "i-built-an-agent-evaluation-harness-for-local-ai-what-most-people-get-wrong", "title": "I Built an Agent Evaluation Harness for Local AI — What Most People Get Wrong", "summary": "A developer built an agent evaluation harness for local AI agents used in trading, addressing common pitfalls in agent testing such as lack of variation, failure modes, baselines, continuous testing, and cost measurement. The harness generates 300 test cases per agent per run, tracking metrics like accuracy, precision, recall, F1, latency, cost, and hallucination rate, and achieved a trading profit of ₹96,000 over six months.", "body_md": "*DOYR | Not financial/legal/tax advice. For educational purposes only.*\n\nThree months ago, I started building AI agents for my trading business.\n\nFirst agent: Fetches Nifty option chain data.\n\nSecond agent: Analyzes PCR, OI, max pain.\n\nThird agent: Predicts direction using XGBoost.\n\nFourth agent: Sends Telegram alerts.\n\nI had **4 agents** doing 5 jobs. And I had **no idea if they were any good**.\n\nSure, my trading results were +₹96,000 over 6 months. But was that because my agents were smart, or because I was overriding their bad decisions?\n\nI couldn't answer that question. So I built something to find out.\n\n**An Agent Evaluation Harness.**\n\nAn **Agent Evaluation Harness** is a systematic framework for testing AI agents.\n\nIt answers one question: **\"How good is this agent, actually?\"**\n\nMost people skip evaluation. They build an agent, test it once or twice manually, and call it \"done.\" Then they wonder why it fails in production.\n\nAn evaluation harness forces you to:\n\nThis is not optional. This is **engineering 101**.\n\nI reviewed 50+ \"agent evaluation\" frameworks online. Here's what I found:\n\n**What they do:** Test the agent on one task. \"Can it book a flight?\" → Yes/No.\n\n**What's wrong:** Real agents face **thousands of variations** of the same task. \"Book a flight from Delhi to Mumbai on Friday\" vs \"Book a flight from Delhi to Mumbai next Friday\" vs \"Book a flight from Delhi to Mumbai on August 15th.\"\n\nA good harness tests **variations**, not just one example.\n\n**What they do:** Test happy paths only. \"Book a flight when everything works.\"\n\n**What's wrong:** Real world is messy. What if:\n\nA good harness tests **failure modes**, not just success paths.\n\n**What they do:** Report \"agent accuracy: 85%.\"\n\n**What's wrong:** 85% compared to what? A random guess? A human? A previous version?\n\nA good harness always has a **baseline** for comparison.\n\n**What they do:** Test once, ship, never test again.\n\n**What's wrong:** Every code change can break something. A good harness runs **continuously** and flags regressions.\n\n**What they do:** Measure accuracy only.\n\n**What's wrong:** An agent that's 95% accurate but costs ₹10 per query is worse than an agent that's 85% accurate and costs ₹0.01 per query.\n\nA good harness measures **accuracy AND cost**.\n\nI built a harness specifically for **local AI agents**. Here's the architecture:\n\n```\n┌─────────────────────────────────────────┐\n│         Agent Evaluation Harness        │\n├─────────────────────────────────────────┤\n│                                         │\n│  ┌─────────────┐  ┌──────────────┐     │\n│  │ Test Suite  │  │   Runner     │     │\n│  │ Generator   │  │   Engine     │     │\n│  └─────────────┘  └──────────────┘     │\n│         ↓                  ↓            │\n│  ┌─────────────┐  ┌──────────────┐     │\n│  │   Agent     │  │   Metrics    │     │\n│  │   Under     │  │  Collector   │     │\n│  │   Test      │  │              │     │\n│  └─────────────┘  └──────────────┘     │\n│         ↓                  ↓            │\n│  ┌─────────────┐  ┌──────────────┐     │\n│  │  Baseline   │  │   Report     │     │\n│  │  Comparator │  │  Generator   │     │\n│  └─────────────┘  └──────────────┘     │\n│                                         │\n└─────────────────────────────────────────┘\n```\n\nCreates test cases automatically.\n\n**For my trading agents, I generate:**\n\n**Total: 300 test cases per agent per evaluation run.**\n\nExecutes each test case against the agent.\n\n**For each test case:**\n\n**Runtime:** ~30 seconds per agent for 300 test cases.\n\nTracks multiple metrics, not just accuracy.\n\n**Metrics I track:**\n\n| Metric | Definition | Target |\n|---|---|---|\nAccuracy |\n% of correct predictions | >60% |\nPrecision |\n% of BUY signals that were profitable | >65% |\nRecall |\n% of profitable trades that were signaled | >60% |\nF1 Score |\nHarmonic mean of precision/recall | >62% |\nLatency |\nTime per prediction | <1s |\nCost |\n₹ per prediction | <₹0.01 |\nHallucination rate |\n% of outputs with made-up numbers | <5% |\nConsistency |\nAccuracy across market regimes | ±5% |\nOverride rate |\n% of signals I overrode | <20% |\nFalse positive rate |\nBUY signals that lost money | <35% |\nFalse negative rate |\nMissed profitable trades | <40% |\n\nCompares current agent against:\n\nCreates a detailed report:\n\n```\n## Agent Evaluation Report: Nifty Signal Agent v2.3\n\n### Overall Performance\n- Accuracy: 62.3% (target: 60%) ✅\n- Precision: 67.1% (target: 65%) ✅\n- Recall: 61.8% (target: 60%) ✅\n- F1 Score: 64.4% (target: 62%) ✅\n\n### Cost & Latency\n- Latency: 0.18s (target: <1s) ✅\n- Cost: ₹0.001/trade (target: <₹0.01) ✅\n- Hallucination rate: 3.2% (target: <5%) ✅\n\n### Comparison to Baselines\n- Random: 50.0% ❌ (agent wins)\n- Previous version (v2.2): 61.5% ✅ (improved)\n- Human baseline: 67.0% ❌ (human still better)\n- Sensibull: 70.0% ❌ (commercial tool better)\n\n### Regressions\n- No regressions detected ✅\n\n### Recommendations\n1. Precision improved, but recall dropped — need to reduce false negatives\n2. Hallucination rate still above 2% target — add stricter output validation\n3. Human baseline still better — investigate override patterns\npython\nimport pandas as pd\nimport numpy as np\nfrom datetime import datetime, timedelta\n\nclass TestSuiteGenerator:\n    def __init__(self, data_path):\n        self.data = pd.read_csv(data_path)\n\n    def generate_normal_cases(self, n=100):\n        \"\"\"Generate test cases from normal market conditions.\"\"\"\n        cases = []\n        for _ in range(n):\n            snapshot = self.data.sample(1).iloc[0]\n            cases.append({\n                'type': 'normal',\n                'input': snapshot.to_dict(),\n                'expected': self._get_expected_signal(snapshot)\n            })\n        return cases\n\n    def generate_high_volatility_cases(self, n=50):\n        \"\"\"Generate test cases from high VIX periods.\"\"\"\n        high_vix = self.data[self.data['vix'] > 20]\n        cases = []\n        for _ in range(n):\n            snapshot = high_vix.sample(1).iloc[0]\n            cases.append({\n                'type': 'high_volatility',\n                'input': snapshot.to_dict(),\n                'expected': self._get_expected_signal(snapshot)\n            })\n        return cases\n\n    def generate_expiry_week_cases(self, n=50):\n        \"\"\"Generate test cases from expiry week.\"\"\"\n        expiry_week = self.data[self.data['days_to_expiry'] <= 2]\n        cases = []\n        for _ in range(n):\n            snapshot = expiry_week.sample(1).iloc[0]\n            cases.append({\n                'type': 'expiry_week',\n                'input': snapshot.to_dict(),\n                'expected': self._get_expected_signal(snapshot)\n            })\n        return cases\n\n    def generate_edge_cases(self, n=50):\n        \"\"\"Generate edge cases with missing/corrupt data.\"\"\"\n        cases = []\n        for _ in range(n):\n            snapshot = self.data.sample(1).iloc[0].to_dict()\n\n            # Randomly corrupt 1-2 features\n            corrupt_count = np.random.randint(1, 3)\n            features_to_corrupt = np.random.choice(\n                ['pcr', 'oi_change', 'max_pain', 'rsi'],\n                size=corrupt_count,\n                replace=False\n            )\n\n            for feat in features_to_corrupt:\n                snapshot[feat] = np.nan  # Missing data\n\n            cases.append({\n                'type': 'edge_case',\n                'input': snapshot,\n                'expected': 'NO_TRADE'  # Agent should skip when data is missing\n            })\n        return cases\n\n    def generate_all(self):\n        \"\"\"Generate complete test suite.\"\"\"\n        return {\n            'normal': self.generate_normal_cases(),\n            'high_volatility': self.generate_high_volatility_cases(),\n            'expiry_week': self.generate_expiry_week_cases(),\n            'edge_cases': self.generate_edge_cases()\n        }\npython\nclass AgentEvaluator:\n    def __init__(self, agent, test_suite):\n        self.agent = agent\n        self.test_suite = test_suite\n        self.results = []\n\n    def run_evaluation(self):\n        \"\"\"Run all test cases and collect results.\"\"\"\n        for category, cases in self.test_suite.items():\n            for i, test in enumerate(cases):\n                try:\n                    # Run agent\n                    start_time = time.time()\n                    output = self.agent.predict(test['input'])\n                    latency = time.time() - start_time\n\n                    # Record result\n                    result = {\n                        'category': category,\n                        'test_id': i,\n                        'input': test['input'],\n                        'expected': test['expected'],\n                        'actual': output['signal'],\n                        'confidence': output.get('confidence', 0),\n                        'latency': latency,\n                        'correct': output['signal'] == test['expected']\n                    }\n\n                    self.results.append(result)\n\n                except Exception as e:\n                    # Agent crashed on this test case\n                    self.results.append({\n                        'category': category,\n                        'test_id': i,\n                        'input': test['input'],\n                        'expected': test['expected'],\n                        'actual': 'ERROR',\n                        'confidence': 0,\n                        'latency': 0,\n                        'correct': False,\n                        'error': str(e)\n                    })\n\n    def calculate_metrics(self):\n        \"\"\"Calculate all metrics from results.\"\"\"\n        df = pd.DataFrame(self.results)\n\n        metrics = {\n            'accuracy': df['correct'].mean(),\n            'precision': self._precision(df),\n            'recall': self._recall(df),\n            'f1': self._f1(df),\n            'avg_latency': df['latency'].mean(),\n            'max_latency': df['latency'].max(),\n            'error_rate': (df['actual'] == 'ERROR').mean(),\n            'override_potential': (df['confidence'] < 0.7).mean()\n        }\n\n        return metrics\n\n    def _precision(self, df):\n        \"\"\"Precision: of all BUY signals, how many were correct?\"\"\"\n        buy_signals = df[df['actual'] == 'BUY']\n        if len(buy_signals) == 0:\n            return 0\n        return buy_signals['correct'].mean()\n\n    def _recall(self, df):\n        \"\"\"Recall: of all correct BUY opportunities, how many did we catch?\"\"\"\n        correct_buys = df[df['expected'] == 'BUY']\n        if len(correct_buys) == 0:\n            return 0\n        caught = correct_buys[correct_buys['actual'] == 'BUY']\n        return len(caught) / len(correct_buys)\n\n    def _f1(self, df):\n        \"\"\"F1 score: harmonic mean of precision and recall.\"\"\"\n        precision = self._precision(df)\n        recall = self._recall(df)\n        if precision + recall == 0:\n            return 0\n        return 2 * (precision * recall) / (precision + recall)\npython\nclass BaselineComparator:\n    def __init__(self, agent_metrics, baselines):\n        self.agent_metrics = agent_metrics\n        self.baselines = baselines\n\n    def compare(self):\n        \"\"\"Compare agent against all baselines.\"\"\"\n        report = []\n\n        for metric, value in self.agent_metrics.items():\n            row = {'metric': metric, 'agent': value}\n\n            for baseline_name, baseline_metrics in self.baselines.items():\n                baseline_value = baseline_metrics.get(metric, 0)\n                row[baseline_name] = baseline_value\n                row[f'{baseline_name}_diff'] = value - baseline_value\n\n            report.append(row)\n\n        return pd.DataFrame(report)\n\n    def generate_report(self):\n        \"\"\"Generate human-readable report.\"\"\"\n        df = self.compare()\n\n        report = []\n        report.append(\"# Agent Evaluation Report\")\n        report.append(f\"Generated: {datetime.now().strftime('%Y-%m-%d %H:%M')}\")\n        report.append(\"\")\n\n        for _, row in df.iterrows():\n            metric = row['metric']\n            agent_value = row['agent']\n\n            report.append(f\"## {metric.upper()}\")\n            report.append(f\"- Agent: {agent_value:.3f}\")\n\n            for col in df.columns:\n                if col.endswith('_diff') and col != 'metric':\n                    baseline_name = col.replace('_diff', '')\n                    diff = row[col]\n                    status = \"✅\" if diff > 0 else \"❌\"\n                    report.append(f\"- {baseline_name}: {row[baseline_name]:.3f} ({diff:+.3f}) {status}\")\n\n            report.append(\"\")\n\n        return \"\\n\".join(report)\n```\n\nI ran the harness on my Nifty signal agent (v2.3). Here's what happened:\n\n| Category | Accuracy | Precision | Recall | F1 | Latency |\n|---|---|---|---|---|---|\nNormal |\n64.0% | 68.2% | 63.5% | 65.8% | 0.17s |\nHigh volatility |\n58.0% | 62.1% | 57.8% | 59.9% | 0.19s |\nExpiry week |\n56.0% | 59.8% | 55.2% | 57.4% | 0.21s |\nEdge cases |\n72.0% | 75.3% | 70.1% | 72.6% | 0.15s |\nOverall |\n62.3% |\n67.1% |\n61.8% |\n64.4% |\n0.18s |\n\n| Baseline | Accuracy |\n|---|---|\nRandom |\n50.0% |\nPrevious version (v2.2) |\n61.5% |\nHuman (me alone) |\n67.0% |\nSensibull AI |\n70.0% |\n\n**1. Agent improved from v2.2 to v2.3**\n\n**2. Agent still worse than human**\n\n**3. Agent worse than commercial tool**\n\n**4. Agent beats random**\n\n**5. Edge cases are surprisingly good**\n\n**6. Expiry week is the hardest**\n\nI thought my agent was 62% accurate. The harness showed **61.5% on unseen data** (previous version).\n\nThat 0.5% gap is **overfitting**. The agent was memorizing patterns in training data.\n\n**Fix:** I added more regularization (max_depth=3 instead of 5). New accuracy: 62.3%.\n\nThis means my agent is **conservative**. It only signals when it's confident. It skips ambiguous setups.\n\n**Is this good?** Yes and no.\n\n**Fix:** Lower confidence threshold from 0.65 to 0.60. New recall: 64.2%. Precision drops to 65.8%. **Better balance.**\n\nI expected the agent to fail on missing data. Instead, it correctly returned \"NO_TRADE\" 72% of the time.\n\n**Why?** The training data had many missing values. The agent learned to skip when uncertain.\n\n**Lesson:** Diverse training data = robust agent.\n\nExpiry week accuracy: 56% vs normal: 64%. **8% gap.**\n\n**Why?** Expiry week has unique dynamics:\n\n**Fix:** Add expiry-specific features:\n\nThe harness showed I overrode **23% of signals**. I thought it was 15%.\n\n**Why the discrepancy?** I forgot to log some manual overrides. The harness captured everything.\n\n**Is 23% too high?** Yes. If the agent is well-tuned, I should override <15%.\n\n**Fix:** Investigate the 23% overrides. Find patterns. Retrain agent on those cases.\n\nBefore the harness, my workflow was:\n\nAfter the harness:\n\nThis is **evaluation-driven development**. You don't ship until the numbers prove it's better.\n\n**Response:** My trading agent started small. Now it manages real money. The harness costs ₹0 to run. It takes 30 minutes. The value is **preventing costly mistakes**.\n\n**Response:** Manual testing is biased. You test cases you expect to pass. The harness tests **everything**, including cases you never thought of.\n\n**Response:** Simple agents fail in unexpected ways. My agent is \"simple\" (XGBoost on tabular data), but it still had overfitting, precision/recall imbalance, and expiry week weaknesses.\n\n**Response:** Yes, it adds 30 minutes per iteration. But it prevents **hours of debugging in production**. The math: 10 iterations × 30 minutes = 5 hours. 1 production bug = 20 hours. **Evaluation saves time.**\n\nAfter building and using mine for 3 months, here's what I think makes a good harness:\n\nDon't manually create test cases. Generate them from real data.\n\n**Why:** Manual test cases are biased. Generated test cases cover edge cases you never thought of.\n\nDon't just track accuracy. Track precision, recall, latency, cost, hallucination rate.\n\n**Why:** A model can have high accuracy but high cost. Or high precision but low recall. You need the full picture.\n\nAlways compare against something: random, previous version, human, commercial tool.\n\n**Why:** \"85% accuracy\" means nothing without context. 85% vs 50% (random) = great. 85% vs 90% (commercial) = needs work.\n\nRun the harness on every code change.\n\n**Why:** One wrong line of code can drop accuracy from 62% to 55%. You won't notice until you test.\n\nIntegrate the harness into your CI/CD pipeline.\n\n**Why:** If a PR breaks the agent, CI catches it before merge.\n\nTrack not just accuracy, but cost per prediction.\n\n**Why:** An agent that's 95% accurate but costs ₹10 per query is worse than an agent that's 85% accurate and costs ₹0.01 per query.\n\nThe harness should tell you **why** the agent failed, not just that it failed.\n\n**Why:** \"Accuracy dropped 3%\" is useless. \"Accuracy dropped 3% because expiry week features are missing\" is actionable.\n\nTwo weeks ago, I updated my agent's feature engineering pipeline. I added 3 new features.\n\n**Before update:**\n\n**After update:**\n\n**Next day:**\n\n**What happened?** The new features introduced multicollinearity. The model was overfitting to noise.\n\n**How the harness helped:**\n\n**Without harness:** I would have traded for 1-2 weeks with a broken agent. Potential loss: ₹10,000-20,000.\n\n**With harness:** Caught in 30 minutes. Rollback in 5 minutes. Loss: ₹0.\n\nYou don't need my full harness. Start small.\n\n```\n# 1. Collect 100 historical examples\n# 2. For each example, run agent\n# 3. Compare output to expected\n# 4. Calculate accuracy\n\ntest_cases = load_test_cases('test_suite.json')\nresults = []\n\nfor case in test_cases:\n    output = agent.run(case['input'])\n    results.append({\n        'correct': output == case['expected']\n    })\n\naccuracy = sum(r['correct'] for r in results) / len(results)\nprint(f\"Accuracy: {accuracy:.1%}\")\n# Add precision, recall, latency, cost\n\nmetrics = {\n    'accuracy': accuracy,\n    'precision': calculate_precision(results),\n    'recall': calculate_recall(results),\n    'avg_latency': avg_latency,\n    'cost_per_prediction': total_cost / len(results)\n}\n# Compare to random, previous version, human\n\nbaselines = {\n    'random': 0.50,\n    'previous_version': 0.615,\n    'human': 0.67\n}\n\nfor baseline_name, baseline_acc in baselines.items():\n    print(f\"vs {baseline_name}: {accuracy - baseline_acc:+.1%}\")\n# Run on every code change\n# Generate report\n# Alert if regression\n\nif accuracy < previous_accuracy - 0.05:\n    send_alert(\"Accuracy regression detected!\")\n```\n\nWe're in the **agent gold rush**. Everyone is building agents. Few are evaluating them properly.\n\nThis is dangerous.\n\n**Un-evaluated agents will:**\n\n**Evaluated agents will:**\n\nBased on my experience, here's what I'd want from an agent evaluation platform:\n\nGive me real data. I want the platform to generate test cases automatically.\n\n**Features:**\n\nShow me more than accuracy.\n\n**Metrics:**\n\nLet me compare against:\n\nTell me when my agent breaks.\n\n**Features:**\n\nTell me **why** the agent failed.\n\n**Features:**\n\nI built an Agent Evaluation Harness because I couldn't answer a simple question: **\"How good is my agent, really?\"**\n\nMost people skip this step. They build, ship, hope.\n\nI don't hope. I measure.\n\n**The harness taught me:**\n\n**The harness saved me from:**\n\n**If you're building agents, build an evaluation harness first.**\n\nNot after. **Before.**\n\nBecause the question isn't \"Is my agent working?\"\n\nThe question is: **\"How do I know my agent is working?\"**\n\n**AI proposes. You dispose. Measure before you trust.**\n\n**P.S.** My evaluation harness is open-source. It's built in Python, runs locally, costs ₹0. If you want to use it or contribute, DM me.\n\n**Tags:** aiagents, evaluation, localai, testing, python, trading, opensource, 2026\n\n**Meta:** Building an Agent Evaluation Harness for local AI agents. Architecture, code, and results from testing a Nifty trading agent. Key findings: overfitting detection, precision/recall tradeoffs, expiry week weaknesses. 5 surprises from 300 test cases. Why evaluation-driven development matters for AI agents.", "url": "https://wpnews.pro/news/i-built-an-agent-evaluation-harness-for-local-ai-what-most-people-get-wrong", "canonical_source": "https://dev.to/shaktitiwari/i-built-an-agent-evaluation-harness-for-local-ai-what-most-people-get-wrong-1k33", "published_at": "2026-08-05 18:46:21+00:00", "updated_at": "2026-08-05 18:57:26.952145+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "mlops", "developer-tools"], "entities": ["XGBoost", "Telegram", "Nifty"], "alternates": {"html": "https://wpnews.pro/news/i-built-an-agent-evaluation-harness-for-local-ai-what-most-people-get-wrong", "markdown": "https://wpnews.pro/news/i-built-an-agent-evaluation-harness-for-local-ai-what-most-people-get-wrong.md", "text": "https://wpnews.pro/news/i-built-an-agent-evaluation-harness-for-local-ai-what-most-people-get-wrong.txt", "jsonld": "https://wpnews.pro/news/i-built-an-agent-evaluation-harness-for-local-ai-what-most-people-get-wrong.jsonld"}}