I Built an Agent Evaluation Harness for Local AI — What Most People Get Wrong

A developer built an agent evaluation harness for local AI agents used in trading, addressing common pitfalls in agent testing such as lack of variation, failure modes, baselines, continuous testing, and cost measurement. The harness generates 300 test cases per agent per run, tracking metrics like accuracy, precision, recall, F1, latency, cost, and hallucination rate, and achieved a trading profit of ₹96,000 over six months.

DOYR | Not financial/legal/tax advice. For educational purposes only. Three months ago, I started building AI agents for my trading business. First agent: Fetches Nifty option chain data. Second agent: Analyzes PCR, OI, max pain. Third agent: Predicts direction using XGBoost. Fourth agent: Sends Telegram alerts. I had 4 agents doing 5 jobs. And I had no idea if they were any good . Sure, my trading results were +₹96,000 over 6 months. But was that because my agents were smart, or because I was overriding their bad decisions? I couldn't answer that question. So I built something to find out. An Agent Evaluation Harness. An Agent Evaluation Harness is a systematic framework for testing AI agents. It answers one question: "How good is this agent, actually?" Most people skip evaluation. They build an agent, test it once or twice manually, and call it "done." Then they wonder why it fails in production. An evaluation harness forces you to: This is not optional. This is engineering 101 . I reviewed 50+ "agent evaluation" frameworks online. Here's what I found: What they do: Test the agent on one task. "Can it book a flight?" → Yes/No. What's wrong: Real agents face thousands of variations of the same task. "Book a flight from Delhi to Mumbai on Friday" vs "Book a flight from Delhi to Mumbai next Friday" vs "Book a flight from Delhi to Mumbai on August 15th." A good harness tests variations , not just one example. What they do: Test happy paths only. "Book a flight when everything works." What's wrong: Real world is messy. What if: A good harness tests failure modes , not just success paths. What they do: Report "agent accuracy: 85%." What's wrong: 85% compared to what? A random guess? A human? A previous version? A good harness always has a baseline for comparison. What they do: Test once, ship, never test again. What's wrong: Every code change can break something. A good harness runs continuously and flags regressions. What they do: Measure accuracy only. What's wrong: An agent that's 95% accurate but costs ₹10 per query is worse than an agent that's 85% accurate and costs ₹0.01 per query. A good harness measures accuracy AND cost . I built a harness specifically for local AI agents . Here's the architecture: ┌─────────────────────────────────────────┐ │ Agent Evaluation Harness │ ├─────────────────────────────────────────┤ │ │ │ ┌─────────────┐ ┌──────────────┐ │ │ │ Test Suite │ │ Runner │ │ │ │ Generator │ │ Engine │ │ │ └─────────────┘ └──────────────┘ │ │ ↓ ↓ │ │ ┌─────────────┐ ┌──────────────┐ │ │ │ Agent │ │ Metrics │ │ │ │ Under │ │ Collector │ │ │ │ Test │ │ │ │ │ └─────────────┘ └──────────────┘ │ │ ↓ ↓ │ │ ┌─────────────┐ ┌──────────────┐ │ │ │ Baseline │ │ Report │ │ │ │ Comparator │ │ Generator │ │ │ └─────────────┘ └──────────────┘ │ │ │ └─────────────────────────────────────────┘ Creates test cases automatically. For my trading agents, I generate: Total: 300 test cases per agent per evaluation run. Executes each test case against the agent. For each test case: Runtime: ~30 seconds per agent for 300 test cases. Tracks multiple metrics, not just accuracy. Metrics I track: | Metric | Definition | Target | |---|---|---| Accuracy | % of correct predictions | 60% | Precision | % of BUY signals that were profitable | 65% | Recall | % of profitable trades that were signaled | 60% | F1 Score | Harmonic mean of precision/recall | 62% | Latency | Time per prediction | <1s | Cost | ₹ per prediction | <₹0.01 | Hallucination rate | % of outputs with made-up numbers | <5% | Consistency | Accuracy across market regimes | ±5% | Override rate | % of signals I overrode | <20% | False positive rate | BUY signals that lost money | <35% | False negative rate | Missed profitable trades | <40% | Compares current agent against: Creates a detailed report: Agent Evaluation Report: Nifty Signal Agent v2.3 Overall Performance - Accuracy: 62.3% target: 60% ✅ - Precision: 67.1% target: 65% ✅ - Recall: 61.8% target: 60% ✅ - F1 Score: 64.4% target: 62% ✅ Cost & Latency - Latency: 0.18s target: <1s ✅ - Cost: ₹0.001/trade target: <₹0.01 ✅ - Hallucination rate: 3.2% target: <5% ✅ Comparison to Baselines - Random: 50.0% ❌ agent wins - Previous version v2.2 : 61.5% ✅ improved - Human baseline: 67.0% ❌ human still better - Sensibull: 70.0% ❌ commercial tool better Regressions - No regressions detected ✅ Recommendations 1. Precision improved, but recall dropped — need to reduce false negatives 2. Hallucination rate still above 2% target — add stricter output validation 3. Human baseline still better — investigate override patterns python import pandas as pd import numpy as np from datetime import datetime, timedelta class TestSuiteGenerator: def init self, data path : self.data = pd.read csv data path def generate normal cases self, n=100 : """Generate test cases from normal market conditions.""" cases = for in range n : snapshot = self.data.sample 1 .iloc 0 cases.append { 'type': 'normal', 'input': snapshot.to dict , 'expected': self. get expected signal snapshot } return cases def generate high volatility cases self, n=50 : """Generate test cases from high VIX periods.""" high vix = self.data self.data 'vix' 20 cases = for in range n : snapshot = high vix.sample 1 .iloc 0 cases.append { 'type': 'high volatility', 'input': snapshot.to dict , 'expected': self. get expected signal snapshot } return cases def generate expiry week cases self, n=50 : """Generate test cases from expiry week.""" expiry week = self.data self.data 'days to expiry' <= 2 cases = for in range n : snapshot = expiry week.sample 1 .iloc 0 cases.append { 'type': 'expiry week', 'input': snapshot.to dict , 'expected': self. get expected signal snapshot } return cases def generate edge cases self, n=50 : """Generate edge cases with missing/corrupt data.""" cases = for in range n : snapshot = self.data.sample 1 .iloc 0 .to dict Randomly corrupt 1-2 features corrupt count = np.random.randint 1, 3 features to corrupt = np.random.choice 'pcr', 'oi change', 'max pain', 'rsi' , size=corrupt count, replace=False for feat in features to corrupt: snapshot feat = np.nan Missing data cases.append { 'type': 'edge case', 'input': snapshot, 'expected': 'NO TRADE' Agent should skip when data is missing } return cases def generate all self : """Generate complete test suite.""" return { 'normal': self.generate normal cases , 'high volatility': self.generate high volatility cases , 'expiry week': self.generate expiry week cases , 'edge cases': self.generate edge cases } python class AgentEvaluator: def init self, agent, test suite : self.agent = agent self.test suite = test suite self.results = def run evaluation self : """Run all test cases and collect results.""" for category, cases in self.test suite.items : for i, test in enumerate cases : try: Run agent start time = time.time output = self.agent.predict test 'input' latency = time.time - start time Record result result = { 'category': category, 'test id': i, 'input': test 'input' , 'expected': test 'expected' , 'actual': output 'signal' , 'confidence': output.get 'confidence', 0 , 'latency': latency, 'correct': output 'signal' == test 'expected' } self.results.append result except Exception as e: Agent crashed on this test case self.results.append { 'category': category, 'test id': i, 'input': test 'input' , 'expected': test 'expected' , 'actual': 'ERROR', 'confidence': 0, 'latency': 0, 'correct': False, 'error': str e } def calculate metrics self : """Calculate all metrics from results.""" df = pd.DataFrame self.results metrics = { 'accuracy': df 'correct' .mean , 'precision': self. precision df , 'recall': self. recall df , 'f1': self. f1 df , 'avg latency': df 'latency' .mean , 'max latency': df 'latency' .max , 'error rate': df 'actual' == 'ERROR' .mean , 'override potential': df 'confidence' < 0.7 .mean } return metrics def precision self, df : """Precision: of all BUY signals, how many were correct?""" buy signals = df df 'actual' == 'BUY' if len buy signals == 0: return 0 return buy signals 'correct' .mean def recall self, df : """Recall: of all correct BUY opportunities, how many did we catch?""" correct buys = df df 'expected' == 'BUY' if len correct buys == 0: return 0 caught = correct buys correct buys 'actual' == 'BUY' return len caught / len correct buys def f1 self, df : """F1 score: harmonic mean of precision and recall.""" precision = self. precision df recall = self. recall df if precision + recall == 0: return 0 return 2 precision recall / precision + recall python class BaselineComparator: def init self, agent metrics, baselines : self.agent metrics = agent metrics self.baselines = baselines def compare self : """Compare agent against all baselines.""" report = for metric, value in self.agent metrics.items : row = {'metric': metric, 'agent': value} for baseline name, baseline metrics in self.baselines.items : baseline value = baseline metrics.get metric, 0 row baseline name = baseline value row f'{baseline name} diff' = value - baseline value report.append row return pd.DataFrame report def generate report self : """Generate human-readable report.""" df = self.compare report = report.append " Agent Evaluation Report" report.append f"Generated: {datetime.now .strftime '%Y-%m-%d %H:%M' }" report.append "" for , row in df.iterrows : metric = row 'metric' agent value = row 'agent' report.append f" {metric.upper }" report.append f"- Agent: {agent value:.3f}" for col in df.columns: if col.endswith ' diff' and col = 'metric': baseline name = col.replace ' diff', '' diff = row col status = "✅" if diff 0 else "❌" report.append f"- {baseline name}: {row baseline name :.3f} {diff:+.3f} {status}" report.append "" return "\n".join report I ran the harness on my Nifty signal agent v2.3 . Here's what happened: | Category | Accuracy | Precision | Recall | F1 | Latency | |---|---|---|---|---|---| Normal | 64.0% | 68.2% | 63.5% | 65.8% | 0.17s | High volatility | 58.0% | 62.1% | 57.8% | 59.9% | 0.19s | Expiry week | 56.0% | 59.8% | 55.2% | 57.4% | 0.21s | Edge cases | 72.0% | 75.3% | 70.1% | 72.6% | 0.15s | Overall | 62.3% | 67.1% | 61.8% | 64.4% | 0.18s | | Baseline | Accuracy | |---|---| Random | 50.0% | Previous version v2.2 | 61.5% | Human me alone | 67.0% | Sensibull AI | 70.0% | 1. Agent improved from v2.2 to v2.3 2. Agent still worse than human 3. Agent worse than commercial tool 4. Agent beats random 5. Edge cases are surprisingly good 6. Expiry week is the hardest I thought my agent was 62% accurate. The harness showed 61.5% on unseen data previous version . That 0.5% gap is overfitting . The agent was memorizing patterns in training data. Fix: I added more regularization max depth=3 instead of 5 . New accuracy: 62.3%. This means my agent is conservative . It only signals when it's confident. It skips ambiguous setups. Is this good? Yes and no. Fix: Lower confidence threshold from 0.65 to 0.60. New recall: 64.2%. Precision drops to 65.8%. Better balance. I expected the agent to fail on missing data. Instead, it correctly returned "NO TRADE" 72% of the time. Why? The training data had many missing values. The agent learned to skip when uncertain. Lesson: Diverse training data = robust agent. Expiry week accuracy: 56% vs normal: 64%. 8% gap. Why? Expiry week has unique dynamics: Fix: Add expiry-specific features: The harness showed I overrode 23% of signals . I thought it was 15%. Why the discrepancy? I forgot to log some manual overrides. The harness captured everything. Is 23% too high? Yes. If the agent is well-tuned, I should override <15%. Fix: Investigate the 23% overrides. Find patterns. Retrain agent on those cases. Before the harness, my workflow was: After the harness: This is evaluation-driven development . You don't ship until the numbers prove it's better. Response: My trading agent started small. Now it manages real money. The harness costs ₹0 to run. It takes 30 minutes. The value is preventing costly mistakes . Response: Manual testing is biased. You test cases you expect to pass. The harness tests everything , including cases you never thought of. Response: Simple agents fail in unexpected ways. My agent is "simple" XGBoost on tabular data , but it still had overfitting, precision/recall imbalance, and expiry week weaknesses. Response: Yes, it adds 30 minutes per iteration. But it prevents hours of debugging in production . The math: 10 iterations × 30 minutes = 5 hours. 1 production bug = 20 hours. Evaluation saves time. After building and using mine for 3 months, here's what I think makes a good harness: Don't manually create test cases. Generate them from real data. Why: Manual test cases are biased. Generated test cases cover edge cases you never thought of. Don't just track accuracy. Track precision, recall, latency, cost, hallucination rate. Why: A model can have high accuracy but high cost. Or high precision but low recall. You need the full picture. Always compare against something: random, previous version, human, commercial tool. Why: "85% accuracy" means nothing without context. 85% vs 50% random = great. 85% vs 90% commercial = needs work. Run the harness on every code change. Why: One wrong line of code can drop accuracy from 62% to 55%. You won't notice until you test. Integrate the harness into your CI/CD pipeline. Why: If a PR breaks the agent, CI catches it before merge. Track not just accuracy, but cost per prediction. Why: An agent that's 95% accurate but costs ₹10 per query is worse than an agent that's 85% accurate and costs ₹0.01 per query. The harness should tell you why the agent failed, not just that it failed. Why: "Accuracy dropped 3%" is useless. "Accuracy dropped 3% because expiry week features are missing" is actionable. Two weeks ago, I updated my agent's feature engineering pipeline. I added 3 new features. Before update: After update: Next day: What happened? The new features introduced multicollinearity. The model was overfitting to noise. How the harness helped: Without harness: I would have traded for 1-2 weeks with a broken agent. Potential loss: ₹10,000-20,000. With harness: Caught in 30 minutes. Rollback in 5 minutes. Loss: ₹0. You don't need my full harness. Start small. 1. Collect 100 historical examples 2. For each example, run agent 3. Compare output to expected 4. Calculate accuracy test cases = load test cases 'test suite.json' results = for case in test cases: output = agent.run case 'input' results.append { 'correct': output == case 'expected' } accuracy = sum r 'correct' for r in results / len results print f"Accuracy: {accuracy:.1%}" Add precision, recall, latency, cost metrics = { 'accuracy': accuracy, 'precision': calculate precision results , 'recall': calculate recall results , 'avg latency': avg latency, 'cost per prediction': total cost / len results } Compare to random, previous version, human baselines = { 'random': 0.50, 'previous version': 0.615, 'human': 0.67 } for baseline name, baseline acc in baselines.items : print f"vs {baseline name}: {accuracy - baseline acc:+.1%}" Run on every code change Generate report Alert if regression if accuracy < previous accuracy - 0.05: send alert "Accuracy regression detected " We're in the agent gold rush . Everyone is building agents. Few are evaluating them properly. This is dangerous. Un-evaluated agents will: Evaluated agents will: Based on my experience, here's what I'd want from an agent evaluation platform: Give me real data. I want the platform to generate test cases automatically. Features: Show me more than accuracy. Metrics: Let me compare against: Tell me when my agent breaks. Features: Tell me why the agent failed. Features: I built an Agent Evaluation Harness because I couldn't answer a simple question: "How good is my agent, really?" Most people skip this step. They build, ship, hope. I don't hope. I measure. The harness taught me: The harness saved me from: If you're building agents, build an evaluation harness first. Not after. Before. Because the question isn't "Is my agent working?" The question is: "How do I know my agent is working?" AI proposes. You dispose. Measure before you trust. P.S. My evaluation harness is open-source. It's built in Python, runs locally, costs ₹0. If you want to use it or contribute, DM me. Tags: aiagents, evaluation, localai, testing, python, trading, opensource, 2026 Meta: Building an Agent Evaluation Harness for local AI agents. Architecture, code, and results from testing a Nifty trading agent. Key findings: overfitting detection, precision/recall tradeoffs, expiry week weaknesses. 5 surprises from 300 test cases. Why evaluation-driven development matters for AI agents.