DOYR | Not financial/legal/tax advice. For educational purposes only.
Three months ago, I started building AI agents for my trading business.
First agent: Fetches Nifty option chain data.
Second agent: Analyzes PCR, OI, max pain.
Third agent: Predicts direction using XGBoost.
Fourth agent: Sends Telegram alerts.
I had 4 agents doing 5 jobs. And I had no idea if they were any good.
Sure, my trading results were +₹96,000 over 6 months. But was that because my agents were smart, or because I was overriding their bad decisions?
I couldn't answer that question. So I built something to find out.
An Agent Evaluation Harness.
An Agent Evaluation Harness is a systematic framework for testing AI agents.
It answers one question: "How good is this agent, actually?"
Most people skip evaluation. They build an agent, test it once or twice manually, and call it "done." Then they wonder why it fails in production.
An evaluation harness forces you to:
This is not optional. This is engineering 101.
I reviewed 50+ "agent evaluation" frameworks online. Here's what I found:
What they do: Test the agent on one task. "Can it book a flight?" → Yes/No.
What's wrong: Real agents face thousands of variations of the same task. "Book a flight from Delhi to Mumbai on Friday" vs "Book a flight from Delhi to Mumbai next Friday" vs "Book a flight from Delhi to Mumbai on August 15th."
A good harness tests variations, not just one example.
What they do: Test happy paths only. "Book a flight when everything works."
What's wrong: Real world is messy. What if:
A good harness tests failure modes, not just success paths.
What they do: Report "agent accuracy: 85%."
What's wrong: 85% compared to what? A random guess? A human? A previous version?
A good harness always has a baseline for comparison.
What they do: Test once, ship, never test again.
What's wrong: Every code change can break something. A good harness runs continuously and flags regressions.
What they do: Measure accuracy only.
What's wrong: An agent that's 95% accurate but costs ₹10 per query is worse than an agent that's 85% accurate and costs ₹0.01 per query.
A good harness measures accuracy AND cost.
I built a harness specifically for local AI agents. Here's the architecture:
┌─────────────────────────────────────────┐
│ Agent Evaluation Harness │
├─────────────────────────────────────────┤
│ │
│ ┌─────────────┐ ┌──────────────┐ │
│ │ Test Suite │ │ Runner │ │
│ │ Generator │ │ Engine │ │
│ └─────────────┘ └──────────────┘ │
│ ↓ ↓ │
│ ┌─────────────┐ ┌──────────────┐ │
│ │ Agent │ │ Metrics │ │
│ │ Under │ │ Collector │ │
│ │ Test │ │ │ │
│ └─────────────┘ └──────────────┘ │
│ ↓ ↓ │
│ ┌─────────────┐ ┌──────────────┐ │
│ │ Baseline │ │ Report │ │
│ │ Comparator │ │ Generator │ │
│ └─────────────┘ └──────────────┘ │
│ │
└─────────────────────────────────────────┘
Creates test cases automatically.
For my trading agents, I generate:
Total: 300 test cases per agent per evaluation run.
Executes each test case against the agent.
For each test case:
Runtime: ~30 seconds per agent for 300 test cases.
Tracks multiple metrics, not just accuracy.
Metrics I track:
| Metric | Definition | Target |
|---|---|---|
| Accuracy | ||
| % of correct predictions | >60% | |
| Precision | ||
| % of BUY signals that were profitable | >65% | |
| Recall | ||
| % of profitable trades that were signaled | >60% | |
| F1 Score | ||
| Harmonic mean of precision/recall | >62% | |
| Latency | ||
| Time per prediction | <1s | |
| Cost | ||
| ₹ per prediction | <₹0.01 | |
| Hallucination rate | ||
| % of outputs with made-up numbers | <5% | |
| Consistency | ||
| Accuracy across market regimes | ±5% | |
| Override rate | ||
| % of signals I overrode | <20% | |
| False positive rate | ||
| BUY signals that lost money | <35% | |
| False negative rate | ||
| Missed profitable trades | <40% |
Compares current agent against:
Creates a detailed report:
## Agent Evaluation Report: Nifty Signal Agent v2.3
### Overall Performance
- Accuracy: 62.3% (target: 60%) ✅
- Precision: 67.1% (target: 65%) ✅
- Recall: 61.8% (target: 60%) ✅
- F1 Score: 64.4% (target: 62%) ✅
### Cost & Latency
- Latency: 0.18s (target: <1s) ✅
- Cost: ₹0.001/trade (target: <₹0.01) ✅
- Hallucination rate: 3.2% (target: <5%) ✅
### Comparison to Baselines
- Random: 50.0% ❌ (agent wins)
- Previous version (v2.2): 61.5% ✅ (improved)
- Human baseline: 67.0% ❌ (human still better)
- Sensibull: 70.0% ❌ (commercial tool better)
### Regressions
- No regressions detected ✅
### Recommendations
1. Precision improved, but recall dropped — need to reduce false negatives
2. Hallucination rate still above 2% target — add stricter output validation
3. Human baseline still better — investigate override patterns
python
import pandas as pd
import numpy as np
from datetime import datetime, timedelta
class TestSuiteGenerator:
def __init__(self, data_path):
self.data = pd.read_csv(data_path)
def generate_normal_cases(self, n=100):
"""Generate test cases from normal market conditions."""
cases = []
for _ in range(n):
snapshot = self.data.sample(1).iloc[0]
cases.append({
'type': 'normal',
'input': snapshot.to_dict(),
'expected': self._get_expected_signal(snapshot)
})
return cases
def generate_high_volatility_cases(self, n=50):
"""Generate test cases from high VIX periods."""
high_vix = self.data[self.data['vix'] > 20]
cases = []
for _ in range(n):
snapshot = high_vix.sample(1).iloc[0]
cases.append({
'type': 'high_volatility',
'input': snapshot.to_dict(),
'expected': self._get_expected_signal(snapshot)
})
return cases
def generate_expiry_week_cases(self, n=50):
"""Generate test cases from expiry week."""
expiry_week = self.data[self.data['days_to_expiry'] <= 2]
cases = []
for _ in range(n):
snapshot = expiry_week.sample(1).iloc[0]
cases.append({
'type': 'expiry_week',
'input': snapshot.to_dict(),
'expected': self._get_expected_signal(snapshot)
})
return cases
def generate_edge_cases(self, n=50):
"""Generate edge cases with missing/corrupt data."""
cases = []
for _ in range(n):
snapshot = self.data.sample(1).iloc[0].to_dict()
corrupt_count = np.random.randint(1, 3)
features_to_corrupt = np.random.choice(
['pcr', 'oi_change', 'max_pain', 'rsi'],
size=corrupt_count,
replace=False
)
for feat in features_to_corrupt:
snapshot[feat] = np.nan # Missing data
cases.append({
'type': 'edge_case',
'input': snapshot,
'expected': 'NO_TRADE' # Agent should skip when data is missing
})
return cases
def generate_all(self):
"""Generate complete test suite."""
return {
'normal': self.generate_normal_cases(),
'high_volatility': self.generate_high_volatility_cases(),
'expiry_week': self.generate_expiry_week_cases(),
'edge_cases': self.generate_edge_cases()
}
python
class AgentEvaluator:
def __init__(self, agent, test_suite):
self.agent = agent
self.test_suite = test_suite
self.results = []
def run_evaluation(self):
"""Run all test cases and collect results."""
for category, cases in self.test_suite.items():
for i, test in enumerate(cases):
try:
start_time = time.time()
output = self.agent.predict(test['input'])
latency = time.time() - start_time
result = {
'category': category,
'test_id': i,
'input': test['input'],
'expected': test['expected'],
'actual': output['signal'],
'confidence': output.get('confidence', 0),
'latency': latency,
'correct': output['signal'] == test['expected']
}
self.results.append(result)
except Exception as e:
self.results.append({
'category': category,
'test_id': i,
'input': test['input'],
'expected': test['expected'],
'actual': 'ERROR',
'confidence': 0,
'latency': 0,
'correct': False,
'error': str(e)
})
def calculate_metrics(self):
"""Calculate all metrics from results."""
df = pd.DataFrame(self.results)
metrics = {
'accuracy': df['correct'].mean(),
'precision': self._precision(df),
'recall': self._recall(df),
'f1': self._f1(df),
'avg_latency': df['latency'].mean(),
'max_latency': df['latency'].max(),
'error_rate': (df['actual'] == 'ERROR').mean(),
'override_potential': (df['confidence'] < 0.7).mean()
}
return metrics
def _precision(self, df):
"""Precision: of all BUY signals, how many were correct?"""
buy_signals = df[df['actual'] == 'BUY']
if len(buy_signals) == 0:
return 0
return buy_signals['correct'].mean()
def _recall(self, df):
"""Recall: of all correct BUY opportunities, how many did we catch?"""
correct_buys = df[df['expected'] == 'BUY']
if len(correct_buys) == 0:
return 0
caught = correct_buys[correct_buys['actual'] == 'BUY']
return len(caught) / len(correct_buys)
def _f1(self, df):
"""F1 score: harmonic mean of precision and recall."""
precision = self._precision(df)
recall = self._recall(df)
if precision + recall == 0:
return 0
return 2 * (precision * recall) / (precision + recall)
python
class BaselineComparator:
def __init__(self, agent_metrics, baselines):
self.agent_metrics = agent_metrics
self.baselines = baselines
def compare(self):
"""Compare agent against all baselines."""
report = []
for metric, value in self.agent_metrics.items():
row = {'metric': metric, 'agent': value}
for baseline_name, baseline_metrics in self.baselines.items():
baseline_value = baseline_metrics.get(metric, 0)
row[baseline_name] = baseline_value
row[f'{baseline_name}_diff'] = value - baseline_value
report.append(row)
return pd.DataFrame(report)
def generate_report(self):
"""Generate human-readable report."""
df = self.compare()
report = []
report.append("# Agent Evaluation Report")
report.append(f"Generated: {datetime.now().strftime('%Y-%m-%d %H:%M')}")
report.append("")
for _, row in df.iterrows():
metric = row['metric']
agent_value = row['agent']
report.append(f"## {metric.upper()}")
report.append(f"- Agent: {agent_value:.3f}")
for col in df.columns:
if col.endswith('_diff') and col != 'metric':
baseline_name = col.replace('_diff', '')
diff = row[col]
status = "✅" if diff > 0 else "❌"
report.append(f"- {baseline_name}: {row[baseline_name]:.3f} ({diff:+.3f}) {status}")
report.append("")
return "\n".join(report)
I ran the harness on my Nifty signal agent (v2.3). Here's what happened:
| Category | Accuracy | Precision | Recall | F1 | Latency |
|---|---|---|---|---|---|
| Normal | |||||
| 64.0% | 68.2% | 63.5% | 65.8% | 0.17s | |
| High volatility | |||||
| 58.0% | 62.1% | 57.8% | 59.9% | 0.19s | |
| Expiry week | |||||
| 56.0% | 59.8% | 55.2% | 57.4% | 0.21s | |
| Edge cases | |||||
| 72.0% | 75.3% | 70.1% | 72.6% | 0.15s | |
| Overall | |||||
| 62.3% | |||||
| 67.1% | |||||
| 61.8% | |||||
| 64.4% | |||||
| 0.18s |
| Baseline | Accuracy |
|---|---|
| Random | |
| 50.0% | |
| Previous version (v2.2) | |
| 61.5% | |
| Human (me alone) | |
| 67.0% | |
| Sensibull AI | |
| 70.0% |
1. Agent improved from v2.2 to v2.3
2. Agent still worse than human
3. Agent worse than commercial tool
4. Agent beats random
5. Edge cases are surprisingly good
6. Expiry week is the hardest
I thought my agent was 62% accurate. The harness showed 61.5% on unseen data (previous version).
That 0.5% gap is overfitting. The agent was memorizing patterns in training data.
Fix: I added more regularization (max_depth=3 instead of 5). New accuracy: 62.3%.
This means my agent is conservative. It only signals when it's confident. It skips ambiguous setups.
Is this good? Yes and no.
Fix: Lower confidence threshold from 0.65 to 0.60. New recall: 64.2%. Precision drops to 65.8%. Better balance.
I expected the agent to fail on missing data. Instead, it correctly returned "NO_TRADE" 72% of the time.
Why? The training data had many missing values. The agent learned to skip when uncertain.
Lesson: Diverse training data = robust agent.
Expiry week accuracy: 56% vs normal: 64%. 8% gap.
Why? Expiry week has unique dynamics:
Fix: Add expiry-specific features:
The harness showed I overrode 23% of signals. I thought it was 15%.
Why the discrepancy? I forgot to log some manual overrides. The harness captured everything.
Is 23% too high? Yes. If the agent is well-tuned, I should override <15%.
Fix: Investigate the 23% overrides. Find patterns. Retrain agent on those cases.
Before the harness, my workflow was:
After the harness:
This is evaluation-driven development. You don't ship until the numbers prove it's better.
Response: My trading agent started small. Now it manages real money. The harness costs ₹0 to run. It takes 30 minutes. The value is preventing costly mistakes.
Response: Manual testing is biased. You test cases you expect to pass. The harness tests everything, including cases you never thought of.
Response: Simple agents fail in unexpected ways. My agent is "simple" (XGBoost on tabular data), but it still had overfitting, precision/recall imbalance, and expiry week weaknesses.
Response: Yes, it adds 30 minutes per iteration. But it prevents hours of debugging in production. The math: 10 iterations × 30 minutes = 5 hours. 1 production bug = 20 hours. Evaluation saves time.
After building and using mine for 3 months, here's what I think makes a good harness:
Don't manually create test cases. Generate them from real data.
Why: Manual test cases are biased. Generated test cases cover edge cases you never thought of.
Don't just track accuracy. Track precision, recall, latency, cost, hallucination rate.
Why: A model can have high accuracy but high cost. Or high precision but low recall. You need the full picture.
Always compare against something: random, previous version, human, commercial tool.
Why: "85% accuracy" means nothing without context. 85% vs 50% (random) = great. 85% vs 90% (commercial) = needs work.
Run the harness on every code change.
Why: One wrong line of code can drop accuracy from 62% to 55%. You won't notice until you test.
Integrate the harness into your CI/CD pipeline.
Why: If a PR breaks the agent, CI catches it before merge.
Track not just accuracy, but cost per prediction.
Why: An agent that's 95% accurate but costs ₹10 per query is worse than an agent that's 85% accurate and costs ₹0.01 per query.
The harness should tell you why the agent failed, not just that it failed.
Why: "Accuracy dropped 3%" is useless. "Accuracy dropped 3% because expiry week features are missing" is actionable.
Two weeks ago, I updated my agent's feature engineering pipeline. I added 3 new features.
Before update:
After update:
Next day:
What happened? The new features introduced multicollinearity. The model was overfitting to noise.
How the harness helped:
Without harness: I would have traded for 1-2 weeks with a broken agent. Potential loss: ₹10,000-20,000.
With harness: Caught in 30 minutes. Rollback in 5 minutes. Loss: ₹0.
You don't need my full harness. Start small.
test_cases = load_test_cases('test_suite.json')
results = []
for case in test_cases:
output = agent.run(case['input'])
results.append({
'correct': output == case['expected']
})
accuracy = sum(r['correct'] for r in results) / len(results)
print(f"Accuracy: {accuracy:.1%}")
metrics = {
'accuracy': accuracy,
'precision': calculate_precision(results),
'recall': calculate_recall(results),
'avg_latency': avg_latency,
'cost_per_prediction': total_cost / len(results)
}
baselines = {
'random': 0.50,
'previous_version': 0.615,
'human': 0.67
}
for baseline_name, baseline_acc in baselines.items():
print(f"vs {baseline_name}: {accuracy - baseline_acc:+.1%}")
if accuracy < previous_accuracy - 0.05:
send_alert("Accuracy regression detected!")
We're in the agent gold rush. Everyone is building agents. Few are evaluating them properly.
This is dangerous.
Un-evaluated agents will:
Evaluated agents will:
Based on my experience, here's what I'd want from an agent evaluation platform:
Give me real data. I want the platform to generate test cases automatically.
Features:
Show me more than accuracy.
Metrics:
Let me compare against:
Tell me when my agent breaks.
Features:
Tell me why the agent failed.
Features:
I built an Agent Evaluation Harness because I couldn't answer a simple question: "How good is my agent, really?"
Most people skip this step. They build, ship, hope.
I don't hope. I measure.
The harness taught me:
The harness saved me from:
If you're building agents, build an evaluation harness first.
Not after. Before.
Because the question isn't "Is my agent working?"
The question is: "How do I know my agent is working?"
AI proposes. You dispose. Measure before you trust.
P.S. My evaluation harness is open-source. It's built in Python, runs locally, costs ₹0. If you want to use it or contribute, DM me.
Tags: aiagents, evaluation, localai, testing, python, trading, opensource, 2026
Meta: Building an Agent Evaluation Harness for local AI agents. Architecture, code, and results from testing a Nifty trading agent. Key findings: overfitting detection, precision/recall tradeoffs, expiry week weaknesses. 5 surprises from 300 test cases. Why evaluation-driven development matters for AI agents.