{"slug": "harness-tuner-eval-driven-loops-turn-agent-tuning-from-guesswork-into", "title": "Harness Tuner: Eval-Driven Loops Turn Agent Tuning from Guesswork into Measurement", "summary": "A developer built Harness Tuner, an agentic optimizer that automates agent tuning by planning experiments across tools, prompts, and model selection, measuring each on a training set and validating on a held-out set while never touching the test set until a candidate is explicitly promoted. The tool consumes Kiln task definitions, runs each experiment multiple times (default 3) to account for stochastic LLM outputs, penalizes high-variance configurations, and logs every experiment to a SQLite database for later inspection. Winning configurations are exported as JSON for loading into an agent runtime; the tool does not deploy agents itself.", "body_md": "Most agent tuning happens in a loop: tweak the prompt, run it manually, check the output, repeat. Harness Tuner replaces that manual cycle with an automated optimization loop that plans experiments, measures performance on a training set, validates on a held-out set, and never touches the test set until you explicitly promote a candidate.\n\nThe tool targets three tuning dimensions: agent tools (which functions the agent can call), prompts (system and user instructions), and model selection (which LLM backend to use). Instead of guessing which change will improve performance, you define evals and let the loop measure.\n\nHarness Tuner is an agentic optimizer. It reads task definitions from Kiln (a structured eval format), plans experiments, runs them, and tracks results. The loop follows a train/val/test discipline borrowed from ML pipelines:\n\nThe optimizer is itself an agent. It decides which dimension to tune next based on current performance metrics. If tool selection is the bottleneck, it experiments with different tool combinations. If the prompt is unclear, it generates variations and measures which one improves task completion.\n\nThe optimizer maintains a state machine:\n\nThe loop never reads the test set during optimization. This prevents the classic ML mistake of tuning to your test data.\n\nHarness Tuner consumes Kiln task definitions. A task is a JSON structure that includes:\n\nHere's a simplified example:\n\n```\n{\n  \"task_id\": \"summarize_article\",\n  \"input\": {\n    \"url\": \"https://example.com/article\"\n  },\n  \"expected_output\": {\n    \"summary\": \"A concise summary of the article\",\n    \"key_points\": [\"point1\", \"point2\"]\n  },\n  \"tools\": [\"fetch_url\", \"extract_text\"],\n  \"prompt\": \"You are a summarization agent. Fetch the article and extract key points.\"\n}\n```\n\nThe optimizer runs this task with different configurations and compares outputs to the expected result. Scoring can be exact match, semantic similarity, or a custom function.\n\nLLM outputs are stochastic. Harness Tuner runs each experiment multiple times (configurable, default is 3) and aggregates results. It tracks:\n\nHigh variance signals that a configuration is unstable. The optimizer penalizes configurations with high variance even if mean performance is good.\n\nThe optimizer can add, remove, or swap tools. If an agent has access to `fetch_url`, `search_web`, and `extract_text`, the optimizer might test:\n\n`search_web` to see if it's a distraction.`summarize_text` to improve output quality.`fetch_url` for a cached version to reduce latency.\nEach change is measured on the train set. If removing a tool improves performance (e.g., the agent was misusing it), the optimizer keeps the change.\n\nThe optimizer generates prompt variations using an LLM. It might:\n\nPrompt changes are risky because they can degrade performance in unexpected ways. The optimizer runs multiple variations in parallel and picks the one with the best train score, then validates it.\n\nThe optimizer can swap the underlying LLM. If you start with GPT-4, it might test:\n\nModel selection is measured on cost, latency, and accuracy. The optimizer balances these metrics based on weights you provide.\n\nAutomated tuning can overfit to the train set. Harness Tuner includes several guardrails:\n\nThe optimizer also logs every experiment. You can inspect the history to see which changes were tried and why they were accepted or rejected.\n\nHarness Tuner runs as a standalone process. You point it at a Kiln task directory, configure the optimization budget (how many experiments to run), and let it work.\n\n```\nharness-tuner optimize \\\n  --task-dir ./tasks \\\n  --budget 50 \\\n  --train-split 0.7 \\\n  --val-split 0.2 \\\n  --test-split 0.1\n```\n\nThe optimizer writes results to a SQLite database. You can query it to see experiment history, performance trends, and promoted configurations.\n\nFor production, you export the winning configuration as a JSON file and load it into your agent runtime. Harness Tuner doesn't deploy agents; it just finds the best configuration.\n\nThe optimizer logs every experiment to a structured format. Each log entry includes:\n\nYou can export logs to your observability stack (e.g., Datadog, Grafana) or query them locally. The optimizer also exposes a simple web UI that shows:\n\n| Failure Mode | Cause | Mitigation | \n|---|---|---|\n| Overfitting to train set | Optimizer runs too many experiments on the same data | Use a large train set and strict validation gate | \n| Unstable configurations | High variance in LLM outputs | Increase number of runs per experiment and penalize variance | \n| Expensive experiments | Testing large models or slow tools | Set cost and latency budgets; optimizer will skip expensive configs | \n| Prompt degradation | Optimizer generates a worse prompt that still scores well on train | Manual review of promoted prompts before production | \n| Tool misuse | Optimizer adds a tool that the agent uses incorrectly | Monitor tool call logs and add negative examples to the eval set | \n\n**Use it when:**\n\n**Avoid it when:**\n\nHarness Tuner is a closed-loop optimizer for agent configurations. It automates the tedious work of testing tool combinations, prompt variations, and model choices. The train/val/test split prevents overfitting, and the experiment tracking gives you a clear history of what worked and why.\n\nThe main risk is garbage-in-garbage-out. If your eval set is bad, the optimizer will tune to the wrong target. Spend time building good evals before you turn on automated tuning.\n\nFor teams running production agents, this is a useful primitive. It turns agent tuning from a manual art into a measured process. Just don't expect it to fix a fundamentally broken agent design.", "url": "https://wpnews.pro/news/harness-tuner-eval-driven-loops-turn-agent-tuning-from-guesswork-into", "canonical_source": "https://dev.to/mech_app_ai/harness-tuner-eval-driven-loops-turn-agent-tuning-from-guesswork-into-measurement-8b0", "published_at": "2026-11-14 00:08:36+00:00", "updated_at": "2026-11-14 00:22:05.376316+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "mlops", "large-language-models", "developer-tools"], "entities": ["Harness Tuner", "Kiln", "GPT-4", "SQLite"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/harness-tuner-eval-driven-loops-turn-agent-tuning-from-guesswork-into", "markdown": "https://wpnews.pro/news/harness-tuner-eval-driven-loops-turn-agent-tuning-from-guesswork-into.md", "text": "https://wpnews.pro/news/harness-tuner-eval-driven-loops-turn-agent-tuning-from-guesswork-into.txt", "jsonld": "https://wpnews.pro/news/harness-tuner-eval-driven-loops-turn-agent-tuning-from-guesswork-into.jsonld"}}