{"slug": "google-open-sources-harness2-self-improving-ai-agents-without-retraining", "title": "Google Open-Sources Harness²: Self-Improving AI Agents Without Retraining", "summary": "Google Research has open-sourced Harness², an Apache 2.0-licensed framework that lets AI agents improve their own prompts, tools, memory, skills, scripts, subagents and plugins without retraining or changing model weights. Harness² runs two recursive loops: a task-level propose-probe-compose loop that learns from behavioral contrasts between executions without ground-truth feedback, and a harnessing-level loop in which an experience reflector stores adopted harnesses and lessons in a persistent improvement harness for the next task. The release ships with a research paper, \"Harness²: Recursive Agent Harnessing for an Open World,\" and a project site.", "body_md": "# Google Open-Sources Harness²: Self-Improving AI Agents Without Retraining\n\nGoogle's Harness² lets AI agents improve their prompts, tools, memory, and scripts through two recursive loops without touching model weights. Learn more, including how to run the free demo yourself.\n\nGoogle Research has open-sourced **[Harness²](https://github.com/google-research/harness2)** (or **Harness2**), a framework that lets AI agents improve themselves without retraining the underlying model. Instead of touching model weights, Harness² rewrites the harness around the model: its prompts, tools, memory, skills, and scripts. Then it carries what it learned into the next task.\n\nThe code ships under the **[Apache 2.0 license](https://github.com/google-research/harness2/blob/main/LICENSE)**, alongside a research paper, [Harness²: Recursive Agent Harnessing for an Open World](https://arxiv.org/abs/2610.14224), and a [project site](https://harness2.github.io). The release is already being [discussed on X](https://x.com/kimmonismus/status/2112121074775441806).\n\nThis article explains how Harness² works and what its benchmark results show, then walks you through Google's free demo and a real benchmark run on Vertex AI. If you're looking for an open-source agent framework for self-improving AI agents, start here.\n\n## What Is Harness²?\n\nA **harness** is everything around a model that lets it act as an agent: prompts, tools, memory, skills, and orchestration. **Harnessing** is the work of optimizing it. Today, most teams do it by hand: an engineer watches the agent fail, tweaks the prompt, adds a tool, and tries again. That works for tasks you planned for, but breaks down in what the paper calls an **open world**, where agents meet tasks nobody anticipated.\n\nHarness² automates that tuning \"while keeping model weights fixed\" (README). There's no fine-tuning and no new training data; only the model's surroundings change. It can edit [eight components](https://github.com/google-research/harness2#harness-components):\n\n| Component | What it does in the agent | \n|---|---|\n| System prompt | The standing instructions the model reads first | \n| Guardrails | Rules about what the agent must or must not do | \n| Memory | Long-term notes carried across steps | \n| Skills | Reusable instructions for specific kinds of work | \n| Tool guidance | How and when to use each tool | \n| Scripts | Helper code the agent can run | \n| Subagents | Delegated helpers for parts of a task | \n| Plugins | Middleware that shapes behavior inside the tool loop | \n\nFrom the README, the first four build the model's context; tool guidance and scripts define its tool interface, subagents handle delegation, and plugins control middleware. Together they cover nearly everything about how an agent behaves except the model itself.\n\n## How Harness² Works: Two Levels of Recursion\n\nThe superscript **\"2\"** in the name refers to two loops. One improves the harness for the current task; the other improves how harnesses get improved across tasks.\n\nThe top loop runs on every task. The highlighted improvement harness persists between tasks and shapes the next round of edits.\n\n#### Loop 1: Task-Level Recursion (Propose—Probe—Compose)\n\n1. **Propose** : A harness editor suggests invariant (general) and domain (field-specific) edits\n2. **Probe** : The agent reruns the same task with each edited harness\n3. **Compose** : A composer compares the runs and merges the edits that helped\n\nThe key detail is that Harness² learns \"from behavioral contrasts between executions, without ground-truth feedback.\" It never needs the correct answer, only the difference between two runs of the same task. That matters because in the open world, nobody hands an agent an answer key.\n\n#### Loop 2: Harnessing-Level Recursion\n\nAfter each task, an **experience reflector** stores the adopted harness and \"evidence-supported lessons\" in a persistent **improvement harness**, which guides refinement on the next task. So the first loop makes the agent better at this task, and the second makes the framework better at improving agents on future tasks. You'll see both loops leave files on disk in the demo below.\n\n## Parallel vs. Sequential Recursion\n\nYou can spend more compute on Loop 1 with a budget, **k**, and a mode ([experiment guide](https://github.com/google-research/harness2/blob/main/docs/EXPERIMENTS.md#recursion-modes)):\n\n- **Parallel** proposes k invariant and k domain candidates, then runs k independent compositions over the full pool. It explores wider.\n- **Sequential** runs k Propose—Probe—Compose steps in a row, carrying the refined harness forward. It refines deeper.\n\nThe budget drives cost. Each task takes **4k executor rollouts and 3k improver sessions** ([run_sweep.py](https://github.com/google-research/harness2/blob/main/harness2/cli/run_sweep.py)), so `--k 3` means 12 rollouts and 9 improver sessions per task. LAB alone has 1,143 tasks ([experiment guide](https://github.com/google-research/harness2/blob/main/docs/EXPERIMENTS.md#task-selection-and-cost)), which is why every example here starts with one.\n\n## Benchmark Results\n\nGoogle tested GLM-5 and Gemini 3.5 Flash across four model—harness configurations on three benchmarks. A single step (**k = 1**) improved both over their base harnesses:\n\n| Benchmark | What it tests | Best reported gain | \n|---|---|---|\n| [WorkBuddy-Bench](https://github.com/google-research/harness2/blob/main/docs/BENCHMARKS.md#workbuddy-bench) | Office, web, and code tasks | Up to +11.5 points | \n| [LAB](https://github.com/google-research/harness2/blob/main/docs/BENCHMARKS.md#lab) | Legal work, from [Harvey's harvey-labs](https://github.com/harveyai/harvey-labs) | Up to +11.8 percentage points (partial pass rate) | \n| [JobBench](https://github.com/google-research/harness2/blob/main/docs/BENCHMARKS.md#jobbench) | Professional work tasks | Up to +4.5 points | \n\nThese are best-case \"up to\" figures, not averages, and LAB's is on its **Partial** score — the fraction of grading criteria passed. The README adds that a larger **k** \"brings further gains,\" and the [paper](https://arxiv.org/abs/2610.14224) has per-configuration numbers. These are Google's own results, so independent reproductions will matter.\n\n## Why This Matters\n\nThe usual way to make a model better at a job is to train it further, which needs training data, GPUs, and access to its weights. Most teams using a model through an API have none of these. Harness² keeps every improvement in plain files instead: a rewritten prompt, a fixed script, a new guardrail. That has three practical upsides.\n\n- You can read what changed. An improved harness is text and code, so you can review it and diff it against the original. Harness² even writes down why it adopted each change, as the demo shows.\n- It works with models you can't train. The executor and improver are just model IDs in a config file (setup guide). The release itself pairs GLM-5 and Gemini 3.5 Flash as agents with Vertex Gemini or Claude as the improver.\n- It doesn't need an answer key. Refinement comes from contrasts between runs, so it works on tasks nobody has labeled yet.\n\n## Running the Harness² Demo (No API Key Needed)\n\nGoogle's demo runs both recursion modes with \"*deterministic editors and real Python executions*\" and \"*requires no model credentials*\", so it's free.\n\n**Prerequisites**\n\n- **Linux** . On Windows, use WSL; on macOS, a Linux VM or container.\n- **Python 3.12** . 3.13 is also supported, per[pyproject.toml](https://github.com/google-research/harness2/blob/main/pyproject.toml) .\n- **[uv](https://docs.astral.sh/uv/getting-started/installation/)** , Astral's Python package manager.\n- **git** .\n\n#### Step 1: Installing uv\n\n```\ncurl -LsSf https://astral.sh/uv/install.sh | sh\n```\n\n`curl` downloads Astral's installer script. The flags make it follow redirects (**-L**), stay quiet (**-s**) while still showing errors (**-S**), and fail cleanly on a bad response (**-f**). The `| sh` part pipes the script into your shell, which installs uv into your user directory. Open a new terminal and run `uv --version` to confirm. `pip install uv` also works.\n\n#### Step 2: Cloning the Repository\n\n```\ngit clone https://github.com/google-research/harness2.git\ncd harness2\n```\n\n`git clone` downloads the full repository from Google Research's GitHub into a new `harness2` folder, and `cd` moves you into it. Run every later command from this repository root.\n\n#### Step 3: Creating and Activating a Virtual Environment\n\n```\nuv venv --python 3.12 .venv\nsource .venv/bin/activate\n```\n\n`uv venv` creates an isolated Python 3.12 environment in `.venv`, downloading Python if needed. `source` activates it, so `python` and installed commands now come from `.venv`. Your prompt usually shows `(.venv)` once it's active. Rerun the `source` line in any new terminal.\n\n#### Step 4: Installing Harness²\n\n```\nuv pip install -e '.[render]'\n```\n\n`-e` installs in editable mode, using the source files in place. The `[render]` extra adds `openpyxl` and `python-pptx` for spreadsheet and slide outputs ([pyproject.toml](https://github.com/google-research/harness2/blob/main/pyproject.toml)), and the quotes stop your shell from misreading the brackets. Installing registers the commands used below: `harness2`, `harness2-demo`, `harness2-setup`, `harness2-check`, `harness2-judge`, `harness2-report`, and `harness2-bridge`.\n\n#### Step 5: Running the Demo\n\n```\nharness2-demo --output runs/demo --k 2\n```\n\nThe demo runs the real recursion and library workflow, with scripted editors in place of a model. `--output` sets the results folder and `--k 2` sets the recursion budget. Without `--mode`, it runs both parallel and sequential modes. It finished in about two seconds for me:\n\n```\n [accounting] 1/2 ok           ledger-1\n  [accounting] 2/2 ok           ledger-2\nLocal demo: 2 tasks completed; 0 failed. /home/you/harness2/runs/demo/demo.json\n```\n\nThere's one domain, `accounting`, with two tasks. Both succeeded, and `demo.json` holds the summary.\n\n#### Step 6: Seeing What Harness² Fixed\n\nThe starting harness ships with a bug, and Harness² has to find it on its own. Here's the base harness's helper script:\n\n```\ncat runs/demo/base/scripts/summarize.py\npython\nimport csv\nimport sys\nfrom decimal import Decimal\n\nwith open(sys.argv[1]) as source:\n    amounts = [Decimal(row['amount']) for row in csv.DictReader(source)]\ntotal = sum((amount for amount in amounts if amount > 0), Decimal(0))\nprint(f'Total: {total:.2f}')\n```\n\nThe script reads the `amount` column of a CSV as exact `Decimal` values and adds them up. The `if amount > 0` filter is the bug: it silently drops negative entries.\n\nNow run the original and the adopted script on the same ledger:\n\n```\ncat runs/demo/sources/ledger-1/ledger.csv\npython runs/demo/base/scripts/summarize.py runs/demo/sources/ledger-1/ledger.csv\npython runs/demo/library/k2/seq/accounting/harnesses/ledger-1/harness/scripts/summarize.py runs/demo/sources/ledger-1/ledger.csv\namount\n100.25\n-20.10\n5.00\nTotal: 105.25\nTotal: 85.15\n```\n\nThe first command prints the ledger: three entries, one of them negative. The base script ignores it and reports 105.25; the adopted script reports the correct 85.15 (100.25 − 20.10 + 5.00). Harness² removed the filter without ever being told the right total.\n\n#### Step 7: Reading the Lesson It Wrote Down\n\n```\ncat runs/demo/library/k2/seq/accounting/harnesses/ledger-1/EXPERIENCE.md\n# ledger-1\n\n- adopted: `c1`  (run `demo-oc-seq2/ledger-1/c1`)\n- components: 1 scripts; 62 always-on chars\n\n## Why this was chosen\nThe signed total matches the source ledger.\n\n## What each edit did\ncomponent: scripts/summarize.py\ncandidates: inv1\nchange: dropped the positive-only filter from the amount sum\nobserved: the probe total included the credits the base run omitted\nverdict: helped\n\n## Transferable lesson\n- **cue**: When a ledger includes credits and debits\n- **lesson**: Include both signs when summing in scripts/summarize.py.\n- **evidence**: The scripts/summarize.py row -- removing the positive-only filter changed the probe total to include the credits omitted by the base trajectory.\n- **confidence**: high for this observed arithmetic contrast\n```\n\nThis is the experience reflector's note. `adopted: c1` names the winning harness; sequential compositions are named **c1** to **ck** and parallel ones `agg1` to `aggk` ([ledger.py](https://github.com/google-research/harness2/blob/main/harness2/core/ledger.py)). \"What each edit did\" records the probe evidence: the edit from invariant candidate `inv1` \"helped.\" The \"Transferable lesson\" pairs a cue with a lesson, evidence, and confidence — and it's what carries forward.\n\nNext, check how the lesson reached the second task:\n\n```\ncat runs/demo/demo.json\n{\n  \"domain\": \"accounting\",\n  \"n_tasks\": 2,\n  \"seed\": null,\n  \"done\": 2,\n  \"failed\": 0,\n  \"tasks\": [\n    {\n      \"task\": \"ledger-1\",\n      \"status\": \"ok\",\n      \"library_entries_before\": { \"agg\": 0, \"seq\": 0 }\n    },\n    {\n      \"task\": \"ledger-2\",\n      \"status\": \"ok\",\n      \"library_entries_before\": { \"agg\": 1, \"seq\": 1 }\n    }\n  ],\n  \"execution_count\": 28,\n  \"editor_calls\": 32\n}\n```\n\n`library_entries_before` counts stored lessons when each task started, with `agg` for parallel mode and `seq` for sequential ([library code](https://github.com/google-research/harness2/blob/main/harness2/core/library.py)). The first task started with none; the second started with one per mode — the lesson from `ledger-1`. That's the improvement harness growing between tasks.\n\n#### Step 8: Exploring the Rest of the Output\n\nThe `runs/demo` folder has four parts worth knowing:\n\n- `base/` holds the starting harness:`systemprompt.md` ,`guardrails.md` ,`LongTermMEMORY.md` ,`tool_descriptions/tool_guidance.md` , and`scripts/summarize.py` .\n- `runs/` holds every candidate harness per task: invariant candidates (**inv1** ,**inv2** ), domain candidates (**dom1** ,**dom2** ), and the compositions.\n- `executions/` holds the output of every probe run.\n- `library/` holds the improvement harness: adopted harnesses,`EXPERIENCE.md` notes, and an`INDEX.md` that future tasks browse. Each index entry repeats its cue — such as \"When a ledger includes credits and debits\" — so a later task can tell at a glance whether a lesson applies.\n\n## Running a Real Benchmark (LAB on Vertex AI)\n\nA real run uses models in every role: executor, improver, and judge. We'll use LAB, the legal benchmark the README uses as its example. **Model-backed runs cost money**, so every command here is limited to one task. I verified each command up to the first paid model call; your model output will vary.\n\n#### Extra Prerequisites\n\nYou'll also need **[Node.js](https://nodejs.org/en/download)** 20+ for the OpenCode executor, the **[Google Cloud CLI](https://cloud.google.com/sdk/docs/install)**, a Google Cloud project with Vertex AI enabled, and Pandoc and Poppler for LAB's documents. **[Docker](https://docs.docker.com/engine/install/)** is only needed for WorkBuddy-Bench.\n\n```\nsudo apt-get update\nsudo apt-get install -y git curl unzip ca-certificates pandoc poppler-utils\n```\n\n`apt-get update` refreshes the package list, and the second line installs the tools, with `-y` skipping the prompt. `pandoc` converts documents and `poppler-utils` reads PDFs; setup uses the rest to download code and data.\n\n#### Step 1: Creating Your .env File\n\nThe README says to copy `.env.example`, but that file isn't in the repository at the time of writing (it returns a 404). Create `.env` in the repo root yourself, using the settings from the [setup guide](https://github.com/google-research/harness2/blob/main/docs/SETUP.md#models-and-credentials) and [config.py](https://github.com/google-research/harness2/blob/main/harness2/config.py):\n\n```\n# Your Google Cloud project and Vertex AI region\nGOOGLE_CLOUD_PROJECT=your-gcp-project-id\nGOOGLE_CLOUD_LOCATION=global\n\n# Executor (runs the task) and improver (edits the harness)\nHARNESS2_SOLVER_MODEL=google-vertex/YOUR_GEMINI_MODEL_ID\nHARNESS2_IMPROVER_MODEL=google-vertex/YOUR_GEMINI_MODEL_ID\n\n# Judge (grades the output) - bare model ID, no prefix\nHARNESS2_JUDGE_MODEL=YOUR_GEMINI_MODEL_ID\n```\n\nHarness² loads these `KEY=VALUE` lines at startup.\n\n- `GOOGLE_CLOUD_PROJECT` is your project ID; leave`GOOGLE_CLOUD_LOCATION` as`global` .\n- `HARNESS2_SOLVER_MODEL` is the executor. It must use OpenCode's provider/model format, so Vertex Gemini IDs start with`google-vertex/` .\n- `HARNESS2_IMPROVER_MODEL` edits the harness and defaults to the executor model. Vertex Claude also works here.\n- `HARNESS2_JUDGE_MODEL` grades the results. It takes the bare model ID with no prefix, because Harness² adds the right prefix for each benchmark itself.\n\nReplace each `YOUR_GEMINI_MODEL_ID` with the exact model ID from your Vertex AI console. Values set in your shell override `.env`, which is handy for quick experiments.\n\n#### Step 2: Authenticating with Google Cloud\n\n```\ngcloud auth application-default login\n```\n\nThis signs you in through the browser and saves Application Default Credentials, which Harness² uses to call Vertex AI. On a headless server, add `--no-browser`.\n\n#### Step 3: Installing the LAB Benchmark\n\n```\nharness2-setup lab\n```\n\nSetup downloads pinned versions of LAB (from [Harvey's harvey-labs](https://github.com/harveyai/harvey-labs)) and the OpenCode executor, applies Google's patches, installs Bun, builds LAB's own Python environment, and fetches the tasks into `dependencies/`. Add `--dry-run` to preview the plan first. Keep your main `.venv` active; Harness² finds LAB's interpreter itself.\n\n#### Step 4: Checking Your Installation\n\n```\nharness2-check --bench lab\n```\n\nThis verifies dependencies, credentials, and settings without calling a model, and prints a `FAIL:` line for anything missing. For example, without a `.env` you'll see `FAIL: HARNESS2_SOLVER_MODEL is required for executor rollouts`. It can't confirm your account can use the models; that only shows on the first real call.\n\n#### Step 5: Previewing with a Dry Run\n\n```\nharness2 --bench lab --k 1 --mode parallel \\\n  --only-domains antitrust-competition --max-tasks 1 --dry-run\n```\n\n`--dry-run` prints the selected task and makes no model calls. `--bench lab` picks the benchmark, `--k 1 --mode parallel` sets one parallel step, `--only-domains` limits the run to one practice area, and `--max-tasks 1` caps it at one task per domain.\n\n#### Step 6: Running the Refinement\n\n```\nharness2 --bench lab --k 1 --mode parallel \\\n  --only-domains antitrust-competition --max-tasks 1\n```\n\nWithout `--dry-run`, this runs the full loop: 4 executor rollouts and 3 improver sessions at `--k 1`. It re-checks your configuration, including the judge, before spending anything. It then runs the base harness, proposes edits, probes them, composes a refined harness, and reflects on the result. If it stops partway, rerun the same command to resume.\n\n#### Step 7: Judging and Reporting\n\n```\nharness2-judge --bench lab --k 1 --mode parallel\nharness2-report --bench lab --k 1 --mode parallel\n```\n\nThe judge grades the refined harness's outputs against LAB's rubric, and the report summarizes **Partial** and **All-pass** scores. Use the same benchmark, `--k`, `--mode`, and task selection as the run so both commands find its results.\n\nTo compare against the unimproved harness:\n\n```\nharness2-judge --bench lab --k 1 --mode parallel --bases\n```\n\n`--bases` scores the original harness's runs into `base_judged.json`, next to `judged.json`.\n\nResults land in `runs/lab/`. Before changing models or task selection, point Harness² at a new folder with `export HARNESS2_RUNS_DIR=runs/my-new-experiment`. To run Steps 4 to 7 in one go:\n\n```\nbash scripts/run_lab.sh --only-domains antitrust-competition --max-tasks 1\n```\n\n`run_lab.sh` chains the check, refinement, judging, and reporting, passing your flags to the refinement step. Put `K=3 MODE=sequential` in front of it to change the budget or mode.\n\n## Wrapping Up\n\nHarness² hints at a future where agents improve by building up experience rather than by retraining: lesson libraries grown over weeks of real work, and harness changes that teams review like code. The open questions are how well lessons transfer across domains, how costs compare with fine-tuning, and how to keep self-editing agents safe. With the code open, the community can now find out.\n\nTo recap:\n\n- [Harness²](https://github.com/google-research/harness2) is Google Research's open-source framework for self-improving AI agents, and it keeps model weights fixed.\n- Task-level recursion refines each task's harness through Propose—Probe—Compose without ground-truth answers, and harnessing-level recursion carries lessons forward.\n- Parallel mode explores wider, sequential mode refines deeper, and k costs 4k rollouts and 3k improver sessions per task.\n- One step gained up to 11.5 points on WorkBuddy-Bench, 4.5 on JobBench, and 11.8 percentage points on LAB.\n- The free demo shows Harness² fixing a buggy script and recording the lesson; real runs need Vertex AI and a hand-made `.env` .\n- Lock down network access, watch costs, and treat it as research code.\n\nFor the full method, read the [paper on arXiv](https://arxiv.org/abs/2610.14224). Then clone the repo and run the demo yourself.\n\n## Sources\n\n \n\n \n\n[**\\[Shittu Olumide\\](https://www.linkedin.com/in/olumide-shittu/)**](https://www.linkedin.com/in/olumide-shittu) is a software engineer and technical writer passionate about leveraging cutting-edge technologies to craft compelling narratives, with a keen eye for detail and a knack for simplifying complex concepts. You can also find Shittu on [Twitter](https://twitter.com/Shittu_Olumide_).", "url": "https://wpnews.pro/news/google-open-sources-harness2-self-improving-ai-agents-without-retraining", "canonical_source": "https://www.kdnuggets.com/google-open-sources-harness-self-improving-ai-agents-without-retraining", "published_at": "2026-10-22 12:00:00+00:00", "updated_at": "2026-10-22 12:12:56.613836+00:00", "lang": "en", "topics": ["ai-agents", "ai-research", "artificial-intelligence", "ai-tools", "developer-tools"], "entities": ["Google Research", "Harness²", "Harness2", "Vertex AI", "Apache 2.0"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/google-open-sources-harness2-self-improving-ai-agents-without-retraining", "markdown": "https://wpnews.pro/news/google-open-sources-harness2-self-improving-ai-agents-without-retraining.md", "text": "https://wpnews.pro/news/google-open-sources-harness2-self-improving-ai-agents-without-retraining.txt", "jsonld": "https://wpnews.pro/news/google-open-sources-harness2-self-improving-ai-agents-without-retraining.jsonld"}}