{"slug": "harness-aware-evaluation-of-plugins-openai-cookbook", "title": "Harness-aware evaluation of plugins (OpenAI Cookbook)", "summary": "OpenAI published a Cookbook walkthrough showing that a plugin scoring well in isolated tests can still fail inside the product it ships in, because the harness around the model controls the prompt and the tools it can reach. The walkthrough runs one set of test cases at three fidelity levels — direct tool tests, a standalone function-calling loop over MCP, and the Codex harness now part of the ChatGPT desktop app — using a simplified BLS-data plugin on synthetic data with five curated evaluation cases. A separate evaluation of the production BLS plugin covered 108 questions across 56 BLS surveys, with code and data not shipped alongside the article.", "body_md": "## Overview\n\nA plugin that scores well in your own evaluation can still fail once it runs inside the product it ships in. The harness around the model accounts for much of that gap, since it controls the prompt the model sees and the tools it can reach. This article runs one set of test cases at three levels of fidelity and shows how to trace a difference back to the level that caused it.\n\nModels often need current, specialized data and actions that training alone cannot provide. A\n[plugin](https://developers.openai.com/plugins/concepts/plugins) can package skills or an MCP server\nwith callable tools alongside natural-language instructions. This plugin connects a Codex harness,\nnow part of the ChatGPT desktop app, to Bureau of Labor Statistics (BLS) data.\nBecause the [BLS API](https://www.bls.gov/developers/) requires known series IDs, the plugin first\nresolves a user’s question into ranked candidate series.\n\nA plugin like this one is best evaluated across multiple levels, to balance speed, cost, and fidelity. The same model can resolve every ambiguous query correctly in a small, controlled loop, then barely call a tool once it’s running inside a real agent. A score from one setup doesn’t transfer to the other, since it measures the setup as much as it measures the model.\n\n| Approach | What it tests | What it misses | What it requires | Cost | Fidelity | \n|---|---|---|---|---|---|\n| **Direct tool tests** | Structured inputs sent directly to the tools, ideal for debugging tool logic | Natural-language interpretation and tool selection | Tool code and a structured corpus | Free | Lowest | \n| **Standalone function-calling loop** | Natural-language interpretation and tool selection in a controlled function-calling loop over MCP | Product harness behavior | A running MCP server, model API access, and the loop code | Moderate | Moderate | \n| **Product harness** | The same queries run through the [Codex](https://openai.com/codex/) harness with the plugin installed | Client-specific behavior that the evaluation setup might not replicate. Compare representative samples with the target client to identify gaps | A running MCP server, installed plugin, Codex authentication (here with API key), and harness configuration | Highest | Highest, short of the ChatGPT desktop app itself | \n\nThe walkthrough uses a simplified example plugin that runs on synthetic data and deliberately covers only part of the domain, so evaluation failures show up. It also uses a curated set of five evaluation cases.\n\nThe results chapter reports a separate evaluation of the production BLS plugin, 108 questions across 56 BLS surveys. Its code and data do not ship with this article.\n\n## Setup\n\nThe runnable example for this article ships alongside it. The companion [README](https://github.com/openai/openai-cookbook/blob/main/examples/partners/harness_aware_plugin_evals/README.md) lists the\nprerequisites and walks through every step in detail,\nwhile the [Makefile](https://github.com/openai/openai-cookbook/blob/main/examples/partners/harness_aware_plugin_evals/Makefile) wraps each one in a single target.\n\nThe BLS domain is too dense to teach evaluation mechanics on directly. From here on, the article works against a miniature of it, small enough to read in full and to run on a laptop in minutes.\n\n### Dependencies\n\nThe example needs [uv](https://docs.astral.sh/uv/getting-started/installation/),\n[Node.js with npm](https://nodejs.org/en/download), and an OpenAI API key for the runs\nthat call a model.\n\nThe Python dependencies install from the shipped lockfile into the kernel’s environment:\n\n```\n!uv sync --active --inexact --no-dev --quiet\n```\n\nCopy `.env.example` to `.env` and fill in the key. The standalone loop takes its configuration from\n`.env`. Codex takes the model and key from `.env` and the server URL from the plugin’s\n`.mcp.json`.\n\nThe direct tool tests do not use `.env` and cost nothing.\nThe other two approaches require a valid OpenAI API key and make paid model calls.\nA run costs roughly $0.01 at API prices with the default settings.\nSet `EVAL_MODEL` to a model your key can reach. It is independent of the models compared in\nthe results chapter.\n\nBecause these two approaches call the MCP server over HTTP, the server has to be running\nbefore either eval starts. `make server` runs it in the foreground, so it needs its own terminal.\n\nThe notebook that ships with this article costs nothing to run. The direct tool tests run live.\nThe other two approaches replay the recorded runs under `data/`. With `RUN_LIVE=1`, the\nstandalone loop and the Codex run without server instructions go live.\nThe Codex run with server instructions stays recorded unless you\nreproduce it by hand.\n`prepare()` in `notebook_setup.py` loads `.env` and points Codex at this example. In\nlive mode, it also checks the key, clears `outputs/`, writes the Codex config, and starts the\nserver if needed:\n\n``` python\nfrom notebook_setup import prepare, stop_server\n\nRUN_LIVE = prepare()\n```\n\n[Promptfoo](https://www.promptfoo.dev/) is an open-source eval runner. It loads test cases from a\nYAML file, sends each to a provider (the model or script that answers it), and grades the output\nwith assertions. It runs both the standalone loop and Codex in this example, and its config file\ncontains the test cases all three approaches share. Install the Node packages, including promptfoo,\nfrom the lockfile:\n\n```\nif RUN_LIVE:\n    !npm ci --silent\n```\n\n### The example plugin\n\nThe article keeps four terms apart:\n\n- Tool: a single function exposed to the model, defined by a schema for its parameters.\n- MCP server: a process that implements the Model Context Protocol and publishes callable tools to clients.\n- Plugin: a package that bundles those parts with the metadata a product needs to install it.\n- Harness: everything around the model at run time, including the agent loop, prompt formatting, tool call execution, turn limits, and sandbox policy.\n\nThe server exposes two tools, both defined in `server.py` and served over HTTP through\n[FastMCP](https://gofastmcp.com/). `resolve(indicators)` takes one or more economic indicators\nand returns a ranked list of candidate BLS series, each with a confidence score.\n`fetch_bls_data(series_ids)` takes series IDs and returns their observations, or an error record\nfor an ID the catalog does not hold.\n\nBehind those two tools sits synthetic fixture data built to reproduce, in miniature, three\nproperties of the real domain (a default among several valid answers, a true ambiguity, and a data\ngap), with CPI deliberately left out. The series IDs look like real BLS codes,\nsuch as `LNS14000000` and `CES0000000001`, but have no internal structure of their own. Three\nindicators represent the three properties:\n\n- `unemployment rate` has a deliberate ambiguity.`resolve` returns the seasonally adjusted\nseries and the unadjusted one, ranked, with the adjusted version as the conventional default, and\nleaves the choice to the caller.\n- `employment` returns two candidates that measure different things, tied at the same confidence:\nthe household survey’s employment level and nonfarm payrolls, with no default to fall back on.\n- `consumer price index` has no match. The resolver returns an empty candidate list\nalongside a note saying the gap is definitive.\n\nCalling `resolve` directly shows all three at once:\n\n``` python\nfrom server import IndicatorQuery, resolve\n\nresolve(indicators=[\n    IndicatorQuery(indicator=\"unemployment rate\"),\n    IndicatorQuery(indicator=\"employment\"),\n    IndicatorQuery(indicator=\"consumer price index\"),\n])\n{'results': [{'indicator': 'unemployment rate',\n   'candidates': [{'series_id': 'LNS14000000',\n     'title': 'Unemployment Rate - Seasonally Adjusted',\n     'confidence': 1.0},\n    {'series_id': 'LNU04000000',\n     'title': 'Unemployment Rate - Not Seasonally Adjusted',\n     'confidence': 0.6}]},\n  {'indicator': 'employment',\n   'candidates': [{'series_id': 'LNS12000000',\n     'title': 'Employment Level - Seasonally Adjusted',\n     'confidence': 0.5},\n    {'series_id': 'CES0000000001',\n     'title': 'All Employees, Total Nonfarm - Seasonally Adjusted',\n     'confidence': 0.5}]},\n  {'indicator': 'consumer price index',\n   'candidates': [],\n   'note': 'No series in this catalog covers that indicator. This is a definitive answer, not a transient failure.'}]}\n```\n\n### The example dataset\n\nEvery case in the example corpus has two forms side by side: a structured field the\ndeterministic check passes to the resolver, and a natural-language query that a\nmodel has to turn into the same request on its own. `expected_series_ids` is the shared expectation\nfield. An empty string marks a case\nwhere no series should be fetched. The ambiguous case has one extra field,\n`expected_top_ids`: the resolver should return both tied candidates, while the model should\nfetch nothing and ask. All five cases are in one file, `promptfooconfig.yaml`, so\nevery approach reads the same corpus. Three of them appear below:\n\n```\ntests:\n  - description: unemployment-rate\n    vars:\n      indicator: unemployment rate\n      query: What is the U.S. unemployment rate?\n      expected_tools: \"resolve,fetch_bls_data\"\n      expected_series_ids: \"LNS14000000\"\n    metadata:\n      mechanism: ambiguous_default\n      expected_type: exact\n\n  - description: consumer-price-index\n    vars:\n      indicator: consumer price index\n      query: What is the current consumer price index?\n      expected_tools: \"resolve\"\n      expected_series_ids: \"\"\n    metadata:\n      mechanism: coverage_gap\n      expected_type: gap\n\n  - description: employment-ambiguous\n    vars:\n      indicator: employment\n      query: What is U.S. employment?\n      expected_tools: \"resolve\"\n      expected_series_ids: \"\"\n      expected_top_ids: \"LNS12000000,CES0000000001\"\n    metadata:\n      mechanism: ambiguous_concept\n      expected_type: clarify\n```\n\n`mechanism` tags what a case tests and `expected_type` what counts as a correct answer. Both\nare stored as promptfoo metadata. The grader does not use them. They show up in the run report\nand let you filter a run by those fields:\n\n```\nnpx promptfoo eval -c promptfooconfig.yaml --filter-metadata mechanism=coverage_gap\n```\n\nWith that filter, you can spot a pattern like “the model only fails on coverage gaps” without reading every row manually.\n\n## Direct tool tests: A deterministic check with no model in the loop\n\nThe direct tool tests run this same example corpus’s structured form through the plugin’s `resolve`\ntool. There is no natural-language question to interpret and no model deciding which tool to\ncall. The check loads `promptfooconfig.yaml` and calls `resolve` directly from `server.py`, with no\nMCP client, promptfoo, or network involved.\n\n``` python\nimport yaml\n\nfrom server import IndicatorQuery, resolve\n\nwith open(\"promptfooconfig.yaml\") as f:\n    CASES = [test[\"vars\"] for test in yaml.safe_load(f)[\"tests\"]]\n\ndef top_candidates(indicator: str) -> set[str]:\n    result = resolve(indicators=[IndicatorQuery(indicator=indicator)])\n    candidates = result[\"results\"][0][\"candidates\"]\n    if not candidates:\n        return set()\n    best = max(candidate[\"confidence\"] for candidate in candidates)\n    return {c[\"series_id\"] for c in candidates if c[\"confidence\"] == best}\n\ntop_candidates(\"consumer price index\")\nset()\n```\n\nThe runner compares that set against each case’s expectation and prints one line per case.\nThe consumer price index case expects no series because it isn’t one of the four\nindicators the example plugin recognizes. The runner also checks that `resolve` returns an empty\ncandidate list there instead of guessing.\n`level_one.py` runs all five cases in about a second:\n\n```\n%run level_one.py\nphp\nPASS  'unemployment rate'                 -> LNS14000000\nPASS  'labor force participation rate'    -> LNS11300000\nPASS  'nonfarm payroll employment'        -> CES0000000001\nPASS  'consumer price index'              -> None\nPASS  'employment'                        -> CES0000000001, LNS12000000\n\n5/5 passed\n```\n\nStrengths of the direct tool tests include:\n\n- Fast and free: it needs no API key, network call, or model.\n- Deterministic: the same input produces the same output on every run, without the run-to-run noise a model would add, and the script exits non-zero when a case fails, so it drops into CI.\n- Precisely diagnostic: a failure points at the resolver’s own logic, with no question about whether the model or the harness caused it.\n\nThe direct tool tests do not check whether a model can turn a plain-language question into appropriate tool calls and a final answer. The next two approaches add a model, starting with a standalone function-calling loop on the same five cases.\n\n## A standalone function-calling loop\n\nA standalone evaluation uses a small, application-owned function-calling loop instead of a full agent harness such as Codex. The model receives the MCP tools, chooses which ones to call, observes their results, and produces a final answer.\n\nThe main advantages are:\n\n- Lower cost: the same query uses fewer tokens than it would inside a full agent harness.\n- Simple setup: there is no plugin installation, agent home directory, shell, approval policy, or containerized Codex runtime.\n- Straightforward debugging: no hidden behavior, one configuration file, and failures that are easy to isolate.\n\nThe code snippets can be adapted to a different MCP server by changing the prompts and the tool names. Promptfoo runs this example, but another eval runner can implement the same three approaches.\n\n### The function-calling loop\n\nThe loop uses the [Responses API](https://developers.openai.com/api/reference/responses/overview). It exposes\nMCP tools and records every call for later inspection. It carries the whole conversation in `input` and sets `store: False`, so it needs no\nserver-side state and works on a zero-data-retention key.\n\nThe tools the server advertises become the tools the model sees:\n\n```\nasync with AsyncOpenAI(api_key=os.environ[\"OPENAI_API_KEY\"]) as openai, Client(mcp_url) as mcp:\n    mcp_tools = await mcp.list_tools()\n    tools = [\n        {\n            \"type\": \"function\",\n            \"name\": tool.name,\n            \"description\": tool.description or \"\",\n            \"parameters\": tool.inputSchema,\n        }\n        for tool in mcp_tools\n    ]\n```\n\nEach step sends the whole conversation, and a response that asks for no tool ends the run:\n\n```\nfor step in range(max_steps + 1):\n    response = await openai.responses.create(input=conversation, **request)\n    function_calls = [item for item in response.output if item.type == \"function_call\"]\n    if not function_calls:\n        return {\n            \"answer\": response.output_text,\n            \"tool_calls\": recorded_calls,\n            \"completed\": True,\n        }\n    if step == max_steps:\n        break\n\n    # The API rejects round-tripped items that still carry null-valued fields.\n    conversation += [item.model_dump(exclude_none=True) for item in response.output]\n    for call in function_calls:\n        arguments = json.loads(call.arguments or \"{}\")\n        result = await mcp.call_tool(call.name, arguments)\n        recorded_calls.append({\"name\": call.name, \"arguments\": arguments, \"result\": result.structured_content})\n```\n\nEach tool result goes back into the conversation as a `function_call_output` item, so the next\nstep sees it.\n\n### The assertion\n\nThe standalone loop and Codex share `grade` in `eval_grading.py`. It checks the tool calls\nof a run, not the answer text. A case passes when the expected tools ran, the expected series\nwere returned, no other series was fetched, and the run finished with a non-empty answer. The\n“no other series” rule also catches a made-up ID fetched next to a real one.\nFor `employment-ambiguous`, it checks that `resolve` was called and neither candidate was\nfetched. It cannot tell whether that answer asks the user to clarify, or whether any answer states the\nretrieved value. None of these checks proves the answer is correct. A shipped evaluation also\nneeds an answer-level check, for example one of the [promptfoo assertions](https://www.promptfoo.dev/docs/configuration/expected-outputs/).\nThe five-case counts in the walkthrough come from this grader. The production pass rates come\nfrom a separate LLM rubric judge.\n\nCalling `grade` on a hand-written run that invented a CPI series returns this verdict:\n\n``` python\nfrom eval_grading import grade\n\ninvented = [{\"name\": \"resolve\", \"arguments\": {\"indicators\": [\"consumer price index\"]}, \"result\": {}},\n            {\"name\": \"fetch_bls_data\", \"arguments\": {\"series_ids\": [\"CUUR0000SA0\"]}, \"result\": {}}]\ngrade(invented, {\"expected_tools\": \"resolve\", \"expected_series_ids\": \"\"}, \"unavailable\", completed=True)\n{'pass': False,\n 'score': 0.0,\n 'reason': \"fetched series outside the expectation: ['CUUR0000SA0']\"}\n```\n\n### Wiring the dataset to the provider\n\nThe dataset is the same `promptfooconfig.yaml` shown in Setup, `tests` block included. Each\ncase’s `vars.query` is the natural-language half of the corpus, and `vars.expected_series_ids`\nis the shared verdict. The rest of the file wires that dataset to the provider and the\nassertion:\n\n```\ndescription: Standalone BLS MCP function-calling evaluation\n\nproviders:\n  - id: file://provider.py\n    label: standalone-mcp\n    config:\n      model: \"{{ env.EVAL_MODEL }}\"\n      model_reasoning_effort: \"{{ env.EVAL_REASONING_EFFORT | default('medium', true) }}\"\n      mcp_url: \"{{ env.MCP_URL }}\"\n\nprompts:\n  - \"{{query}}\"\n\ndefaultTest:\n  assert:\n    - type: python\n      value: file://assert_result.py\n```\n\n### Running the evaluation\n\nThe eval is one promptfoo invocation. `--no-cache` keeps\nrepeats as real model runs instead of promptfoo replays, and the run writes its whole trace to\n`outputs/l2-results.json`:\n\n```\nif RUN_LIVE:\n    !npx promptfoo eval -c promptfooconfig.yaml --no-cache --output outputs/l2-results.json\n```\n\nA run with failing cases exits non-zero, which the Codex run below does on purpose, so the verdict comes from the grading further down instead.\n\n*The promptfoo report for the standalone loop’s recorded five-case run. Each row is one case, and the\nOutputs column holds the model’s final answer alongside the recorded tool calls the grader\nreads.*\n\nIn the included recorded run, all five cases pass, including the consumer price index case.\n`resolve` returns no candidate, and the loop’s system prompt forbids invented series IDs, so the\nmodel reports the gap. The same case fails under Codex.\n\n## Product harness: Codex\n\nPromptfoo’s `openai:codex-sdk` provider sends the same five queries to Codex. It starts Codex\nwith a Codex home that has the BLS plugin installed, and adds the prefix a user gets by typing\n`@BLS` in the desktop app.\n\nThe two setups also get different instructions. The standalone loop’s system prompt tells the model not to invent series IDs and to name both candidates when they tie. Codex gets no such instruction.\n\n### The Codex provider setup\n\n`config.toml` points at the `.codex-home/` marketplace by absolute path and enables the plugin. The repository\nships `config.toml.example`. A live run writes the path for the current checkout, as `make setup`\ndoes for the terminal.\n\nThe same file defines a [permission profile](https://developers.openai.com/codex/permissions)\nto scope what a run may reach:\n\n```\ndefault_permissions = \"eval\"\n\n[permissions.eval.filesystem]\n\":workspace_roots\" = \"deny\"\n```\n\nThe main provider settings are:\n\n```\nproviders:\n  - id: openai:codex-sdk\n    config:\n      model: \"{{ env.EVAL_MODEL }}\"\n      working_dir: .\n      enable_streaming: true\n      cli_env:\n        CODEX_HOME: \"{{ env.CODEX_HOME }}\"\n```\n\nThe rest of the file sets the reasoning effort, disables network access and web search, turns off\napprovals and the git repository check, and points `tests` at `level_3_tests.py`, which loads the\nshared corpus.\n\nHanding an agent a shell inside the project folder creates a serious evaluation risk.\n`promptfooconfig.yaml` holds the expected answers for every test case, and `.env` holds the live\nAPI key. A model with file access could read them and clear each case from the answer\nfile. The permission profile prevents this. The `deny` rule locks the directory against\nreads and writes to keep the measurement about the plugin. The evaluation still runs, because\nthe filesystem rules do not apply to MCP traffic over HTTP. Explicit `CODEX_HOME` points Codex\nat the plugin. Without it, Codex starts from the user’s default home, the BLS plugin does\nnot load, and the model cannot call `resolve` or `fetch_bls_data`.\n\nThe prompt itself stays as the natural-language query. The prefix goes in through\n`defaultTest.options.prefix`, and the text below matches what Codex sees when the user adds a\n`@BLS` mention in the desktop app.\n\n```\nprompts:\n  - \"{{query}}\"\n\ndefaultTest:\n  options:\n    prefix: \"[@BLS](plugin://BLS@cookbook-plugins) \"\n```\n\n`enable_streaming: true` makes promptfoo retain the Codex item stream in the provider response.\nThe assertion reads `mcp_tool_call` items from that trace, fails the case when any of them failed,\nand normalizes them into the same\n`{\"name\", \"arguments\", \"result\"}` format the standalone loop records, so both approaches reach the same grader in\n`eval_grading.py`.\n\n```\nmcp_items = [item for item in raw.get(\"items\", []) if item.get(\"type\") == \"mcp_tool_call\"]\nfor item in mcp_items:\n    if item.get(\"error\") or item.get(\"status\") == \"failed\":\n        return {\n            \"pass\": False,\n            \"score\": 0.0,\n            \"reason\": f\"MCP tool call failed: {item.get('tool')}\",\n        }\n\ncalls = [\n    {\n        \"name\": item.get(\"tool\"),\n        \"arguments\": _arguments(item),\n        \"result\": (item.get(\"result\") or {}).get(\"structured_content\"),\n    }\n    for item in mcp_items\n]\n```\n\nThe rest of `assert_codex_result.py` pulls the trace out of the provider response and counts a run\nas finished when the provider reported no error and the final response is not empty.\n\n### Running the example\n\nWith `.env` filled in and the server running, the plugin goes into that same `CODEX_HOME`, and the\nevaluation runs against `promptfooconfig.codex.yaml`. `CODEX_HOME` reaches both commands through the\nenvironment loaded in Setup:\n\n``` python\nimport subprocess\n\nif RUN_LIVE:\n    subprocess.run([\"npx\", \"codex\", \"plugin\", \"add\", \"BLS@cookbook-plugins\"], check=True)\n    !npx promptfoo eval -c promptfooconfig.codex.yaml --no-cache --output outputs/l3-results.json\n```\n\nA failed install stops the cell. Codex would otherwise reach no tools, and every case would fail for the wrong reason. The grading would report all five as missing the tools they expected.\n\n*The same five cases through the Codex harness, before the server publishes any guidance.\n`consumer-price-index` fails because it fetched an ID the resolver never returned, and\n`employment-ambiguous` because it fetched candidates the resolver did return instead of asking\nwhich one the user wanted.*\n\nIn the runs recorded\nunder `data/`, the standalone loop passes all five cases while Codex scores three out of five. A fresh\nlive run may produce a different count because both failures depend on the path the model happens to\ntake.\n\nThe example server has no CPI series, and `resolve` correctly returns\nnothing. Codex then supplies an ID on its own: it calls `fetch_bls_data` with `CUUR0000SA0`, the\nreal BLS CPI ID, which the server never returned. The answer Codex writes is true because\nit tells the user the figure is unavailable, but the unsupported fetch fails the case.\n\nThe employment case fails differently.\nCodex may narrow the request to one measure before calling `resolve`,\nchoose one of the tied candidates afterward, or fetch both.\nThe exact path can vary from run to run, but each of those retrieves at least one series\ninstead of stopping to clarify, so the grader fails it.\n\nThe server can give Codex the missing instructions. MCP servers send an\n`instructions` field during the\n[handshake](https://modelcontextprotocol.io/specification/2025-06-18/basic/lifecycle), which the\nspecification describes as a hint a client may add to the system prompt. Codex passes it\nto the model. The standalone loop uses its own system prompt and ignores the field.\n\nThe example leaves that field empty, so a fresh run can still show these failures. One string\non the `FastMCP` constructor provides that guidance:\n\n```\nGUIDANCE = \"\"\"Answer only from this catalog. Call resolve first, passing the indicator the user\nasked about without narrowing it. Pass fetch_bls_data only the series IDs that resolve returned,\nnever one from your own knowledge. If resolve returns no candidate, say the data is unavailable. If\nseveral candidates tie for the best confidence, do not call fetch_bls_data at all. Name them and ask\nwhich one the user wants.\"\"\"\n\nmcp = FastMCP(\"BLS evaluation fixture\", instructions=GUIDANCE)\n```\n\nAfter editing `server.py`, restart the MCP server and rerun the Codex eval. The server sends\n`instructions` only in its `initialize` response, and a running server keeps sending the old,\nempty value.\n\nIn the recorded run with the instructions (`data/l3-trace-after.json`), all five Codex cases\npass, and neither the tools nor the resolver changed.\n\nIn the recorded runs a case took about 10 seconds under Codex and 25 in the standalone loop.\n\n### Replaying the three runs\n\nThe three recorded runs are in `data/`. The graders can evaluate them again without an API\nkey, server, or Codex login. The run with instructions is always loaded from its recording,\nbecause reproducing it live means applying the code change and rerunning the Codex cell.\n`replay.py` regrades all three runs, and `make replay` runs it from a terminal:\n\n```\n%run replay.py\nStandalone loop: 5/5 from data/l2-trace.json\nCodex, empty instructions: 3/5 from data/l3-trace-before.json\n    consumer-price-index: fetched series outside the expectation: ['CUUR0000SA0']\n    employment-ambiguous: fetched series outside the expectation: ['CES0000000001', 'LNS12000000']\nCodex, with instructions: 5/5 from data/l3-trace-after.json\n```\n\nThe notebook stops the server it started:\n\n```\nstop_server()\n```\n\n### A spot check against the ChatGPT desktop app\n\nEvery product harness number in this article comes from the OpenAI Codex SDK provider, not from the\nChatGPT desktop app itself. A sanity check on this example shows the same tool calls as the Codex\neval, including the inferred `CUUR0000SA0` ID. A thorough check in an actual project should cover a wider set of examples.\n\n*The consumer price index case run by hand in the ChatGPT desktop app.*\n\n## Reading the results: Model-harness compatibility\n\nThe example plugin’s CPI case illustrated model-harness compatibility on a small scale: the same five questions produced different results in the standalone loop and under Codex. The production plugin with live API calls and more tools has a more typical scope, and the evaluation covers 108 questions across 56 BLS surveys. The benchmark results come from this 108-question corpus and use an LLM rubric judge rather than the tool-call checks from the five-case example.\n\n| Setting | Value | \n|---|---|\n| Dataset | Held-out test split of the production corpus, 108 single-turn questions across 56 BLS surveys | \n| Judge | LLM rubric over the final answer, `gpt-5.6-sol` | \n| Versions | Codex 0.159.2, promptfoo 0.123.1 | \n| Repeats | Table 1 uses one run per model, Tables 2 and 3 use three runs, with the response cache disabled | \n\n*The production benchmark configuration.*\n\n### Three models, one harness\n\nThree models run through that corpus under Codex, the product harness. The model is the only variable:\n\n| Model | Pass rate | Tool calls / question | Reasoning tokens / question | \n|---|---|---|---|\n| GPT-6.1 Sol | 0.94 | 4.02 | 19 | \n| GPT-6 Astra | 0.94 | 3.74 | 14 | \n| GPT-6 Luna | 0.88 | 5.54 | 1,229 | \n\n*Table 1: Pass rates and costs for the production plugin under Codex, across the 108-question\nprimary evaluation dataset, with the same grader for all models.*\n\nGPT-6.1 Sol and GPT-6 Astra differ by one question out of the 108 in the corpus in this run. These results provide too little evidence to rank the two reliably.\n\nGPT-6 Luna scores lower than the larger models while spending about 40% more tool calls than GPT-6.1 Sol and over 60 times its reasoning tokens per question.\n\n### Observed gap changes\n\nA gap between models also depends on the harness that produced it. GPT-5-mini is a legacy model, kept as a comparison because it was the cheap baseline the plugin was developed against. Its drop under Codex prompted the harness comparison in this article. GPT-6 Luna and GPT-5-mini are far apart under Codex and much closer in the standalone loop:\n\n| Harness | GPT-6 Luna | GPT-5-mini | \n|---|---|---|\n| Product harness (Codex) | 0.88 | 0.43 | \n| Standalone function-calling loop | 0.74 | 0.64 | \n\n*Table 2: The production plugin across two harnesses, the same questions as Table 1.*\n\nIn the standalone loop GPT-6 Luna and GPT-5-mini differ by about 10 points on average and by 1 to 19 points in a single run. Under Codex the gap is much larger. GPT-6 Luna gains 14 points and GPT-5-mini loses 21 compared with the standalone loop.\n\n### Why GPT-5-mini performs worse under Codex\n\nAbout half of GPT-5-mini’s failures there involve the shell. The model calls the agent’s shell and runs commands that do nothing. About half of its answers end with a question to the user. Across the whole corpus it calls a tool in about 85 in 100 questions under Codex, against nearly every question for GPT-6 Luna on that same harness. The standalone loop has no shell. On the identical corpus there, GPT-5-mini calls a tool in about 95 in 100 questions.\n\n### Why GPT-6 Luna performs worse in the standalone loop\n\nGPT-6 Luna fails for a different reason. In the standalone loop most of its failed answers either say the requested data is unavailable or end at the loop’s step limit. Under Codex those turns continue, which would explain most of the extra tool calls per question there, though these runs do not isolate the mechanism.\n\n## Accounting for non-determinism\n\nTreat the pass rates in this article as relative measures. Pass rates differed by 14 and 21 percentage points between the standalone loop and Codex. The repeat study below illustrates how results can vary across runs.\n\nThe direct tool tests should return the same verdict every time because they run with no model in the loop. Runs that involve a model, however, can produce different results across runs of the same configuration:\n\n- A model can choose an alternative path to solve the task.\n- An LLM judge can evaluate the same answer differently.\n- Differences may also be caused by external updates of the models or the harness.\n\nThe same Codex configuration, run three times with GPT-6 Luna over the 108 questions from the benchmark configuration above, gives the following spread:\n\n| Repeat | 1 | 2 | 3 | \n|---|---|---|---|\n| Pass rate | 0.88 | 0.89 | 0.86 | \n| Tool calls / question | 5.54 | 5.70 | 5.20 | \n| Reasoning tokens / question | 1,229 | 1,195 | 1,146 | \n\n*Table 3: Three repeated runs of the production plugin under Codex.*\n\n- The overall pass rate barely moves: a spread of 3 points across all three runs.\n- Individual questions move far more than that spread suggests. 23 of the 108 (21%) flipped between a pass and a fail somewhere across the three runs while the aggregate stayed steady.\n- Tool calls swing just as hard: one question took 5 tool calls in one repeat and 19 in another.\n\n## Practical takeaways\n\nUse the cheapest approach that can expose the issue you are investigating:\n\n- Run the direct tool tests whenever resolver logic or fixture data changes.\n- Use the standalone loop while iterating on tool descriptions, prompts, grading, or model choice.\n- Run the product harness for each new model instead of assuming the two approaches agree. If they do, iterate in the standalone loop and keep the product harness for release checks and results for stakeholders.\n\nWrite test cases where a plausible answer is wrong, such as when the correct answer is a short list\nof series, a request to clarify (“are prices rising?” can mean consumer or producer prices), or no\ndata at all. Tag each case with what it tests and what counts as correct, as the example’s\n`mechanism` and `expected_type` do. A run can then be filtered to the slice under development,\nand the same tags show which areas need new test cases most.\n\nCompare eval traces with production runs to catch differences the configuration misses, such as agent instructions, tool availability, or runtime versions. A pass rate measured once can also look far steadier than the case-by-case reality underneath it. Repeat the same cases automatically to measure how stable they are, and treat that stability as a metric of its own. A system whose verdict on the same question swings between runs may be less useful than one with a slightly lower average score but more predictable behavior.\n\nCheaper harnesses and smaller models help during fast iteration, but they only approximate what ships. Production behavior comes from the combination of the deployed harness and model. A few things to confirm before trusting that final run:\n\n- The model sees the same framing in the eval that it sees in production, and every config value meant to reach it does.\n- The same configuration has run more than once with caching turned off.\n- Tool calls and reasoning tokens per question are checked alongside pass rate.\n- A human has reviewed a representative sample of the answers.\n\n### Trying it yourself\n\nThe multi-level structure and the pattern of sharing one dataset and grader are reusable, but adapting them to another plugin requires changes in the individual components:\n\n| Area | Files | Action | What to change | \n|---|---|---|---|\n| MCP server | `server.py` | Replace | Use the target server and update the MCP URLs consumed by the standalone loop and the plugin. | \n| Test cases | `promptfooconfig.yaml` | Update | Supply domain-specific queries, structured inputs, expectations, and metadata. | \n| Evaluation logic | `level_one.py` ,`eval_grading.py` | Update | Adapt the direct tool tests and shared grading to the target tool schemas and expectations. The grader checks tool names, argument formats, and expected-result fields, so it has to change when those do. | \n| Instructions | `provider.py` and the MCP server’s`instructions` | Update | Align the standalone system prompt and product-visible guidance with the intended behavior. A product harness usually supplies its own system prompt, so guidance for it goes in the tool descriptions or the MCP server’s instructions. | \n| Plugin configuration | `.codex-home/marketplaces/` ,`.mcp.json` ,`config.toml.example` | Update | Set the plugin identity, marketplace location, MCP endpoint, and isolated Codex home. | \n| Evaluation structure | All three approaches | Retain | Keep the separation between direct tool tests, a standalone loop, and the product harness. | \n\n## Further reading\n\n- [AI Agents That Matter](https://arxiv.org/abs/2407.01502) explains how evaluating agents differs from evaluating models.\n- [τ-bench](https://arxiv.org/abs/2406.12045) introduces`pass^k` , the probability that all`k` independent attempts succeed.\n- [Moving from OpenAI Evals to Promptfoo](https://developers.openai.com/cookbook/examples/evaluation/moving-from-openai-evals-to-promptfoo) rebuilds an OpenAI Evals suite in promptfoo case by case.\n- [Codex sandboxing](https://developers.openai.com/codex/concepts/sandboxing) covers the modes a Codex run can execute under and what each one allows.\n\n## Contributors\n\nThis cookbook serves as a joint collaboration effort between OpenAI and [deepsense.ai](https://deepsense.ai/), who built and evaluated the BLS connector.", "url": "https://wpnews.pro/news/harness-aware-evaluation-of-plugins-openai-cookbook", "canonical_source": "https://developers.openai.com/cookbook/examples/partners/harness_aware_plugin_evals/harness_aware_plugin_evals", "published_at": "2026-10-27 14:32:08+00:00", "updated_at": "2026-10-27 14:41:42.693470+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "agent-protocols", "ai-products"], "entities": ["OpenAI", "Codex", "ChatGPT", "Bureau of Labor Statistics", "OpenAI Cookbook", "MCP"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/harness-aware-evaluation-of-plugins-openai-cookbook", "markdown": "https://wpnews.pro/news/harness-aware-evaluation-of-plugins-openai-cookbook.md", "text": "https://wpnews.pro/news/harness-aware-evaluation-of-plugins-openai-cookbook.txt", "jsonld": "https://wpnews.pro/news/harness-aware-evaluation-of-plugins-openai-cookbook.jsonld"}}