{"slug": "oss-radar-07-can-promptfoo-preserve-the-evidence-behind-an-ai-answer", "title": "OSS Radar #07: Can Promptfoo Preserve the Evidence Behind an AI Answer?", "summary": "Promptfoo version 0.122.2 can preserve citation evidence through its database and JSON exports if the data is captured explicitly, according to September 8 checks by OSS Radar. The built-in OpenRouter summary omitted the fixture's structured citation fields, while the transport cache retained them, and a custom provider preserved successful and failed attempt records after a database restart. The checks used synthetic responses, and OSS Radar concluded Promptfoo is worth trying for citation evaluation with a tested capture layer.", "body_md": "Promptfoo's evaluation runner can keep citation evidence through its database and JSON exports, provided we capture it explicitly. In our September 8 checks of version 0.122.2, the built-in OpenRouter summary omitted the fixture's structured citation fields; the transport cache retained them. A custom provider also preserved successful and failed attempt records after a database restart. This makes Promptfoo worth trying for citation evaluation, with a tested capture layer. All responses in these checks were synthetic. [Original comparison](https://github.com/gkoreli/blog/blob/dabd081506de2e0a8dba6778b4e43c0bb83d5c13/packages/blog/drafts/research/oss-radar-07/repro/installed/README.md); [failure and export checks](https://github.com/gkoreli/blog/blob/5fc2dc5dc40ad2397a78325e3a192485b44e8cb5/packages/blog/drafts/research/oss-radar-07/repro/capture/README.md).\n\n- **The adapter decides what reaches the evaluator.** Answer text survived all three original paths; structured citations survived the cache and custom capture.\n- **The repair survives more than a successful response.** Ten controlled cases retained every attempt through the library exporter and a fresh CLI process. One local trace also kept the explicit link to its saved attempt.\n- **Preservation is the first step in citation evaluation.** Counting an answer's references, checking their support, and explaining what triggered a run need different records.\n\n## What Promptfoo is building\n\nPromptfoo is a command-line tool and library for testing AI applications, with an [MIT-licensed open-source implementation](https://github.com/promptfoo/promptfoo/blob/89052308bce06f53645b1f189ada5ac9d1897347/LICENSE). You supply test cases, choose models or an application to call, and define checks. It runs the cases and lets you compare results. The [project overview](https://github.com/promptfoo/promptfoo/blob/89052308bce06f53645b1f189ada5ac9d1897347/README.md) also puts adversarial testing and code review within its scope.\n\nIts public purpose is to help teams ship secure, reliable AI applications. Founders Ian Webster and Michael D'Angelo [describe a shift](https://www.promptfoo.dev/blog/promptfoo-joining-openai/) from systematic application testing toward the security and behavioral risks that blocked deployment. That explains why the project reaches beyond comparing answers.\n\n| Part | What it does | Evidence | \n|---|---|---|\n| Evaluations | Runs chosen cases against providers or applications and checks their responses | Our [installed evaluation](https://github.com/gkoreli/blog/blob/dabd081506de2e0a8dba6778b4e43c0bb83d5c13/packages/blog/drafts/research/oss-radar-07/repro/installed/README.md) exercises this path | \n| Red teaming | Generates adversarial cases, applies attack strategies, and tests a target application | The [architecture](https://github.com/promptfoo/promptfoo/blob/89052308bce06f53645b1f189ada5ac9d1897347/site/docs/red-team/architecture.md) defines the parts; the[run implementation](https://github.com/promptfoo/promptfoo/blob/89052308bce06f53645b1f189ada5ac9d1897347/src/redteam/shared.ts) connects generation to evaluation | \n| Code scanning | Reviews code changes for LLM-related security risks and supplies findings in the development workflow | The [scanner implementation](https://github.com/promptfoo/promptfoo/blob/89052308bce06f53645b1f189ada5ac9d1897347/src/codeScan/scanner/index.ts) calls a scan service; the[GitHub Action](https://github.com/promptfoo/promptfoo/blob/89052308bce06f53645b1f189ada5ac9d1897347/code-scan-action/src/main.ts) handles the review workflow | \n\nAugust's work strengthened the records behind those tests. The [August 26 release](https://github.com/promptfoo/promptfoo/releases/tag/0.122.1) added per-test roots, target spans, agent telemetry integrations, and fuller token accounting. The [August 28 scanner-action release](https://github.com/promptfoo/promptfoo/releases/tag/code-scan-action-0.2.0) hardened its installation and dependencies. These are different release tracks. Our installed runner came from the separate [0.122.2 release](https://github.com/promptfoo/promptfoo/releases/tag/0.122.2), also published August 28. The scanner itself predates August: its [engineering introduction](https://www.promptfoo.dev/blog/building-a-security-scanner-for-llm-apps/) appeared in December 2025.\n\n[Promptfoo 0.123.0](https://github.com/promptfoo/promptfoo/releases/tag/0.123.0) shipped on September 10. The experiments here remain pinned to 0.122.2; we have not rerun them on the newer release.\n\nI read that work as an effort to make AI testing part of the normal development process: run a test, inspect the execution, locate a failure, and check a repair. Traces and accounting help explain what happened inside a test. They do not make its judgments correct or its evidence complete.\n\nThe project now [identifies itself as part of OpenAI](https://www.promptfoo.dev/about/). Its founders' March announcement commits to maintaining open-source red teaming, scanning, and evals across providers. [OpenAI's announcement](https://openai.com/index/openai-to-acquire-promptfoo/) sets out an additional direction: integrate testing, remediation, reporting, and traceability into Frontier. That is a stated product plan; these sources do not establish that the promised integration has shipped.\n\nThe distinction matters for adoption. The MIT runner gives us inspectable code and an extension interface. A connected feature can still depend on a service: the scanner's [default API host](https://github.com/promptfoo/promptfoo/blob/89052308bce06f53645b1f189ada5ac9d1897347/code-scan-action/action.yml) is Promptfoo's cloud. We need to judge the specific path we use. For this blog, the relevant bet is whether its reusable evaluation and execution records can support our citation study; the citation pipeline remains work we must build and verify.\n\n## Citation analytics needs the answer\n\nOur [blog analytics](https://gkoreli.com/first-party-analytics-for-a-personal-blog) can record requests that reach the site. A citation exists in an answer somewhere else. Bringing those observations together starts with preserving what each system can see.\n\nAI citations are a topic; counting them is analytics, and checking whether their sources support an answer is evaluation. There is already publisher-facing prior art. Microsoft's [AI Performance preview](https://blogs.bing.com/webmaster/February-2026/Introducing-AI-Performance-in-Bing-Webmaster-Tools-Public-Preview), announced February 10, reports citation activity across its supported AI surfaces and selected partners. Its coverage is defined by that product. It does not audit every cited claim.\n\nThe opportunity for this blog is to publish inspectable cases: the question, captured answer, cited source, review, and records that connect them. We can then compare those cases with our request observations where a defensible connection exists. Whether the blog appears in the answers remains an open result. The capture method should still help another engineer if it does not.\n\n## Where provider evidence enters the result\n\nPromptfoo's [custom-provider interface](https://github.com/promptfoo/promptfoo/blob/89052308bce06f53645b1f189ada5ac9d1897347/site/docs/providers/custom-api.md) lets an operator bring an application into the runner. For this study, that division is useful: reuse the evaluation machinery and retain provider-specific evidence explicitly.\n\nThe adapter and experiment sections are for engineers choosing or implementing this path. The pinned [OpenRouter provider](https://github.com/promptfoo/promptfoo/blob/89052308bce06f53645b1f189ada5ac9d1897347/src/providers/openrouter.ts) sends the request through a cache-aware transport, then builds a smaller response from answer text, usage, and completion information. It does not copy the fixture's citation list or answer annotations into that response. The evaluator receives the smaller object.\n\nThe shared [response contract](https://github.com/promptfoo/promptfoo/blob/89052308bce06f53645b1f189ada5ac9d1897347/src/contracts/providers.ts) allows both `raw` and arbitrary metadata. This changes the adoption question. We need to test the chosen adapter and export path, rather than assume either that the whole framework preserves everything or that it cannot preserve the evidence at all.\n\n1. \nTransport**Full parsed response** The cache retains the citation list and annotation.adapter reads\n2. \nAdapter**Selected fields** Answer, usage, cache state, cost, and finish reason.evaluator gets\n3. \nSummary**Answer retained** Structured citations are absent in this tested path.\n\n## What survived the installed-package check\n\nThe installed experiment located the omission and tested a repair with the same response. It used one invented payload containing one distinct source URL, represented in a citation list and an annotation. No live model answered the question.\n\nWe ran the built-in provider against a local HTTP server, repeated the evaluation with caching enabled, then sent the same payload through a custom capture provider. Each evaluation used the public summary API and a text assertion. The script serialized the summaries and read them back before checking the fields. [Runnable probe and saved outputs](https://github.com/gkoreli/blog/blob/dabd081506de2e0a8dba6778b4e43c0bb83d5c13/packages/blog/drafts/research/oss-radar-07/repro/installed/README.md).\n\n| Observed path | New local requests | Answer text | Citation-list entries retained | Annotations retained | \n|---|---|---|---|---|\n| Built-in provider, fresh summary | 1 | Preserved | 0 | 0 | \n| Built-in provider, cached summary | 0 | Preserved | 0 | 0 | \n| Custom capture provider, fresh summary | 1 | Preserved | 1 | 1 | \n\nThe [built-in transport cache](https://github.com/gkoreli/blog/blob/dabd081506de2e0a8dba6778b4e43c0bb83d5c13/packages/blog/drafts/research/oss-radar-07/repro/installed/recorded/transport-cache-entry.json) retained the full parsed fixture, including both citation fields. The [custom summary](https://github.com/gkoreli/blog/blob/dabd081506de2e0a8dba6778b4e43c0bb83d5c13/packages/blog/drafts/research/oss-radar-07/repro/installed/recorded/custom-capture.json) retained the exact decoded response text, its hash, the mock request ID, and the structured fields. The omission occurs between the adapter and the summary; the evidence remains available in the cache.\n\nThe repair uses fields Promptfoo already accepts. This is the return object from the [tested capture provider](https://github.com/gkoreli/blog/blob/dabd081506de2e0a8dba6778b4e43c0bb83d5c13/packages/blog/drafts/research/oss-radar-07/repro/installed/probe-installed.ts), after reading the response text and parsing the fixture:\n\n```\nreturn {\n  output: parsed.choices[0].message.content,\n  raw,\n  metadata: { evidence: {\n    providerCitations: parsed.citations,\n    answerAnnotations: parsed.choices[0].message.annotations,\n    requestId: response.headers.get('x-request-id'),\n    responseSha256: sha256(raw),\n    providerReportedUsage: parsed.usage,\n  } },\n};\n```\n\nThis changes what the runner can retain without changing the answer assertion. That original prototype handles the successful fixture only; its saved usage fields are invented inputs, not a billing record.\n\nThe fixture's repeated URL is one source, not two lost citations. Its annotation offsets are not validated, and the text assertion checks transport rather than truth. Tracing, database persistence, CLI export, streaming, failure handling, and live billing were outside that original check. The next experiment covers several of those gaps. The [recorded result](https://github.com/gkoreli/blog/blob/dabd081506de2e0a8dba6778b4e43c0bb83d5c13/packages/blog/drafts/research/oss-radar-07/repro/installed/recorded/result.json) keeps those limits beside the measurements.\n\nFor a real answer, missing citation fields would mean we could not assess citations through that field. They would not mean the model cited nothing. The controlled fixture lets us establish an omission because we know what entered the adapter.\n\n## What survived failures, a restart, and tracing\n\nThe capture layer now preserves failed attempts as well as successful answers. We extended the experiment to ten controlled cases on a Mac, saved each response before parsing it, and checked the same records in the public summary, Promptfoo's JSON exporter, and a separate CLI process reopening the database. All ten retained the complete provider response used by our capture layer. [Method and recorded results](https://github.com/gkoreli/blog/blob/5fc2dc5dc40ad2397a78325e3a192485b44e8cb5/packages/blog/drafts/research/oss-radar-07/repro/capture/README.md).\n\n| Condition | Evaluations | HTTP attempts | What the exported record retains | \n|---|---|---|---|\n| Successful responses | 4 | 4 | The answer plus present, absent, empty, or null citation fields | \n| Five failure conditions | 5 | 5 | Malformed JSON, HTTP 429, an error envelope, no answer, and truncation | \n| HTTP 429 → success | 1 | 2 | Both attempts, their statuses, and the final answer | \n\nThose counts measure record retention. Five evaluations passed the text assertion; five deliberately failed. A successful export of an error is a useful result for this workload. It keeps a later analyst from quietly excluding a failed request or treating it as an answer with no citations. The [retry export](https://github.com/gkoreli/blog/blob/5fc2dc5dc40ad2397a78325e3a192485b44e8cb5/packages/blog/drafts/research/oss-radar-07/repro/capture/recorded/retry-cli-export.json) shows both attempts after restart.\n\nLocal tracing also did useful work. The full-success case produced three spans: the test case, provider target, and assertion. Our capture saved the evaluation ID, test-case ID, and `traceparent` supplied by Promptfoo. They matched the [exported trace](https://github.com/gkoreli/blog/blob/5fc2dc5dc40ad2397a78325e3a192485b44e8cb5/packages/blog/drafts/research/oss-radar-07/repro/capture/recorded/full-cli-export.json), including the exact target span, through both exporters. That is an observed connection between records, rather than a guess from matching timestamps. The pinned [tracing implementation](https://github.com/promptfoo/promptfoo/blob/89052308bce06f53645b1f189ada5ac9d1897347/src/tracing/evaluatorTracing.ts) explains how the runner supplies that context.\n\nA trace still describes only the participating system. We did not capture a remote provider's retrieval history, and the two retry attempts live in our metadata rather than separate spans. The fixture's reported cost remains invented. These checks establish local retention and joins; live response shape, source review, and billing remain untested.\n\n## Who started the task, and what did the answer cite?\n\nAn assistant can use a blog on someone's behalf without that person visiting the site. A person can ask for an article now or commission a report that runs every week. Both serve a human goal. The immediate trigger differs, and neither tells us what the answer cited.\n\nThat distinction has prior art. W3C's [provenance model](https://www.w3.org/TR/2013/REC-prov-dm-20130430/) records what started an activity separately from who bears responsibility for it. A Web Bot Auth [use-case draft](https://datatracker.ietf.org/doc/html/draft-nottingham-webbotauth-use-cases-02#appendix-A.4) likewise questions the human/bot binary. The [protocol draft](https://datatracker.ietf.org/doc/html/draft-ietf-webbotauth-httpsig-protocol-00#section-4.6) leaves human authentication and delegation outside its scope.\n\nA citation belongs to the answer. We need the captured output to observe it, then the source to check whether it supports the claim. Even that leaves a further question: did the model rely on the source? [Research on citation faithfulness](https://arxiv.org/abs/2412.18004) distinguishes actual reliance from a reference that merely agrees with the answer.\n\n| Question | Record that can answer it | What remains unknown | \n|---|---|---|\n| What started this run? | A captured submission or scheduler event linked to the run | A website request alone does not establish that event | \n| Which client made the request? | A verified request signature and its key identity | Human authorization, the immediate trigger, and the later answer | \n| Did the answer cite the article? | The captured answer and its citation fields | Whether a fresh request reached the article's origin | \n| Does the source support the claim? | A review of the claim against the retained source | Whether the model actually relied on that source | \n\nFor controlled runs, we can preserve the submission or scheduler event and link it to the captured output. A receiving website usually lacks that history. The [research artifact](https://github.com/gkoreli/blog/blob/dabd081506de2e0a8dba6778b4e43c0bb83d5c13/packages/blog/drafts/research/oss-radar-07/04-trigger-provenance-and-ai-citations.md) separates task authority, trigger, URL selection, request identity, content access, citation, and source review. Promptfoo can organize records supplied to it; the trace cannot invent the missing ones.\n\n## What still needs code\n\nThe remaining work is in the live capture and source review. The local tests now cover response variants, failed attempts, a bounded retry, database export, and one trace join. They do not establish what a live service returns or what its citations support.\n\nOur [capture provider](https://github.com/gkoreli/blog/blob/5fc2dc5dc40ad2397a78325e3a192485b44e8cb5/packages/blog/drafts/research/oss-radar-07/repro/capture/capture-provider.ts) saves request text and each decoded response before selecting fields. It records missing citation fields separately from empty lists and leaves charge unknown when the response does not expose it. Even a provider-reported charge needs reconciliation with its billing record. Source review needs its own claim boundaries and support rubric. Earlier work such as [ALCE](https://aclanthology.org/2023.emnlp-main.398/) gives citation evaluation a research basis; a returned URL alone is not a support judgment.\n\nThe [proposed measurement contract](https://github.com/gkoreli/blog/blob/dabd081506de2e0a8dba6778b4e43c0bb83d5c13/packages/blog/drafts/research/oss-radar-07/07-citation-measurement-contract.md) keeps counts honest. Citation presence uses assessable answers as its denominator and reports observation coverage alongside it. Support uses reviewed claim–citation relationships. Requests, unique URLs, citation occurrences, and people never become interchangeable units. A capture hash helps detect changed bytes; it does not authenticate the provider or make an answer true.\n\nCollection also needs a known network boundary. Our first attempt encountered an unexpected telemetry request despite the opt-out. The final probe blocked one such POST before transmission, matching the pinned [telemetry implementation](https://github.com/promptfoo/promptfoo/blob/89052308bce06f53645b1f189ada5ac9d1897347/src/telemetry.ts) and the [upstream report](https://github.com/promptfoo/promptfoo/issues/9968), still open at publication. This does not establish prompt disclosure. It does mean the opt-out alone was insufficient for this restricted local run. The method records the interruption and the guard.\n\nThe larger adoption test is whether these additions remain a small capture layer while Promptfoo supplies useful execution and review machinery. A direct runner remains a reasonable alternative if maintaining the integration means rebuilding those parts ourselves.\n\n## The next decision\n\nPromptfoo is worth the next bounded trial for this blog's citation-evaluation workload. The tested interface can retain evidence, and the built-in cache gives useful counterevidence to the strongest loss claim. Anyone who needs complete records from the built-in OpenRouter summary should first add and verify capture on the route they will use.\n\nThe local trace and export checks have passed. The next test is a live preflight that retains the request, answer, exposed citation fields, and charge provenance, then checks the cited source against the answer. A service that does not expose the needed evidence, or an integration that forces us to rebuild most of the runner, would change this decision. The [worklist](https://github.com/gkoreli/blog/blob/main/packages/blog/drafts/research/oss-radar-07/00-worklist-index.md) keeps that test separate from the larger citation-frequency study.\n\nTry Promptfoo when you want its test runner, assertions, and result inspection enough to maintain explicit capture. If you need a complete citation archive from an unmodified adapter, wait or use a direct runner you can audit. Keep the captured records independently replayable; wider adoption depends on the live evidence.\n\n[Installed experiment method · Sep 8, 2026](https://github.com/gkoreli/blog/blob/dabd081506de2e0a8dba6778b4e43c0bb83d5c13/packages/blog/drafts/research/oss-radar-07/repro/installed/README.md)\n\n[Recorded measurements · Sep 8, 2026](https://github.com/gkoreli/blog/blob/dabd081506de2e0a8dba6778b4e43c0bb83d5c13/packages/blog/drafts/research/oss-radar-07/repro/installed/recorded/result.json)\n\n[Saved cache entry · Sep 8, 2026](https://github.com/gkoreli/blog/blob/dabd081506de2e0a8dba6778b4e43c0bb83d5c13/packages/blog/drafts/research/oss-radar-07/repro/installed/recorded/transport-cache-entry.json)\n\n[Saved custom summary · Sep 8, 2026](https://github.com/gkoreli/blog/blob/dabd081506de2e0a8dba6778b4e43c0bb83d5c13/packages/blog/drafts/research/oss-radar-07/repro/installed/recorded/custom-capture.json)\n\n[Runnable installed probe · Sep 8, 2026](https://github.com/gkoreli/blog/blob/dabd081506de2e0a8dba6778b4e43c0bb83d5c13/packages/blog/drafts/research/oss-radar-07/repro/installed/probe-installed.ts)\n\n[Trigger and citation research · Sep 8, 2026](https://github.com/gkoreli/blog/blob/dabd081506de2e0a8dba6778b4e43c0bb83d5c13/packages/blog/drafts/research/oss-radar-07/04-trigger-provenance-and-ai-citations.md)\n\n[Proposed measurement contract · Sep 8, 2026](https://github.com/gkoreli/blog/blob/dabd081506de2e0a8dba6778b4e43c0bb83d5c13/packages/blog/drafts/research/oss-radar-07/07-citation-measurement-contract.md)\n\n[Capture and export experiment · Sep 8, 2026 (PDT)](https://github.com/gkoreli/blog/blob/5fc2dc5dc40ad2397a78325e3a192485b44e8cb5/packages/blog/drafts/research/oss-radar-07/repro/capture/README.md)\n\n[Saved retry export · Sep 8, 2026 (PDT)](https://github.com/gkoreli/blog/blob/5fc2dc5dc40ad2397a78325e3a192485b44e8cb5/packages/blog/drafts/research/oss-radar-07/repro/capture/recorded/retry-cli-export.json)\n\n[Capture implementation · Sep 8, 2026 (PDT)](https://github.com/gkoreli/blog/blob/5fc2dc5dc40ad2397a78325e3a192485b44e8cb5/packages/blog/drafts/research/oss-radar-07/repro/capture/capture-provider.ts)\n\n[Saved full export with trace · Sep 8, 2026 (PDT)](https://github.com/gkoreli/blog/blob/5fc2dc5dc40ad2397a78325e3a192485b44e8cb5/packages/blog/drafts/research/oss-radar-07/repro/capture/recorded/full-cli-export.json)\n\n[Evaluator tracing implementation · Sep 8, 2026 (checked)](https://github.com/promptfoo/promptfoo/blob/89052308bce06f53645b1f189ada5ac9d1897347/src/tracing/evaluatorTracing.ts)\n\n[Promptfoo license · Sep 8, 2026 (checked)](https://github.com/promptfoo/promptfoo/blob/89052308bce06f53645b1f189ada5ac9d1897347/LICENSE)\n\n[Pinned project overview · Sep 8, 2026 (checked)](https://github.com/promptfoo/promptfoo/blob/89052308bce06f53645b1f189ada5ac9d1897347/README.md)\n\n[Founders' announcement · Mar 9, 2026](https://www.promptfoo.dev/blog/promptfoo-joining-openai/)\n\n[Red-team architecture · Sep 8, 2026 (checked)](https://github.com/promptfoo/promptfoo/blob/89052308bce06f53645b1f189ada5ac9d1897347/site/docs/red-team/architecture.md)\n\n[Red-team run implementation · Sep 8, 2026 (checked)](https://github.com/promptfoo/promptfoo/blob/89052308bce06f53645b1f189ada5ac9d1897347/src/redteam/shared.ts)\n\n[Scanner implementation · Sep 8, 2026 (checked)](https://github.com/promptfoo/promptfoo/blob/89052308bce06f53645b1f189ada5ac9d1897347/src/codeScan/scanner/index.ts)\n\n[Scanner action implementation · Sep 8, 2026 (checked)](https://github.com/promptfoo/promptfoo/blob/89052308bce06f53645b1f189ada5ac9d1897347/code-scan-action/src/main.ts)\n\n[Promptfoo 0.122.1 · Aug 26, 2026](https://github.com/promptfoo/promptfoo/releases/tag/0.122.1)\n\n[Promptfoo 0.122.2 · Aug 28, 2026](https://github.com/promptfoo/promptfoo/releases/tag/0.122.2)\n\n[Promptfoo 0.123.0 · Sep 10, 2026](https://github.com/promptfoo/promptfoo/releases/tag/0.123.0)\n\n[Scanner action 0.2.0 · Aug 28, 2026](https://github.com/promptfoo/promptfoo/releases/tag/code-scan-action-0.2.0)\n\n[Building a security scanner for LLM apps · Dec 16, 2025](https://www.promptfoo.dev/blog/building-a-security-scanner-for-llm-apps/)\n\n[Promptfoo about page · Sep 10, 2026 (checked, PDT)](https://www.promptfoo.dev/about/)\n\n[OpenAI acquisition announcement · Mar 9, 2026](https://openai.com/index/openai-to-acquire-promptfoo/)\n\n[Scanner action configuration · Sep 8, 2026 (checked)](https://github.com/promptfoo/promptfoo/blob/89052308bce06f53645b1f189ada5ac9d1897347/code-scan-action/action.yml)\n\n[Custom-provider documentation · Sep 8, 2026 (checked)](https://github.com/promptfoo/promptfoo/blob/89052308bce06f53645b1f189ada5ac9d1897347/site/docs/providers/custom-api.md)\n\n[OpenRouter provider implementation · Sep 8, 2026 (checked)](https://github.com/promptfoo/promptfoo/blob/89052308bce06f53645b1f189ada5ac9d1897347/src/providers/openrouter.ts)\n\n[Provider response contract · Sep 8, 2026 (checked)](https://github.com/promptfoo/promptfoo/blob/89052308bce06f53645b1f189ada5ac9d1897347/src/contracts/providers.ts)\n\n[Telemetry implementation · Sep 8, 2026 (checked)](https://github.com/promptfoo/promptfoo/blob/89052308bce06f53645b1f189ada5ac9d1897347/src/telemetry.ts)\n\n[Promptfoo issue #9968 · Sep 10, 2026 (checked, PDT)](https://github.com/promptfoo/promptfoo/issues/9968)\n\n[Bing AI Performance preview · Feb 10, 2026](https://blogs.bing.com/webmaster/February-2026/Introducing-AI-Performance-in-Bing-Webmaster-Tools-Public-Preview)\n\n[W3C PROV-DM · Apr 30, 2013](https://www.w3.org/TR/2013/REC-prov-dm-20130430/)\n\n[Web Bot Auth use-case draft, Appendix A.4 · Apr 1, 2026](https://datatracker.ietf.org/doc/html/draft-nottingham-webbotauth-use-cases-02#appendix-A.4)\n\n[HTTP Message Signatures for automated traffic, §4.6 · Sep 1, 2026](https://datatracker.ietf.org/doc/html/draft-ietf-webbotauth-httpsig-protocol-00#section-4.6)\n\n[Wallat et al., Correctness is not Faithfulness in RAG Attributions · Dec 23, 2024](https://arxiv.org/abs/2412.18004)\n\n[Gao et al., Enabling Large Language Models to Generate Text with Citations · Dec 2023](https://aclanthology.org/2023.emnlp-main.398/)\n\nDates are publication dates unless marked **checked**. Pinned Promptfoo code and documentation refer to the tested 0.122.2 release. Our artifacts record a synthetic-response experiment; the standards and studies supply concepts and methods, not measurements of this blog's citation rate.\n\n### Research record\n\nThe [worklist](https://github.com/gkoreli/blog/blob/main/packages/blog/drafts/research/oss-radar-07/00-worklist-index.md) links the experiments, research notes, and open work. The [verbatim shaping prompts](https://github.com/gkoreli/blog/blob/main/packages/blog/prompts/oss-radar-07-promptfoo.prompts.md) are preserved for this issue at the author's request. The methods record the runtimes, dependency lock, scripts, results, and failed setup attempts. Codex executed the experiments for this article; Goga supplied the workload and editorial direction. Research-session token totals and hands-on time have not been measured.\n\n## Glossary\n\n| Term / Claim | Source | Date | \n|---|---|---|\n| **Provider adapter:** code that calls a model or application and returns the fields Promptfoo evaluates. | [Custom-provider interface](https://github.com/promptfoo/promptfoo/blob/89052308bce06f53645b1f189ada5ac9d1897347/site/docs/providers/custom-api.md) | Sep 8, 2026 (checked) | \n| **Evaluation:** running defined cases and checking the returned answer or behavior. | [Promptfoo overview](https://github.com/promptfoo/promptfoo/blob/89052308bce06f53645b1f189ada5ac9d1897347/README.md) | Sep 8, 2026 (checked) | \n| **Red teaming:** testing a target with generated adversarial cases and attack strategies. | [Red-team architecture](https://github.com/promptfoo/promptfoo/blob/89052308bce06f53645b1f189ada5ac9d1897347/site/docs/red-team/architecture.md) | Sep 8, 2026 (checked) | \n| **Raw response:** the response text retained by our capture provider before selecting answer and citation fields. | [Tested capture implementation](https://github.com/gkoreli/blog/blob/dabd081506de2e0a8dba6778b4e43c0bb83d5c13/packages/blog/drafts/research/oss-radar-07/repro/installed/probe-installed.ts) | Sep 8, 2026 | \n| **Citation analytics:** counts and trends over observed citations, with a declared coverage boundary. | [Bing AI Performance](https://blogs.bing.com/webmaster/February-2026/Introducing-AI-Performance-in-Bing-Webmaster-Tools-Public-Preview) | Feb 10, 2026 | \n| **Source list / answer citation:** provider-returned sources and references attached to an answer are distinct fields; a source entry alone need not be an answer citation. | [Our proposed measurement contract](https://github.com/gkoreli/blog/blob/dabd081506de2e0a8dba6778b4e43c0bb83d5c13/packages/blog/drafts/research/oss-radar-07/07-citation-measurement-contract.md) | Sep 8, 2026 | \n| **Trigger / responsibility:** the event starting an activity differs from the agent responsible for it. | [W3C PROV-DM](https://www.w3.org/TR/2013/REC-prov-dm-20130430/) | Apr 30, 2013 | \n| **Verified request identity:** a signed request can identify its key holder; it does not establish a fresh human action or a citation. | [Bot-signature draft, §4.6](https://datatracker.ietf.org/doc/html/draft-ietf-webbotauth-httpsig-protocol-00#section-4.6) | Sep 1, 2026 | \n| **Citation support:** whether a cited source supports the associated claim. | [ALCE](https://aclanthology.org/2023.emnlp-main.398/) | Dec 2023 | \n| **Citation faithfulness:** whether the model actually relied on the cited source. | [Wallat et al.](https://arxiv.org/abs/2412.18004) | Dec 23, 2024 |", "url": "https://wpnews.pro/news/oss-radar-07-can-promptfoo-preserve-the-evidence-behind-an-ai-answer", "canonical_source": "https://gkoreli.com/oss-radar-07-promptfoo", "published_at": "2026-09-10 00:00:00+00:00", "updated_at": "2026-09-11 03:26:49.746288+00:00", "lang": "en", "topics": ["ai-tools", "ai-safety", "developer-tools"], "entities": ["Promptfoo", "OpenRouter", "OSS Radar", "Ian Webster", "Michael D'Angelo", "OpenAI", "GitHub"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/oss-radar-07-can-promptfoo-preserve-the-evidence-behind-an-ai-answer", "markdown": "https://wpnews.pro/news/oss-radar-07-can-promptfoo-preserve-the-evidence-behind-an-ai-answer.md", "text": "https://wpnews.pro/news/oss-radar-07-can-promptfoo-preserve-the-evidence-behind-an-ai-answer.txt", "jsonld": "https://wpnews.pro/news/oss-radar-07-can-promptfoo-preserve-the-evidence-behind-an-ai-answer.jsonld"}}