Our approach to working with AI companies

METR published a blog post on Oct 14, 2026, detailing its approach to working with AI companies on evaluations, assessments, and investigations intended to give the public accurate, timely, and independent evidence about AI loss-of-control risk. METR said it retains the right to unilaterally and prominently attach a "redaction summary" to its reports, and cited its May 2026 pilot Frontier Risk Report, which grew out of work with Anthropic, Google, Meta and OpenAI, as an example of the approach. The post also referenced the OpenAI / Hugging Face incident report, in which investigators devoted a section to the scope, timeline, access, and limitations of their investigation and used inline text and footnotes to explain where information was redacted.

This blog post explains how we approach working with AI companies on projects that the public might rely on to understand risks. It was published on Oct 14, 2026, and we expect to update it over time. At METR, we gather and publish evidence about whether humans could lose control of AI agents on a large scale. We work with AI companies to conduct evaluations, assessments, and investigations,<sup 1 fn:1 </sup and to share this evidence publicly. As we have seen increasing expectations for these kinds of voluntary engagements to provide external ‘safety assurance’, we want to explain our perspective. In this post, we clarify our overall goals our-motivations for this work, our approach our-approach , and what our approach looks like in practice in-practice . Our goal is to give the public accurate, timely, and independent evidence about AI loss-of-control risk, so that they along with policymakers and developers can make their own informed decisions . Our primary approach is collecting high-quality, concrete evidence and publishing what we find. Beyond this, we may also work with companies or regulators to share our comments on unpublished sensitive information,<sup 2 fn:2 </sup to review risk reports, or to provide an independent judgment on claims about risk. However, many of the relevant claims involve open scientific questions so can’t be straightforwardly verified. We see it as our role to collect evidence for the public; we do not intend to decide how companies should develop or deploy their agents. Decisions about frontier AI will likely be messy, high-stakes judgment calls relying on uncertain science, which should not be made behind closed doors, much less within any one organization. On the contrary, we think that the current situation requires open discussion about both what the risk levels are given the evidence and what risk levels are acceptable. To correctly interpret our outputs, the public needs to understand the processes, constraints, and incentives that produced those outputs. For instance, if important evidence of risk was redacted by an AI developer that METR collaborates with during an assessment, the reader should be informed of that. METR currently takes the following measures for work we do with AI companies bearing directly on the public’s understanding of risks from AI development such as pre-deployment testing, risk report reviews, and risk assessment exercises : We also aim to meet the following principles, intended to minimize the amount the public needs to defer to us: In the next section, we give more detail on how this approach has worked historically. We arrived at the approach above after several years of working with AI companies. For example, we documented<sup 4 fn:4 </sup a similar approach this May in our pilot Frontier Risk Report https://metr.org/risk-report-feb-mar-2026.pdf , which grew out of work with Anthropic, Google, Meta and OpenAI. In addition to our own experience, we also draw on the work of other https://www.apolloresearch.ai/blog/principles-for-embedded-evaluations organizations https://aievaluatorforum.org/initiatives/evaluation-transparency-letter in the AI evaluation ecosystem. Our typical practice is to disclose limitations, such as our agreement terms, the process our publications go through, and redactions companies make. For example, in the OpenAI / Hugging Face incident report, the investigators devoted a section https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/ investigation-process-and-limitations of the report to explaining the scope, timeline, access, and limitations of their investigation. In the same report, the investigators used inline text and footnotes to explain where information was redacted and in some cases why. One arrangement that we have found works well is for METR to retain the right to unilaterally and prominently attach a “redaction summary” to our reports. This summary is meant to indicate whether a company redacted information that was important to our overall conclusions but did not permit us to adequately describe what was redacted. This enables redaction dialogue between AI developers and METR to focus on the question of whether a redaction would change our report’s overall conclusion, while retaining some flexibility in wording and allowing rephrasing. Beyond formal control over our publications, we have also tried to discuss softer, informal pressures that shape the evidence that the public receives. Our website, our recent funding update https://metr.org/blog/2026-08-14-funding-update/ , and our COI policy https://metr.org/coi-policy.pdf describe some of the measures we have taken to establish independence and address conflicts of interest. While we recognize that other groups can usefully make different independence tradeoffs, to date we have chosen not to accept payment or donations from frontier AI companies, and have declined offers of funding when an AI company would have influence over us, e.g. funding from an AI company executive or foundation. Though we accept a large amount of free model access, we ensure that we can afford to pay for any free tokens we receive without materially affecting our operations. Even with these measures, it has proved challenging to ensure that we have adequate independence to collect and publish evidence under the current voluntary regime. We remain dependent on the companies we work with to voluntarily engage, provide access, and permit us to publish. In many instances, we rely on information that we cannot or do not verify, which might be filtered adversarially. Even with transparency about limitations, it is possible for a company to use its rights to withhold access or redact to exert control over what evidence the public receives from us. This is an excerpt from a METR policy, current as of October 2026. In any new engagements with companies that could be interpreted as providing public assurance, we try to ensure: We have found this to be a tricky line to walk: companies understandably want to protect sensitive information they share with us, but this also means they want the final say on what we can and cannot publish. Moreover, the standard legal agreements that companies use can cut against process transparency. For example, sometimes non-disclosure agreements will treat the existence of the NDA itself as confidential. We’re continuing to iterate on how to implement this in our work with companies. Not all of our previous or in-flight work meets this bar, but our current best practices include: In general, these policies should be applied whenever METR’s outputs, brand, or participation in an engagement are being presented to some audience as evidence about the level of risk. This includes implicitly. The following are other forms of process transparency that we currently try to uphold, or aim to uphold in the future, but have not yet formalized: In new contracts we’re also starting to include terms that protect our ability to disclose our findings to the boards and employees of the AI companies we work with, but we don’t have established best practices on this point. This policy does not prevent us from: We distinguish between AI evaluations, assessments, investigations, and audits, following the taxonomy from PACT AI’s article “An AI Evaluator Is Not an Auditor” https://pactai.org/resources-news/an-ai-evaluator-is-not-an-auditor . ↩ fnref:1 AI companies keep many aspects of their work confidential, and third party organizations like METR can with permission access and interpret this confidential information. For example, AI companies may be unwilling to release all the logs from a serious breach to the public, but willing to release these logs to METR for external review. ↩ fnref:2 We have sometimes called this “meta-transparency”. ↩ fnref:3 The pilot Frontier Risk Report has a section on the process that produced our report and uses the AI Evaluator Forum’s AEF-1 https://aievaluatorforum.org/initiatives/minimum-operating-conditions standard to document the access, contractual terms, and other relevant process details. We intend to use this AEF-1 disclosure template more regularly in future reports. ↩ fnref:4 We have already formalized a and b in existing contract language. c is permitted by default but we may explicitly carve this out going forward. c includes, for instance, a subjective sense of being pressured, a need to sustain commitment to ongoing engagement from the developer, uncertainty that the developer will engage in the future, restrictions on communication with researchers at the developer, lots of legal slowdown, among other things. ↩ fnref:5 Ideally, this includes the right to describe whether we think any redactions or other edits are appropriate and whether they change a conclusion. ↩ fnref:6 In theory, this could include signing a confidential non-disclosure agreement if this would not affect any of our existing affordances to comment on things and was not part of an engagement that was going to be presented to any audience as oversight or evidence about risk. ↩ fnref:7