When AI Evidence Is Redacted, Vendor Approval Needs a Disclosure Gap Register

An independent evaluator can see convincing evidence that customers still cannot inspect. That does not make the assessment useless, and it does not automatically make the vendor untrustworthy. It does create a disclosure gap that procurement and operations must manage deliberately.

The November 5 assessment of AI research acceleration at Anthropic offers a useful example. METR received unusually deep access, published a quantified conclusion, and also said Anthropic redacted much of the evidence most important to that conclusion. For an operating business, the lesson is broader than frontier-model research: an AI vendor’s risk report is an input to approval, not the approval itself.

A practical vendor review should record which claims are independently reproducible, which were reviewed privately, which remain vendor assertions, and which evidence is unavailable. Then it should connect those gaps to permissions, testing, human oversight, monitoring, and an exit path.

What is documented

Anthropic announced in October that it had given six METR employees company hardware, office space, and access roughly comparable to its own risk-assessment teams. Anthropic also described the publication arrangement: METR could publish without an Anthropic veto, while Anthropic could require removal of material within predefined confidential categories.

After an 11-week review, METR reported on November 5 that Anthropic’s internal benchmark scores had been improving at an estimated 1.9 times their earlier rate. METR also reported substantial uncertainty and said the result applied to well-specified tasks—not every form of useful work.

Documented access

METR reviewed internal metrics, interviews, surveys, unreleased-model performance, and other nonpublic information.

Documented redaction

METR said Anthropic redacted a majority of the evidence it found most informative for its conclusions.

Documented uncertainty

METR said some redacted evidence had multiple plausible interpretations and that public readers must rely more heavily on its judgment than would be ideal.

Facts, vendor positions, and unresolved questions

Fact

Anthropic created an unusually high-access evaluation arrangement and retained defined redaction rights. METR disclosed both the access and the limits.

Vendor position

Anthropic said the redactions protected confidential information that could accelerate competitors. That is a stated rationale, not independently testable proof that every redaction was necessary.

Evaluator assessment

METR concluded that capability progress had accelerated and that AI assistance likely explained much of it. The organization explicitly characterized parts of that conclusion as uncertain.

Still unresolved

Public readers cannot independently examine much of the underlying evidence, determine how the result generalizes, or fully reconcile the public and private risk views.

Reuters, Associated Press, and The Verge reported on the findings and the limits created by redacted evidence.

Why this matters to an operating business

Most buyers will never receive employee-like access to an AI provider. They may receive a system card, security questionnaire, audit report, model documentation, contract language, and selected performance results. Those materials can be useful while still leaving important questions unanswered.

The risk becomes material when an AI system can touch customer commitments, product data, inventory, quoting, payments, production records, or operational settings. In those workflows, “the vendor says it is safe” and “an evaluator reviewed private evidence” are not substitutes for controls inside the buyer’s own environment.

The business should approve a bounded use case—not a vendor’s entire platform. This connects vendor review with agent cost and access governance, because evidence gaps should reduce the authority granted to the system.

Create a disclosure gap register

A disclosure gap register is a short operating record of what the business needs to know, what evidence is available, what remains unavailable, and how the remaining uncertainty is controlled.

Claim

Write the exact assertion being evaluated: accuracy, security, data handling, uptime, monitoring, model-change behavior, or another decision-relevant property.

Evidence tier

Mark whether the claim is independently reproducible, privately assessed by a third party, vendor-measured, contractually warranted, or only planned.

Missing evidence

Record the dataset, method, failure cases, incident history, or control details that were not supplied. Do not translate “confidential” into “verified.”

Business consequence

Connect the gap to a real outcome: wrong price, bad order, exposed data, production interruption, customer harm, or unrecoverable record change.

Compensating control

Specify the validation, permission limit, human approval, monitoring rule, transaction cap, or manual fallback that keeps exposure acceptable.

Risk owner

Name the person who can accept, reject, or require remediation. “IT,” “the committee,” and “the vendor” are not accountable owners.

Use an evidence ladder instead of a yes-or-no checklist

Vendor due diligence is stronger when evidence quality changes the approval decision:

  • Reproducible evidence: the business or an independent party can repeat the test using disclosed methods and representative data.
  • Independent but nonpublic evidence: a credible evaluator had access, but the buyer cannot inspect the underlying material. Treat the finding as useful with residual uncertainty.
  • Vendor evidence: the provider supplies results, methods, or attestations. Validate the parts that matter in the buyer’s workflow.
  • Contractual commitment: service levels, incident notice, data-use restrictions, deletion, change notice, audit rights, and exit assistance are enforceable.
  • Roadmap statement: a planned control is not a present control and should not support production authority.

This ladder avoids two bad extremes: dismissing all vendor evidence and accepting every polished report as proof.

Turn uncertainty into operating boundaries

Least privilege

Start read-only or recommendation-only. Grant write access by system, record type, action, and transaction threshold only after evidence and testing justify it.

Deterministic validation

Check required fields, formats, units, price floors, inventory, account status, and cross-system consistency outside the model.

Human approval

Require a named person for irreversible, expensive, regulated, safety-relevant, or customer-binding actions.

Auditability

Log the model and version, prompt, sources, tool calls, validations, approvals, output, and downstream result.

Monitoring

Watch exception rates, corrections, approval overrides, cost, latency, prohibited actions, and business outcomes—not just uptime.

Exit path

Preserve data export, configuration, rollback, replacement routing, and a tested manual procedure before dependency becomes critical.

Apply the standard to the workflow, not the brand

A provider may be acceptable for drafting internal summaries and unacceptable for releasing purchase orders. The same model may be suitable for product-description suggestions but not for changing price or availability without deterministic checks.

For RFQ-to-quote automation, the business can let AI classify requirements and prepare a draft while independently validating part identity, units, quantity, lead time, margin floors, customer terms, and inventory. A named approver owns the quote until the evidence supports narrower automation.

This workflow-level approach keeps adoption moving. It does not require perfect transparency before any use; it requires the authority granted to the system to match the evidence available.

Review disclosure gaps when the system changes

Vendor approval should not be permanent. Reopen the register when the provider changes a model, hosting region, subprocessors, training-data policy, security architecture, tool-use behavior, pricing basis, or retirement schedule.

Connect that review to the model release-control process: regression testing, shadow comparison, canary rollout, tighter early monitoring, rollback, and manual fallback. If a new version reduces the evidence available or changes the consequence profile, approval should narrow until the gap is resolved.

Predefine the incident path as well. The AI incident-response workflow should identify who can suspend access, preserve evidence, notify affected owners, restore the prior process, and decide when service resumes.

Constructive adoption requires honest uncertainty

The goal is not complete disclosure from every vendor. Some information will legitimately remain confidential. The goal is to stop treating unavailable evidence as if it did not affect the decision.

A credible approval records what is known, who examined it, what cannot be verified, which controls compensate for the gap, and who owns the remaining risk.

That approach lets an operating business use better AI without outsourcing its judgment. Bridge Road helps connect governance, systems, operations, commerce, and automation around real workflows. If AI can create a consequential business action, start with the evidence and authority it actually has.

Sources and scope

Scope note: this article uses one high-access evaluation to derive a general vendor-risk workflow. It does not allege that withheld information is false or that the findings apply to every AI provider.