Thesis: An external assessment is more useful when readers can see the constraints and incentives that shaped its evidence, not only its conclusion.
Mechanism: METR’s stated disclosure policies make scope, access restrictions, redactions, conflicts, and other constraints visible alongside findings, allowing readers to interpret evidence rather than relying solely on conclusions. METR says it will describe project scope and duration, access and data limits, consequential redactions, and formal or informal constraints that could materially affect conclusions. It also points to conflict-of-interest disclosures, financial and operational independence, and a redaction-summary arrangement that can preserve notice when a company will not permit adequate description of removed material. These commitments make limitations legible alongside findings rather than leaving readers to infer them.
Why now: METR says expectations are growing for voluntary engagements to provide external safety assurance, while many relevant risk claims remain open scientific questions. The post therefore argues for public evidence and iterative reporting rather than closed-door judgments. First implication: Readers and policymakers can compare assessments more responsibly when process, access, and redactions are reported as part of the evidence. Evidence boundary: This is METR’s account of its intended approach and examples of prior practice; it does not independently establish how consistently future projects will implement the commitments or resolve dependence on company participation.
What to watch: Future METR reports should show whether scope, access limits, redactions, and constraints are disclosed in enough detail for outside experts to evaluate the underlying evidence. This is a distinct analytical record because no prior New Runtime analysis was supplied for this methodology statement.