---
type: analysis
slug: safety-scrutiny-depends-on-what-outside-evaluators-can-actually-see
title: "Safety scrutiny depends on what outside evaluators can actually see"
description: "Three former OpenAI safety researchers call for durable evaluator access and clearer collaboration rules. The practical distinction is between permission to test a model and enough scoped access to inspect the nonpublic information needed to interpret what the test shows."
published_at: 2026-10-10T14:00:00.000Z
topics: ["openai","workflow"]
source_urls: ["https://mikitabalesni.com/letter/letter.pdf","https://openai.com/index/hugging-face-incident-and-the-road-ahead","https://www.wsj.com/tech/ai/openai-chatgpt-model-release-cancel-safety-5a2f9f42?mod=e2tw","https://www.forbes.com/sites/fionariley/2026/10/01/openai-reportedly-fires-3-researches-over-allegedly-mishandling-confidential-information","https://www.wsj.com/tech/ai/openai-chatgpt-model-release-cancel-safety-5a2f9f42?amp%3Breflink=article_copyURL_share&st=C9iWF6"]
author: "Andrey Reshetnikov"
author_profile: https://newruntime.com/owner-profile.md
---
# Safety scrutiny depends on what outside evaluators can actually see

Three former OpenAI safety researchers call for durable evaluator access and clearer collaboration rules. The practical distinction is between permission to test a model and enough scoped access to inspect the nonpublic information needed to interpret what the test shows.

## Summary

Three former OpenAI safety researchers—Tomek Korbak, Jasmine Wang, and Mikita Balesni—wrote to the company’s safety oversight bodies after their dismissals. They recommend preserving independent evaluator access, maintaining monitorability of frontier models, and clarifying procedures for external collaboration. They allege that uncertainty about permitted conduct can deter candid reporting and outside work; the supplied primary source does not independently verify the circumstances of their dismissals or OpenAI’s policies.

## What happened

The letter frames external collaboration and open debate as operational parts of safety work, not simply employment concerns. Its authors say uncertainty about what employees may share or do can discourage escalation and collaboration, and ask OpenAI’s Safety and Security Committee, Safety Advisory Group, and Mission Advisory Council to act. The concrete requests are to preserve independent evaluator access, maintain monitorability, and clarify external-collaboration procedures. The source establishes that the former researchers made these claims and recommendations, not that evaluator access has been curtailed or that colleagues have stopped reporting concerns.

## What the archive adds

The dossier says OpenAI’s guidance on third-party assessments calls for independent scrutiny of safety cases, safeguards, and capability evaluations. It also summarizes OpenAI’s view that evaluations of tool-using, stateful, multi-step models can miss behavior if the test setup is inadequate. A useful distinction follows: ordinary permission to evaluate is not necessarily the same as deeper, limited access to nonpublic information needed to interpret behavior. The dossier does not specify what access OpenAI currently grants in this case, so this is a question to verify, not an established policy description.

## Trend signal

As frontier systems use tools and operate across multi-step workflows, an evaluator’s conclusions depend on more than whether a model can be queried. The scope of access and the ability to understand the tested conditions affect what scrutiny can establish. At the same time, sharing nonpublic information raises legitimate confidentiality concerns. The letter makes that tension immediate: safety governance must define a workable boundary for outside scrutiny and internal escalation, rather than treating either access or secrecy as an all-or-nothing choice.

## Why now

According to the primary letter, the authors connect their request to recent dismissals and their concern that uncertainty can deter escalation and external collaboration; these are their claims, not independently verified effects. According to the dossier’s summary of OpenAI’s third-party assessment guidance, independent scrutiny is considered important for frontier safety cases and capability evaluations. The dossier also says OpenAI has noted that tool use, model state, and multi-step workflows can complicate evaluation, making the adequacy and scope of evaluator access a concrete issue.

## Connections to other events

The second-order risk is a thinner feedback loop: if staff are unsure how to escalate concerns, or evaluators can test only a surface behavior without enough scoped context to interpret it, weaknesses may be harder to identify before deployment decisions. That is an inference from the authors’ argument, not a demonstrated outcome. The key test is therefore not a general assurance that outside evaluation exists, but a clear account of what evaluators can examine, what remains confidential, and how findings reach decision-makers.

## Why it matters

For buyers and builders, a reference to independent evaluation is not by itself evidence of meaningful scrutiny. Ask what behaviors and workflows were tested, what access the evaluators received, and how confidentiality boundaries shaped the work. For OpenAI, explicit procedures could make it easier to preserve both protected information and useful independent challenge. For safety practitioners, the letter points to the value of making escalation and disclosure rules legible before a concern arises.

## What happens next

- **OpenAI will publicly address the letter or clarify evaluator access or external-collaboration procedures.** (3 months, 55%). Watch for: A public response, policy update, or oversight-body statement that specifies access scope or collaboration procedures.
- **Any publicly observable follow-through will describe evaluator scope or process more readily than internal dissent or reporting outcomes.** (6 months, 60%). Watch for: A public protocol or evaluation-access description appears, without a comparable public measure of internal reporting.

## Limits and uncertainty

The primary source is a letter by the former researchers; it supports their recommendations and allegations but does not independently establish the disputed circumstances of their dismissals, internal policies, or current evaluator access. The dossier summarizes OpenAI materials but does not provide the access terms or a response to this letter. The inference about a weakened feedback loop is conditional: existing procedures may already address the concerns, or access may be adequate in practice despite the authors’ uncertainty.

## Skill to build now

Practice access-and-disclosure planning. This week, choose one model behavior to evaluate and write a brief plan specifying the test workflow, the access needed to interpret results, what information is confidential, who can authorize access, and the approved escalation and external-sharing route. Keep observed behavior separate from interpretation.

## Reader takeaway

When evaluating a safety claim, ask for the scope of third-party access: what the evaluator could test, whether limited nonpublic context was available where needed, and how findings were handled under confidentiality rules. If you work in safety, draft a one-page access-and-disclosure plan for a sample evaluation before sharing findings: specify the evidence, access required, confidential elements, escalation route, and permitted external disclosure.

## Open question

What minimum, auditable access to nonpublic context can independent evaluators receive to make findings useful without exposing information that should remain confidential?

## Sources

- [OpenAI Cannot Make AI Safe on Its Own (6 minute read)](https://mikitabalesni.com/letter/letter.pdf)

- [The Hugging Face incident and the road ahead | OpenAI](https://openai.com/index/hugging-face-incident-and-the-road-ahead)

- [Exclusive | OpenAI Scraps Release of GPT-6.1 Astra Model Over Safety Concerns - WSJ](https://www.wsj.com/tech/ai/openai-chatgpt-model-release-cancel-safety-5a2f9f42?mod=e2tw)

- [OpenAI Fires 3 Safety Researchers, Wall Street Journal Reports](https://www.forbes.com/sites/fionariley/2026/10/01/openai-reportedly-fires-3-researches-over-allegedly-mishandling-confidential-information)

- [Exclusive | OpenAI Scraps Release of New AI Model Over Safety Concerns - WSJ](https://www.wsj.com/tech/ai/openai-chatgpt-model-release-cancel-safety-5a2f9f42?amp%3Breflink=article_copyURL_share&st=C9iWF6)

- [Priorities and principles for effective third party assessments](https://openai.com/index/priorities-principles-third-party-assessments/)

- [A shared playbook for trustworthy third party evaluations](https://openai.com/index/trustworthy-third-party-evaluations-foundations/)

- [Early access for safety testing](https://openai.com/index/early-access-for-safety-testing/)

- [Strengthening our safety ecosystem with external testing — OpenAI](https://openai.com/index/strengthening-safety-with-external-testing/)

- [External Researcher Access Program; DeepMind’s external evaluation partnerships](https://support.claude.com/en/articles/9125743-what-is-the-external-researcher-access-program)

- [Developing a Framework for Secure Third-Party Access to Frontier AI — RUSI](https://static.rusi.org/secure-third-party-access-to-frontier-ai-2026-final.pdf)

- [OpenAI research leadership’s response to the researchers’ letter](https://www.theverge.com/ai-artificial-intelligence/2026/10/9/openai-fires-safety-researchers-letter-response)

- [OpenAI Raising Concerns Policy](https://cdn.openai.com/policies/openai-raising-concerns-policy.pdf)

Author: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Openai](https://newruntime.com/topics/openai.md): Explore the openai topic hub.
- topic hub: [Workflow](https://newruntime.com/topics/workflow.md): Explore the workflow topic hub.
- collection: [Analysis](https://newruntime.com/analysis.json): Open the latest authorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)