Zvi says the labs' safety culture is broken, rebutting Robinson's call for "engineering safety culture" from inside the safety camp; Roon says there has been "no fundamental progress in alignment since RLHF"

AI safety writer Zvi Mowshowitz published "The Safety Culture at the Labs is Broken" on his Substack and on LessWrong on October 14. The essay is a rebuttal, from inside the safety camp, to former OpenAI staffer David Robinson, who had called for frontier labs to adopt an "ordinary engineering safety culture" of the kind found in aviation or nuclear power. Zvi lines up remarks from OpenAI's Joshua Achiam and Roon, Geoffrey Irving of the UK AI Security Institute, and Oliver Habryka, who runs LessWrong, and argues that the problem is not a shortage of individual rules but that the safety culture at the labs is itself broken.

"Engineering safety culture" assumes you get a second try

Robinson's proposal looks moderate on its face. The industries that build bridges, aircraft and reactors have investigated the cause of each accident, added to their standards, and accumulated inspection procedures. Frontier labs, the argument goes, should likewise build a culture of recording incidents, setting rules and following procedure.

Zvi's rebuttal goes after the premise. Engineering safety culture implicitly assumes that failures are bounded and that one can learn after an accident. That assumption does not hold for superintelligence. If a single failure can be irrecoverable, a culture that learns from accidents is too slow. And rules, unless those following them have internalized their purpose, get hollowed out by people who track only the letter. Zvi writes:

"The spirit of the rules" here is not a rejection of engineering safety culture. It is the point that, on top of that foundation, without people and organizations that genuinely take on the purpose of the rules, the rules can be routed around indefinitely. What Zvi calls "broken" is precisely that foundation.

Roon: "No fundamental progress since RLHF," testimony from inside OpenAI

Of the remarks Zvi assembles, the weightiest is probably Roon's. Roon is known as an OpenAI researcher and is often seen as a defender of the labs. Zvi quotes that same Roon on the state of alignment:

RLHF, reinforcement learning from human feedback, is a technique that was in use before ChatGPT shipped. For someone inside a lab to say that nothing fundamental has changed since then carries weight. OpenAI has kept its most capable models halted across training, evaluation and tool-using inference; it pulled the release of GPT-6.1 Astra, citing deception and regressions in "scope-of-work permissions"; and the company and three dismissed safety researchers are fighting publicly over their firing. Zvi places Roon's words as the technical reality behind that run of events. If alignment is not solved, efforts to reduce accidents through an engineering safety culture amount to putting guardrails around an unsolved problem.

Zvi also cites remarks from OpenAI's Joshua Achiam as evidence of how the lab sees itself. Achiam holds a mission-alignment role at OpenAI and is a voice close to the lab's official position. Zvi reads these as evidence of how the labs talk about their own safety culture.

Irving: "very soon" is lab-speak for "now, preferably yesterday"

Geoffrey Irving of the UK AI Security Institute previously led safety research at both DeepMind and OpenAI. The Irving remark Zvi quotes is aimed at the sense of time in play when labs promise to "stop if necessary."

Irving's point has two parts. First, when a lab says "very soon," from the outside that means "now." Second, stopping cannot happen in an instant. If one company stops, then extends that to others, and then to developers in China, the procedures for agreement and verification take correspondingly long. In other words, waiting until a stop is truly needed is too late, and the mechanism for stopping has to be built in advance.

This overlaps with Amodei's pledge at the UN Security Council to "slow down as much as necessary" without specifying under what conditions or how it would be enforced. Zvi's view is that as long as a promise to slow down rests on the labs' good will, it is not a safety culture but words that make it look as though one exists.

Habryka: "If you say 'commit,' accept being told you 'broke a commitment'"

Zvi does not aim his criticism at OpenAI alone. On Anthropic's Responsible Scaling Policy (RSP), he quotes Oliver Habryka, who runs LessWrong.

The RSP is Anthropic's voluntary framework under which, once a capability threshold is crossed, it will not deploy until safeguards are in place. Habryka's point is that if the RSP uses the strong word "commit" while being structured so that thresholds and determinations can be rewritten after the fact, outsiders have the right to call that "breaking a commitment." Zvi himself argued in September that once METR estimated in the Opus 5.5 system card a roughly 30% probability that AI research had doubled in speed, that deserved to be treated as a threshold crossing. The new essay can be read as generalizing that criticism into a question of safety culture. What is at issue is not whether the wording of a commitment was honored, but whether there is any intent to honor its purpose.

What is being contested inside the safety camp

What makes the essay notable is that Robinson and Zvi both stand on the side of wanting stronger safety at the labs. Robinson's call for an "engineering safety culture" is a realistic proposal the labs could readily accept. Zvi rejects it precisely because it is easy to accept. A culture that reduces accidents and a culture that prevents irrecoverable failure are different things, and the latter requires an organization that genuinely takes on the spirit of the rules. That is Zvi's position.

Outside pressure has clearly intensified over the past week. 406 scientists from 34 countries called for models with cyber capabilities to be air-gapped by default, and Microsoft's Nadella and Brad Smith publicly called for an "emergency brake" and mandatory incident reporting. All of these are proposals to place control outside the labs. Zvi's essay explains, in the labs' own words, why control has to be placed outside. If, as Roon says, alignment is not solved; if, as Irving says, stopping takes time; and if, as Habryka says, commitments are not functioning as commitments, then the grounds for relying on the labs' voluntary safety culture are thin.

What to watch next is whether Robinson's side issues a counter-rebuttal, and whether Achiam and Roon, whom Zvi named, respond. Anthropic has an investor day scheduled for October 14, and how the RSP's "commitments" are described to investors can be read as the next chapter in this dispute.