"Don't stop alone": safety camp pushes back on Scott Alexander

[Update, November 2, 2026, 5:55 JST] We had the arguments of Kokotajlo and TsviBT reversed, and have corrected them as follows. (1) It was TsviBT who argued that a lab can stay at the frontier without releasing its most advanced models, and Kokotajlo who argued that a single company stopping would be a costly signal (both in LessWrong comments on November 1). We swapped the names in the two headed sections and in the final section, and corrected the key points. (2) "Against SURA" is Scott Alexander's own post; the two replied in its comments. (3) We added what SURA stands for and brief descriptions of Scott Alexander and Kokotajlo.

Should a company that puts AI safety first stop on its own, without waiting for anyone else? Since Irving called for a halt "right now," that question has split the safety camp from within.

Scott Alexander, a well-known American blogger, started it. He argued that if a safety-focused company stops on its own, the race itself goes on, and the only result is that ethical people lose their seat at the negotiating table. He made the case in "Against SURA", posted to LessWrong on October 31. SURA stands for "Stop Unilaterally + Raise Awareness," Alexander's label for the position he criticizes. In the comments, Daniel Kokotajlo, a former OpenAI researcher, and TsviBT pushed back. They made two points. First, a lab can stay at the front without releasing its most advanced models. Second, a single company stopping on its own would be a costly signal that could change the public debate. The debate has also spread on X, including through a post by journalist Sigal Samuel.

This is not a fight between outside critics demanding regulation and people defending the labs. Everyone involved thinks AI is dangerous. What they disagree on is tactics: how to stop.

Alexander: stop, and someone else takes your seat

Alexander's argument is about negotiation and influence. Frontier development today is a race among a handful of companies, and if one drops out, the rest keep going. A company that stops loses its standing to speak as someone holding a frontier model. In talks with governments, in setting industry standards, and wherever evaluation methods get decided, its seat passes to a company that kept going.

That would leave the people who take safety most seriously outside the room at the moments that matter most. Stopping alone keeps one's conscience clean, but it does nothing to reduce risk to the world. If anything, the people willing to hit the brakes end up letting go of the wheel. That is the heart of Alexander's argument.

TsviBT: stay in front, but don't release

TsviBT rejected the either-or choice behind Alexander's argument. The choice is not just between stopping and racing ahead. There is a third path: keep doing frontier research, keep the internal models, and stay at the front of the technology, but stop releasing the most advanced models to the outside world.

On that path, a lab keeps most of its standing at the table. It can still speak to governments and rivals as the party that knows best what the technology can do. At the same time, it no longer pushes up the capability ceiling in the open market, since the biggest driver of the race is the models that rivals release. If "stopping" means stopping releases rather than stopping development, the counterargument goes, the cost Alexander describes — losing a seat at the table — becomes far smaller.

This line already matches some of what is happening. Anthropic has not made its Mythos-class cyber capabilities generally available, giving them only to vetted defenders. Other labs are also giving their most powerful versions only to selected organizations. Staying in front without releasing everything is, to some extent, already happening in practice.

Kokotajlo: the cost is what makes it a signal

Kokotajlo focused on stopping as a signal. Safety-focused labs have been saying for years that the technology is dangerous. Because they keep developing it anyway, many people brush off those warnings as marketing or self-serving cover.

That changes if a company actually stops. It would give up revenue, valuation, and talent. Because the move costs so much, it shows a seriousness that words alone cannot. The shift in the premises of the public debate might matter more than the influence the company loses at the table. Talk of regulation or an international agreement, for example, could move from fantasy to a real option. Kokotajlo's point is that Alexander should count an effect he leaves out.

Why the split, and why now

This dispute is more than abstract because stopping is already something the labs write about in their own filings. In its S-1 filed on October 26, Anthropic said it had delayed pre-training of a new frontier foundation model by more than a month to strengthen protections for its weights. It also said it might delay or pause training if it lacked confidence in its safeguards. In an October 23 document analyzing the Mythos 5 incident in the UK AISI's evaluation, it said capability gains should slow down to match the pace of safety progress.

Within the safety camp, Kulveit had already urged researchers to "quit the labs." Irving's call to "stop now" went one step further and asked the companies themselves to stop. Alexander's argument was a warning aimed at both: stopping alone is a bad move. Kokotajlo and TsviBT have set out to knock that warning down. People inside the labs and people watching from outside share the same sense of alarm and are reaching opposite conclusions.

What lies beyond the split

Each argument rests on assumptions that have not been tested. Alexander's depends on how much worse the companies that keep going would do in place of ethical people at the table. TsviBT's third path depends on whether a company preparing to go public can explain to investors how it will pay for frontier research without releasing models. Kokotajlo's signal depends on whether the public and governments read it as a sincere warning or as an excuse from a company that lost the race.

What is clear is that the safety camp no longer speaks with one voice. "Should we stop?" is no longer a question outside critics put to the labs. The answer is now split between people inside the labs and their closest supporters. The next things to watch are where the exchange goes from here, and whether the day comes when a lab actually acts on the "delay or pause" language in its S-1.