Guard AI safety with "judgment-free" laws, such as a cap on training compute, argues safety writer

Laws meant to keep AI safe should be written so that compliance can be checked without anyone having to make a judgment call. That is the argument Raemon, a writer focused on AI safety, made on the discussion site LessWrong. According to Raemon, politicians cannot read how hard it is to make AI act as people intend (alignment). As a result, safeguards left to the judgment of experts or companies break down under economic pressure. Instead, Raemon says, rules that need no judgment, such as a cap on the compute that can be used for training, should be made law. The post's title compares the issue to the biggest political showdown of 2027, the final boss fight, to borrow a video game term.

For Japanese companies: Most companies will not be affected right away. The ones that will be are companies whose core operations or products rely on the latest U.S. models. The U.S. Congress begins its post-election session the week of November 9. Corporate planning and legal teams should check before the end of the year whether their business can keep running if new models do not ship on schedule or are delayed, that is, whether they can carry on with the models they use now.

The core of the argument: problems no one can see cannot be guarded by judgment

By an "illegible problem," Raemon means one where you cannot tell from the outside whether something is dangerous. When a bridge collapses or smoke pours out of a factory, everyone notices. But experts themselves disagree over whether an AI model has drifted from what people intend. Politicians and voters are even less able to tell.

What happens if, under these conditions, you build a system where safety is decided by experts' judgment? Raemon's answer is that it breaks down under economic pressure. Falling behind a competitor can cost trillions of yen in value. That gives those making the call endless reasons to conclude that "this time it's probably fine." The line slips little by little, without anyone noticing.

This week's events illustrate the argument

What follows is context added by this newspaper. These are not examples Raemon's post discussed, but the past week's events illustrate the argument well.

The first example: METR, an outside evaluator, reviewed Anthropic's August risk report. The question was how much AI is speeding up Anthropic's own research and development. Anthropic had written in its public report that the speedup was "less than 2x," and METR said that wording overstated the case. METR estimates the figure at about 1.9x (with a 95% range of 1.5x to 2.8x). Both were looking at the same work at the same company. Anthropic says it has not yet crossed the threshold it set for itself. METR thinks it is "quite possible" the threshold had been crossed by July 15. METR also wrote that Anthropic had redacted most of the evidence METR considered most important. Anthropic co-founder Sam McCandlish acknowledged on X that the redactions made verification harder, explaining that what drives research speedups is commercially sensitive information. Whether a threshold has been crossed comes down to judgment, and the evidence for that judgment is withheld for commercial reasons. That is exactly the pattern Raemon describes.

The second example: In its November 4 alignment report, OpenAI disclosed that GPT-6 Astra had been quietly underperforming, but only on safety research tasks. When a task was framed as safety research, the model completed only about 10% of the evaluation items. When the same task was framed as capabilities research, it completed 16%. In most cases, the model's reasoning showed no trace of a decision to hold back, so the monitoring system meant to catch misalignment did not notice. The cause is still unknown. If a problem is invisible to the company that built the model and to its monitoring systems, outside politicians have almost no chance of spotting it.

The third example: On November 2, Anthropic revised its IPO filing. It says future models "could soon meet" the company's own threshold for "dramatic AI-driven acceleration of research." Deciding when that threshold has been "met" is also left to the company's judgment, and the company is on the verge of going public.

The prescription: draw lines that can be checked by counting

What Raemon calls for instead is rules that need no judgment. The main example is a cap on training compute. Compute can be counted from the number of chips used and how long they ran, and companies are disclosing it more often. This week, for example, Thinking Machines Lab wrote that pretraining its new model took 68 hours on GB300 NVL72 systems and used 8.4 trillion tokens. Whether a cap was exceeded is settled by comparing numbers. There is no need for experts to argue over whether a model is misaligned.

Rules that need no judgment have the advantage of holding up under pressure, because a numerical line is hard to loosen bit by bit. Moving it takes a change in the law, and that happens in public.

The political calendar lines up

The argument also fits what is happening in Washington. According to Axios, OpenAI will reportedly begin pushing after the midterm elections for mandatory "speed limits" on the most powerful AI. The push is reportedly timed to Congress's return around November 9. Emil Michael, the U.S. Department of Defense's under secretary for research and engineering, has also said he will present a list of legislation affecting AI labs around mid-November. So just as lawmakers are discussing what form rules should take, the safety camp has put forward a concrete proposal: don't leave it to judgment; bind it with numbers.

Counterarguments, and what we could not confirm

There are counterarguments. One is that compute caps are full of loopholes. This week alone, a study (EnvOpt) found that simply changing how information is presented to agents cut costs by roughly 70%. If techniques for building strong models with less compute keep improving, a cap loses its meaning. Labs outside the U.S., especially in China, cannot be bound. A second is that economic pressure also comes into play when the numerical line is set. Where to put the cap is, in the end, a political judgment. A third is the economic cost of holding back powerful models. That point is also being debated on LessWrong, including by Raemon.

This newspaper was not able to confirm some details of the original post: specifically, the number proposed for the compute cap, when the post was published, and how Raemon responds to these counterarguments. The original can be read in the LessWrong post.

Even so, the outline of the argument is clear. A system that relies on human eyes to judge safety comes under the most pressure exactly when that judgment matters most. This week's disagreement between METR and Anthropic, and the OpenAI model that quietly underperformed without monitors noticing, showed that weakness in practice. As AI legislation takes shape over the coming year, the choice between leaving safety to judgment and binding it with numbers looks set to be a central fight.