Former OpenAI executive's startup trains AI in 4-bit, cutting training time by up to 20%

Thinking Machines Lab, a U.S. startup, has published a way to speed up the computation used to train AI models by using coarser numbers than today's standard. The company is led by Mira Murati, OpenAI's former chief technology officer (CTO). According to the company's technical blog, it ran the forward pass of the expert layers (the MoE expert layers), meaning the computation that goes from input to output, in the 4-bit number format NVFP4, consistently from the very start of pre-training to the end of reinforcement learning. Compared with running everything in MXFP8, the 8-bit format that is the current mainstream, training time fell by about 8% in pre-training and about 20% in reinforcement learning. The company says the only trade-off was a drop of about 2% in pre-training compute efficiency. However, it has not released the in-house kernels (the computational building blocks that run on the GPU) behind these results. Outsiders are not yet in a position to reproduce them.

For Japanese companies: For most companies that use AI, there is no immediate impact. The news matters to companies that train or fine-tune their own models, and to companies that rent or lease out GPUs for that purpose. Development leads at those companies should check whether the GPUs they use are NVIDIA's Blackwell generation. They should then review, before the end of the year, whether next fiscal year's training cost estimates assume 8-bit. There is no direct effect on contracts or pricing for Copilot, ChatGPT's enterprise plans, Gemini and similar services.

What it did: coarser numbers, lighter computation

AI training consists of an enormous number of multiplications and additions. Using fewer bits to represent each number lets the same GPU handle more numbers at once, which speeds up training and reduces memory use. The trade-off is that coarser numbers risk causing training to go off track.

In training large models so far, 8-bit formats have been the compromise between speed and accuracy. This time, Thinking Machines used 4-bit, half that, for the forward pass of the expert layers, consistently from pre-training through the end of reinforcement learning. However, it kept the backward pass and the last one-sixth of the layers in 8-bit. NVFP4 is a 4-bit format that NVIDIA's latest generation of GPUs (Blackwell) can compute quickly.

The company has previously said that Inkling 2 Small, an open model whose weights it released on November 12, was also trained in NVFP4 from the start. The model has 594B (594 billion) parameters in total, and running it requires more than 500GB of memory for the NVFP4 version and more than 800GB for the MXFP8 version. The new blog post can be read as revealing the details of that 4-bit training.

What was gained and what was lost

The company presented three figures.

First, the time needed for pre-training (the stage in which the model reads large amounts of text to build its foundation) was about 8% shorter than when running everything in MXFP8.

Second, the time needed for reinforcement learning (the stage in which the model generates answers that are scored as good or bad to train it) was about 20% shorter. This is the "up to 20%" in the headline.

Third, the trade-off was a drop of about 2% in pre-training compute efficiency. Compute efficiency refers to how much training progress is made for the same amount of computation. Coarser numbers slightly reduce the progress made per computation, but the gain from each computation running faster is larger, so overall training finished sooner.

The larger reduction in reinforcement learning appears to be because reinforcement learning spends a long time having the model generate answers repeatedly, and generating answers is forward-pass computation. The effect of moving the forward pass to 4-bit therefore shows up more directly.

If training time shrinks by 20%, the same number of GPUs can run more rounds of retraining, or rental time can be reduced. With GPU prices and electricity costs rising, that is no small difference for those doing the training.

Outsiders cannot yet do the same

There is a major caveat to these figures. Thinking Machines has not released the in-house kernels at the core of the method. The 4-bit format itself is an NVIDIA standard that anyone can use. But the detailed engineering that keeps training from breaking down even with coarse numbers lives inside the kernels. Without them, other companies and researchers cannot run tests under the same conditions to verify whether the 8% and 20% figures can be reproduced.

In other words, the current figures are the company's own measurements. Compared with its release of the model weights under Apache 2.0 (a license that permits free use, including commercial use), its disclosure of the training method goes only halfway.

What could not be confirmed

According to the blog, the backward pass (the computation that propagates errors back to adjust the weights) was not moved to 4-bit and is done in 8-bit MXFP8. The GPUs were NVIDIA's GB300, and pre-training of Inkling 2 Small reportedly finished in 68 hours. The roughly 2% drop was measured as the difference in the amount of compute required to reach the same loss. The figures and charts in the blog reportedly come from the training of Inkling 2 Small. However, the following two points could not be confirmed: the scale of the workloads compared (model size and number of GPUs), and whether the company plans to release the kernels at some point.

What to watch next

There are two things to watch. One is whether Thinking Machines releases its kernels, or whether NVIDIA or another company independently reproduces a similar reduction in training time. The other is whether Inkling 2 Small, trained in 4-bit, performs in external evaluations as well as the score the company published (31 on Artificial Analysis's overall index). If the latter is borne out, the claim that training in 4-bit barely reduces a model's intelligence will gain outside evidence.