"Neural networks are not already clean": OpenAI's Leo Gao rebuts Cammarata, proposes simulation-based KL as the yardstick

Leo Gao, who works on interpretability at OpenAI, has posted "Neural networks are not already clean" on LessWrong (primary source). The post responds to Chris Cammarata's "Already Clean," which argued that the inner workings of neural networks are already organized cleanly enough for humans to read. Gao does not reject that view outright. He accepts that neurons are simpler than had been assumed. What he questions is the standard for calling a network "clean" in the first place.

Neural networks are not already clean

The line, which is also the post's title, does not mean that neurons are chaotic. It means there is not yet enough evidence to call them clean. The argument has moved from whether networks are simple to what someone must show before claiming to understand them.

The concession: neurons are simpler than expected

Gao does not dismiss Cammarata's observation. When researchers look at neurons one at a time, more of them than expected turn out to be things a person can name. Among interpretability researchers, the long-held view was that superposition leaves most neurons polysemantic and unreadable. Gao's concession amounts to saying that, at the least, there is less reason for pessimism than that view suggested.

The concession matters. Gao is not disputing the observation. He is disputing the conclusion drawn from it, that the networks are therefore already clean enough. He frames his rebuttal as a dispute over measurement, not over facts.

The rebuttal: labels that fit top activations are not understanding

The standard way to label a neuron is to collect the inputs that activate it most strongly and look for what they have in common. If the top examples all point to the same concept, that concept is recorded as the neuron's "meaning."

Gao takes issue with this procedure. The top activations cover only one end of a neuron's behavior. A neuron also responds, weakly or moderately, to far more inputs. Even if the label fits the top examples, it says nothing about what the neuron does across most of the rest, and that remainder also affects the model's computation. A label that appears correct is not the same as an account of the role the neuron plays inside the model.

The proposal: replace neurons with simulations and measure degradation in KL

Gao's alternative can be defined operationally. If an explanation of a neuron is really correct, it should be possible to simulate the neuron's activations from that explanation alone. The test is to run the model with the real neuron replaced by the simulated values, then use KL divergence to measure how far the output distribution moves from the original model's. If the model barely degrades, the explanation captures the neuron's role. If it degrades significantly, understanding falls short, however plausible the label may look.

There is precedent for the idea. In 2023, OpenAI showed a method that had a language model write explanations of neurons, simulated activations from those explanations, and scored each explanation by how well the simulated activations correlated with the real ones. Gao's proposal can be read as moving the comparison from activation correlation to the model's outputs. In his view, an explanation is inadequate if the activations roughly match but the model's answers still change. Unlike looking at top examples, this test covers the full range of a neuron's behavior, weighted by how much each part matters to what the model does.

Why measurement matters now

Interpretability is no longer only an academic interest. This month alone has brought research that uses linear probes on hidden states to predict unauthorized actions before they are carried out, and research showing that capabilities can be hidden in a model's weights. Reading a model's internals is now expected to serve as a tool for evaluation, monitoring and even post-incident investigation.

In those uses, the gap between "labeled" and "understood" becomes a direct risk. Suppose a neuron looks clean on its top examples but does work its label does not describe on the remaining inputs. An audit that read the model's internals and judged it safe would miss that work. Gao's rebuttal warns against optimism about simple-looking neurons turning into complacency among auditors.

Whether networks are "already clean" or "not already clean" cannot be settled by impressions. Gao proposed settling it with a number: how much the model degrades when neurons are replaced. Both camps can recheck their claims against this test. The next thing to watch is whether Cammarata, or Anthropic's interpretability team, which favors dictionary-learning features over individual neurons, will answer on these terms.