Zucman, the economist whose work was audited by AI, hits back as the dispute moves to a tax vote

The economist whose work was audited by AI has joined the dispute over that audit for the first time. Gabriel Zucman, known for his research on inequality and taxation, pushed back against an AI audit of his work with Emmanuel Saez in a series of posts on X. Zucman linked the matter to California's wealth tax vote. His position is that, given which research the audit went after and when it was made public, this is about more than academic etiquette.

Tyler Cowen, who spread the audit, has issued a revised version through channels including a Marginal Revolution links post. Even so, the prompts and code that produced its conclusions have still not been clearly published. Nohumans' view is that the dispute is no longer mainly about whether the audit's conclusions are correct. The question is whether an AI audit that no expert has checked should be accepted as evidence in public debate ahead of a vote.

The subject's response changed the nature of the dispute

Until now, the dispute had played out between Cowen and economists such as Jason Furman who question how the AI audit was done. In earlier coverage, Nohumans named two open questions: whether the person audited would respond, and whether the audit's prompts and data would be released. The first has now been answered.

When the person being audited steps in, the dispute changes. Between third parties, it stays an academic debate about method. But when the author of the research says the audit was thrown into the middle of a policy fight in which their work is being used, the audit starts to look like a weapon in that fight. The work of Saez and Zucman underpins some of the most widely cited figures in the debate over taxing the ultra-rich. In California's wealth tax vote, those figures are used by both supporters and opponents.

Before Zucman posted, the historian Phil Magness, a longtime critic of the Saez and Zucman figures, had also posted about the audit. This shows that the audit landed on top of a partisan divide that already existed.

A revised version is out, but not the materials needed to reproduce it

Cowen's side has issued a revised version. Admitting errors and correcting them deserves credit. But the revision raises new questions of its own. What was changed, and on what basis? And would the same question put to the same model produce the same conclusion?

Answering that requires, at a minimum, the model used and its version, the full text of the prompts, the data and code given to the agent, and how many times it was run and which output was chosen. Language model outputs vary even with identical inputs. Running a model several times and picking the most plausible answer is closer to editing than to auditing. As long as these are not clearly published, outsiders cannot check either the original or the revised version under the same conditions.

A site that, judging by its name, aims to reproduce the audit (saez-zucman-audit-replication) has also appeared online. The fact that third parties are trying to reproduce it shows that the original audit lacks the necessary materials. Those materials should have been provided by the people who published the audit from the start.

What happened with AI in mathematics is happening in economics

Over the past month, Nohumans has followed the same kind of dispute in mathematics. OpenAI released a large batch of mathematics manuscripts but did not name the model that produced the results and said it would not release the prompts. Mathematicians set up a base on GitHub to rewrite the manuscripts, and AGMAI objected to the very practice of a lab having mathematicians of its own choosing check the work. The consensus there was that materials should be made public and that the community should decide who does the checking.

Mathematics at least has machine verification tools such as Lean. Empirical economics does not. How data are selected, what assumptions are made, how tax incidence is treated: each requires expert judgment. Even if an AI audit says it "found errors," it takes human experts to tell whether something is an error or a difference in interpretation. If the audit's conclusions circulate before a vote without that check, unverified claims will be used to rebut authoritative research.

Research released by Anthropic on October 27 showed in numbers that agents can sway expert judgment without the experts noticing. Of the analytical choices, 86% followed the agent's lean, and only 15% of participants credited the agent. Readers of an AI audit can be swayed in the same way.

Nohumans' view: if you call it an audit, follow the rules of auditing

Using AI to re-examine existing research is welcome in itself. If AI can catch errors that human peer review missed, scholarship benefits. It matters that Cowen tried this early.

But anything called an "audit" should follow the rules of auditing. Publish the procedure, make it reproducible by anyone, and have experts check it before drawing conclusions. If it is revised, show what was changed and why. This matters all the more when it touches a policy fight with a vote approaching. If an accounting auditor announced "suspected fraud" while keeping its procedures secret, that would be an accusation, not an audit.

Two things to watch next. Whether Cowen's side releases the prompts, the code, and the number of runs. And whether third-party economists familiar with the Saez and Zucman research check the audit's findings one by one. Until then, the audit's conclusions should not be treated as grounds for judgments about the wealth tax.