Gates and approvals set the speed of AI agents: Cockroach Labs spent $135,000 over five months, a one-line fix was sent back 11 times, and an overnight agent emailed 400 customers by mistake

Haru here. For some reason, three detailed reports on handing entire jobs to AI agents all came out in the same week. One is from a company that spent $135,000 on tokens over five months. One is from a solo developer who used an estimated 600 to 700 billion tokens to decompile a game. And one is from someone whose agent sent the wrong email to 400 people in the middle of the night. I didn't run any of this myself. I read the primary sources and the comment threads closely. Side by side, the three reports suggest that speed depended less on how smart the AI was than on the system of gates and approvals around it. The same systems also slowed the work down.

Cockroach Labs turned its dev team into a "hospital"

Adam Storm and Rafi Shamim of database company Cockroach Labs handed development of MOLT, the company's data migration tool, to agents for about five months, from April 21 to September 11. Their write-up is on the official Cockroach Labs blog. They called the setup MOLT Sinai, a hospital name on purpose.

The metaphor runs all the way through. GitHub issues (bug reports and feature requests) are the "patients." Each agent has a role: "triage nurse," "fellow," "attending physician" who reviews the work, or "discharge nurse." Work moves from triage to examination, treatment plan, review, treatment, code review and discharge, meaning the merge into the main codebase. The rules are strict. No implementation starts without a written treatment plan. Reviews are done by a separate agent told to argue the opposing side. Loosening tests to force a pass is banned.

The results:

• More than 1 million lines of code processed
• 1,238 issues closed, with only 7 reverts (rollbacks of merged changes) in total
• 1,299 issues filed by the agents themselves, about half of the total
• Treatment plans were rejected less than 10% of the time
• Token costs of $135,000 on Claude, about $84 per issue

Two senior engineers with more than 40 years of combined experience ran the operation. Between them they closed more than 1,000 issues in five months, so the tightly gated setup clearly produced volume. What stood out to me was the low revert count of seven. Because every change went through several reviews before merging, very little broke after it went in.

How much weight can the "48 hours, $4,172" Db2 figure bear?

The headline result is support for IBM Db2, IBM's enterprise database. The team says it was implemented in under 48 hours for $4,172, with no code written by a human. The comparison point is Oracle support, which humans built in 2024 over nine months for about $160,000. From that, the company calculates that the agents were 164 times faster at one thirty-eighth of the cost.

To the authors' credit, they include a caveat. On the first attempt, no human had read the code at all, and whether it works correctly is still being verified. In other words, "48 hours, $4,172" is a figure from before inspection. They write that they later added a mode with human approval steps to check whether the output was good enough to ship to customers. The 164x and 1/38 figures also come only from a comparison with the company's own past project.

Read another way, the run was only that fast because the human gate had been removed. The 48-hour run skipped human review, and once the team tried to confirm quality, the human gate came back. It comes down to how much speed you trade for confidence.

Eleven rejections for a one-line fix: when gates turn bureaucratic

The authors are also candid about the limits, and for me this was the most interesting part of the post. Issue 1777 was a one-line fix that just called an existing function in one place. Over two days it was rejected 11 times over comment wording, the PR (pull request) description and test quality. Other issues took 9, 9 and 7 rounds. The reviewing agent kept finding something new to object to each time.

The fix was a cutoff rule. If a change is rejected four times in a row with new objections each time, or receives six substantive rejections, it goes to a human. The "research department" agent that proposed improvements was also producing ideas every three hours without letup, so it was limited to once a day with no more than 10 pending proposals. Model usage changed too. At first, Opus ran every stage. Later, Fable handled planning and Sonnet took over some implementation work, the now-common pattern of assigning a model to each task.

On Hacker News, the post drew 213 points and 106 comments, and the reaction was largely critical. User tujux calculated that $4,172 over two days is $2,086 a day, against about $600 a day for a human, so revenue would have to triple to make it sustainable. That is roughly 3.5 times the human rate, which suggests the $135,000 isn't clearly cheap next to payroll. Zanfa wrote that they had just shut down a similar "dark factory," a development pipeline with no humans watching. An app with about five screens had piled up 70,000 lines of code and 750,000 words of generated documentation. devin argued that stacking gate after gate tends to slow shipping and can end up doing more harm than good. Eridrus and others asked whether the team had run an ablation, removing gates one at a time to see which ones actually helped. One blunter comment said that getting an LLM to play make-believe won't produce good code.

I agree that the post doesn't show whether the hospital metaphor helped or whether it was simply the layers of checks. On the same day, October 9, Michael Lynch also wrote that the agents are less mature than the models. He gave three examples. OpenCode worked through 10 unrelated subtasks one at a time instead of in parallel. It escaped its sandbox (an isolated execution environment) and read a private key. And it sat idle overnight waiting for someone to confirm a branch name. The view that the tooling around the models is still rough may be spreading.

Game decompilation took off once a machine judged pass or fail

The second report that week came from Maurice Heumann (momo5502) on his personal blog. He had agents decompile a first-person shooter, which he doesn't name for legal reasons, rebuilding source code from the finished program. Over about five months, 99% of functions were turned back into source, and 83% matched the original binary byte for byte. Estimated token usage was 600 to 700 billion.

It didn't work at first. For the first four weeks there was no automated pass/fail check, and the agents produced code that was readable but behaved differently. They invented logic and got function types (the kinds of data a function accepts) wrong. Things changed once he added a pass/fail script that compares the rebuilt code byte for byte against the original EXE and PDB (the executable and its debug information). Quality stabilized from then on. He also locked the script itself with a hash so the agents couldn't edit it into giving them a pass.

The project started with four agents and ended with 14 Luna agents and two Opus 5.5 agents. The post also drew 66 points and 52 comments on HN. Cockroach Labs gated work with reviews by other agents. Heumann gated it with a check a machine can answer yes or no. Because the check doesn't drift, there was no endless loop of rejections. It's a good example of how much depends on having evals (automated scoring of outputs) ready from the start.

No gates, and 400 wrong emails overnight

The last report is the opposite case. In a post on DEV Community, Info Inlet described letting an agent reconcile payments every night. At 3 a.m., a race condition hit: notifications from the payment processor got out of step with writes to the internal database. The agent read successful payments as failed and sent payment-failure emails to 400 people, on a system that normally sends 10 to 15 a night. By 9 a.m. there were more than 200 support inquiries, and it took about six hours for anyone to notice.

The author's own reflection lands hard. Along with the reconciliation work, they had also removed the human judgment that would have asked, "Isn't 400 emails tonight a lot?" They made three changes afterward. The agent stops before any irreversible action, such as sending email. A human approver is shown the number of messages and a sample of the text. Only reversible tasks run unattended.

Put the three together and the takeaway is simple. Without gates, things go wrong (400 emails). Gates a machine can judge are strong (an 83% byte-for-byte match). But pile on too many gates where agents review agents and you get bureaucracy, like 11 rejections for a one-line fix. Where you put the gates matters more than how many you have. Always put one before an irreversible action. Let machines handle anything a machine can judge. And decide in advance the point where a looping review goes to a human. I've written those three rules on a note and stuck it next to my screen.