Five 27B models tied with Fable 5? More retries, not more smarts, made the difference

A team of five mid-sized AI models, each small enough to run on a local GPU and each given its own role, matched the frontier model Fable 5. That re-run of a benchmark drew 459 points within three days on Hacker News (HN), a forum popular with engineers. We went through the repository, the comment thread and the rebuttal articles. The short version: the win came less from how smart the models were than from how many times they could retry and whether a checker was on hand. It also cost more than first claimed.

What it means for Japanese companies: Companies that want to process data that can't leave the building on their own GPUs will likely hear the pitch "combine small AI models and you can replace expensive ones" more often. When they do, they shouldn't just look at benchmark scores. They should ask how many attempts were allowed, and what each problem cost and how long it took. The lessons here apply directly.

What was done: a team of five 27B models

The project is Memdoor, a tool built by a solo developer. It is a "harness" for running coding agents, a layer that wraps around the AI model and acts as its workbench. The author publishes it in a GitHub repository. This time, the developer took the method and prompts (the instructions given to the AI) from a paper called GVS5H and rebuilt them as-is inside their own harness.

The model used was Qwen3.8-27B, an open-weight model anyone can use. "27B" means it has 27 billion parameters, which is quite small next to frontier models. The developer ran five copies with separate roles: a planner, two proposers that generate ideas, a solver that writes the code, and a supervisor that oversees the whole process. A fixer role that corrects mistakes was also added. Notably, the agents don't talk to each other through chat. They communicate only through files on disk, like passing notes around a desk, which makes it easy to trace afterward who wrote what.

The benchmark was LiveCodeBench Hard, the harder tier of a test that has models solve difficult programming problems and scores them on whether the code runs correctly. Memdoor ran each of the first 10 problems five times. It compared the results with two baselines: the original implementation by the paper's authors, and Fable 5 answering after a single call.

The paper's original implementation repository tested on the 100 most recent Hard problems from LiveCodeBench v6. A single call to Qwen3.8-27B solved only 66.8%, but adding the supervisor raised that to 92.4%, above Fable 5's 90.4%. On its own, that sounds like proof that small models can win if you combine them. But the paper itself says the gap is 2.0 percentage points, the lower bound after accounting for variance is minus 0.8, and the p-value (a rough measure of how likely a gap this size is to arise by chance) is 0.88. Statistically, the paper treats the two as roughly even. "Tied" is more accurate than "won."

Memdoor's numbers

The results posted in the repository are:

• Memdoor's five-model team: average 9.8/10 (49 of 50 runs correct)
• The paper authors' implementation: 9.6/10 (48 of 50)
• Fable 5, single call: 8.0/10 (40 of 50)

Cost and time were also logged. The 50 runs cost a total of $35.79 on OpenRouter, a service that gives access to models from many companies through one endpoint with pay-as-you-go pricing. The median per problem was 40 minutes and $0.39. The models generated 15.3 million output tokens in total (tokens are the units AI models count text in, roughly word fragments). Per correct answer, the Qwen team cost $0.52 and Fable 5 cost $0.68.

The HN post was titled "5 Qwen3.8-27B beat Fable 5 on LiveCodeBench Hard." It reached the front page with 459 points and 230 comments, and as of November 4 the repository had 358 stars and 39 forks. Techniques for combining small models fall under Harness Engineering, the practice of designing an agent's memory and working environment, which is drawing a lot of attention right now. That helps explain why the post landed.

What stood out was the "40 minutes per problem" figure. Fable 5 made one call and returned an answer, while the five-model team spent 40 minutes discussing and retrying. Is that a fair fight? That question became the focus of the comment thread.

Wait: did attempts, not intelligence, win?

The top comment on HN came from a developer who had tried the same thing for weeks. After building a swarm of cheap models to catch up to an expensive one, they wrote that it is surprisingly easy to build a harness that raises a single benchmark score, and surprisingly hard to build one that holds up on other tasks. It is like studying for a practice exam: the score goes up, but the skills don't carry over to different questions on the real test.

Others questioned whether 9.8 and 9.6 can be treated as different numbers when there are only 10 problems. With 10 problems run five times each, a single right or wrong answer moves the average by 0.2. The gap between 49 and 48 is literally one run. The same comment added that the write-up announcing the project read as if Claude had written it.

A critical article published on November 1 went further. It argued that the number of problems, the use of pass@k (counting a problem as solved if any of k attempts succeeds), and how much compute went into each problem were not clearly disclosed. The five-model team could make multiple attempts and had a checker, while Fable 5 got one shot. So what was measured, the article said, was the difference in attempts, not in models. A fair comparison would give Fable 5 the same five attempts and the same verifier. The article closed with: "It measured the wallet, not the model." That point is hard to hear, but it gets to the heart of the issue.

An article on dev.to, a publishing platform for developers, laid out the mismatch in detail. Fable 5 got one call, while the 27B side could go up to 10 rounds. It could also run the public tests (sample test cases that come with each problem) itself to check its answers. A side that can check its work repeatedly before answering naturally has the edge.

The author did include caveats. Only 10 problems were tested, five times each. The problems date from March 29 to April 6, 2025, so the author cannot rule out that they were in the model's training data. The Fable 5 numbers were taken from the paper authors' files and were not re-measured on the Memdoor harness. Spelling all that out was honest. The headline simply took on a life of its own.

The cost wasn't as low as claimed either

The other reason the post spread was the claim that the team "matched Fable 5 at 9% of the cost." But Startup Fortune reported on November 1 that the 9% figure belonged to a different model, Qwen3.8 Flash Next. For the 27B model, the cost was $51.75 versus $61.11, or about 85% of Fable 5. The outlet reported that after re-evaluation, accuracy was 86.4% versus 87.4%, with Fable 5 slightly ahead, and a p-value of 0.73, meaning no real difference.

The dev.to article also calculated that the 27B setup with a supervisor cost $47.82 per run versus $61.11, about 78% of Fable 5. It is cheaper, but by 10–20%, not 90%. And it takes 40 minutes per problem. Factor in speed, and it is hard to call it the better deal for everyday use.

Memdoor's comparison of $0.52 versus $0.68 per correct answer also uses Fable 5's one-shot numbers. Allowing Fable 5 a few retries should raise its number of correct answers, which could narrow the per-problem cost gap further. Either way you compare them, the team is not dramatically cheaper.

Another important factor when running models locally is compression. One HN commenter compared results on two GPUs, the R9700 and the 5060 Ti. They said a heavily quantized version of the model (quantization lowers the precision of the model's numbers to make the files smaller and lighter) and a Q6 version, which uses relatively light compression, were "completely different" in real-world work. They added that those distributing quantized models only show easy benchmarks on which nearly every model scores close to perfect. Even with identical test scores, real-world performance may differ.

So is it worth running locally?

There are still takeaways from the project. One is its scale. According to a roundup article, the 27B model fits on a single 16GB GPU. The five roles run in turn on the same GPU, with only the two proposers running at the same time. The design can be tried without a rack of expensive servers.

The repository also includes a configuration that runs Qwen3-Coder-30B-A3B on an L4 GPU in Google's cloud (about $0.71 per hour). It has instructions for Ollama, llama.cpp, LM Studio and vLLM, popular tools for running AI models on your own computer or server. Adding a small routing model partway through reportedly cut the input for questions about code by 74%, from 54,661 tokens to 14,367. The less text the model has to read, the more money and time it saves, so this is a modest but useful trick.

The conclusion: "Five small AI models working as a team can beat a frontier model" is an overstatement. More precisely, with many retries and a checker, small models can raise their accuracy on hard problems substantially, but it takes longer and costs only 10–20% less. For work where data can't leave the company and speed isn't critical, it is a viable option. For work that needs answers fast, a large model is the simpler choice.

The biggest lesson is that when you see an AI comparison chart, you should check how many problems were tested, how many attempts were allowed, what each problem cost and how long it took, and whether the competitor got the same conditions. This episode is a good example of how far a story can grow when those details get skipped.