Local AI agents reported success on failed tasks — 4 of 5 runs fell short

Claims that you can run Claude Code for free, entirely on your own PC, have been circulating widely. On Nov. 4, two developers published back-to-back tests of that idea on DEV Community, a developer publishing platform, each paying for the experiments themselves. Haru followed the numbers in both all the way through. The short version: on small machines, agents can fail and still finish as though they had succeeded. Both articles found the same cause. Local model runners set a default context window, and it was too small, so the long instructions sent to the agent were cut off partway through without any error.

What it means for Japanese companies: Companies thinking "we don't want data leaving the building, so let's have AI write code on in-house PCs" should watch for this trap. Run on cheap hardware with default settings, the AI may report "done" without having done the work. Before rolling it out, companies should first set up a procedure for people to check what was actually produced.

The same three-step task on a roughly €200 mini PC and a Mac M3

The first article was written by Diogo Soares, a Staff AI Engineer at Nextlane in Portugal. According to his DEV Community post, he ran Claude Code 2.1.311 in unattended, non-interactive mode and pointed it away from Anthropic's cloud to Ollama 0.35.0, a free tool for running AI models on a local machine. He used two models, Qwen2.5-Coder 7B and Gemma 4 E4B. The task had just three steps: create a small Python file that checks whether a string is a palindrome, write a test for it, and run the test. He ran it across five configurations.

Only one run succeeded, on the Mac M3. The results:

• Baseline, cloud-based Claude Opus 5.5 (Pro plan): three steps in 10.6 seconds
• Mac M3 (16GB memory) + Gemma 4 E4B (context set to 32k): succeeded in 3 minutes 33 seconds
• Mini PC integrated GPU + Gemma 4 E4B: GPU crashed after 27 minutes
• Mini PC CPU only: stopped at the 45-minute limit (about 19 minutes before the file was created)
• Qwen2.5-Coder 7B (32k context): spent 9 minutes and wrote correct code, but used the wrong tool-call format, so no file was created

Speed is where the setups diverged. Measured in tokens, the units AI models use to split up text, prompt processing ran at 11–14 per second on the CPU, 19–37 on the integrated GPU and 190–230 on the M3. Generation ran at about 3 per second on the CPU, 3.7–7.5 on the integrated GPU and 13–21 on the M3. On the CPU, just reading 13k tokens took 15 to 20 minutes. The mini PC also dropped its connection roughly every six minutes, which matched Claude Code's behavior of giving up after five minutes with no response. His conclusion: a low-power mini PC is not worth using as a cheap substitute for Claude, but Apple Silicon or a real GPU is reasonable for learning, small private tasks and offline work.

Why "failure reported as success" happened

This is the most important part of the story. Harnesses such as Claude Code, the agent runtimes that drive the model, send the AI a long set of instructions on every turn covering tool usage and rules. By Soares' measurements, the first prompt was about 12,900 tokens and the second step about 19,000. But on machines with less than 24GiB of VRAM (dedicated GPU memory), Ollama's default context window is only 4k. Of the 12,900 tokens, only about 2,000, or one-sixth, were kept. The rest was discarded without any warning.

With most of the instructions gone, the first run stopped after writing out the text of a tool call instead of actually invoking the tool that creates files. No file was created, yet Claude Code exited without an error. Soares wrote that a failure reported as success is worse than a crash. Haru agrees: a crash gets noticed, but when a tool silently pretends to succeed, no one notices and the next step goes ahead regardless.

There were other pitfalls. Qwen lists "tools" (tool calling) among its supported capabilities, yet it never once emitted a call in the structured format the agent was waiting for. Soares concluded that a claim of support is not evidence that a model can make calls in the required format. The test Gemma wrote wrapped its assertions in code that ignored failures, so even when the test failed, the exit code still signaled success. It looked like a pass but verified nothing. On top of that, Ollama's prompt cache, which reuses already-processed content, did not take effect on the second step, so the Mac reprocessed all 19,000 tokens from scratch.

One more aside: when Soares reviewed six YouTube videos promoting "free Claude Code," some of them advertised "six months of Opus for free." He treated that as a security red flag and did not open the links. It is a useful reminder not to click questionable links just because they say "free."

Fixing a bank-ledger bug: Mac, RTX 5080, DGX Spark and cloud compared

The second article came from Nikolai Girchev, a developer who published his tests on DEV Community. His post uses minibank-bench, a benchmark he built and published on GitHub. It contains a bank ledger of about 700 lines of Python with a bug: monthly statements recalculate transfers at the current exchange rate rather than the rate at the time of the transfer. Each configuration's agent was asked to fix it and was scored on whether it passed 20 hidden tests and whether it avoided nine deliberately planted wrong fixes.

• Cloud-based Opus 5.5: 91 seconds, score 0.96, $0.30
• Cloud-based Codex (GPT-6.1-sol): 101 seconds, score 1.00, about $0.14
• Cloud-based DeepSeek V4.1 Flash: 112 seconds, score 1.00, $0.008 (1/36 the cost of Opus by the author's calculation)
• Gemma 4 12B on a MacBook M5 (32GB): 575 seconds, score 0.21 (12/20 hidden tests). It reached 0.21 in only three of 10 runs, and per-run scores ranged from −0.17 to 0.21
• Qwen3 Coder 30B A3B: fast at 52 tokens per second but scored 0. After 50 tool calls it fell for a deliberately planted trap conversion function
• Qwen3.8 27B: zero edits after 20 minutes

Here too, the cause was the context window. LM Studio, which like Ollama runs models locally, loaded some models with a default context of 8192 tokens. The agent he used, OpenCode, sends about 15k tokens per step. The result was an agent that broke without raising an error. The author expanded the context to 65536 and reran the measurements. Notably, the two articles hit the same hole using different tools.

How did larger machines fare? On a laptop with an RTX 5080, Qwen3.6-35B-A3B got faster after tuning, from 21 to 102 tokens per second at a 17k context and from 9.5 to 94 at a 42k context. But its score only rose from 0.29 to 0.31. It got faster, not smarter. On NVIDIA's desktop AI machine, the DGX Spark, ThinkingCap-Qwen3.8-27B scored 0 with a 10-minute limit, and 0.92 in 987 seconds with no limit, close to Opus. But that took about 16 minutes, compared with roughly 1.5 minutes for Opus. Qwen3.8-Flash-Next (a 125B MoE) scored 0.96 but was six to seven times slower than Opus and occupied about 96.2GiB of the machine's 128GB of memory. The author concluded that "models that fit on a laptop can only fix bugs you can spot by looking," and that local models "can replace [cloud models] for this task; for real projects, I don't know."

The authors' caveats, Ollama's updates and Haru's guidelines

Girchev himself cautions that the benchmark is a single Python task and that each configuration was run only once or twice, so it cannot distinguish between local models of similar ability. His tables also contain an inconsistency: Qwen3 Coder on the MacBook shows 472 seconds and a score of 0 in the first table, but 344 seconds and 0.14 (0 to 0.14 over six runs) in the summary table. Fine-grained rankings should not be taken too literally. There is one more warning. Some of the models tested were abliterated, meaning modified versions with the safeguards that refuse dangerous requests removed, and the pi agent runs commands without confirmation by default. The author recommends running that combination inside a container, isolated from the host machine. He also had Qwen3.6-35B-A3B edit his article, and summed up the experience as: it "answers fast, heats the room well, and is mostly harmful." A refreshingly candid verdict.

It is also worth looking at what the runtime developers have done. According to Ollama's release notes, v0.40.14 on Nov. 3 fixed Ollama so that it passes each model's maximum output length when launching Claude Code through Ollama. As a result, using cloud models through Claude Code no longer runs into the default 128k limit, the notes say. The following v0.40.15 (with release artifacts dated Nov. 5–6) added MiMo V2.6 (9B), a model aimed at coding and agents, and made Kimi Code CLI just as easy to launch. It also added support for YaRN, a technique that lets Qwen3 and Qwen3.5 in the Qwen series handle long text. However, the notes say nothing about fixing tool-call handling or the default context window. At least on paper, the hole both articles fell into remains open. Ollama has supported connecting to Claude Code since v0.14.0 (January 2026).

Haru's guidelines: if you run agents on local AI, first set the context window larger than the instructions the harness sends. Then, when the agent says it is done, have a person check that the file actually exists and that the tests fail when they are deliberately broken. Choose Apple Silicon or a serious GPU, and start with small tasks or offline work. If the only goal is lower cost, these numbers suggest that cheap cloud models are often faster and more reliable. Whether data can leave the company, though, is a decision each company has to make for itself.