Hobbyists squeezed a 33B model onto a 12GB GPU. It's fast, but fragile

On October 24, a lab tied to a recruiting platform released a 33B model

Hal here. This one is about a model that, just days after release, had people lining up to say they'd gotten it running on a single home GPU. I haven't run it myself. I went through the model card, the repositories of the people who squeezed it down, and the overseas coverage, and pulled it all together 👀

The model is Nanbeige4.3-33B, released on Hugging Face on October 24. It was built by Nanbeige, a lab whose team is affiliated with the Chinese recruiting service BOSS Zhipin. It has 33.1 billion parameters and ships under Apache 2.0, so it can be used quite freely, including commercially. The key detail is that it is a dense model. Unlike a mixture-of-experts (MoE) design, which calls on only a subset of experts for each token, it runs every parameter every time it answers. That makes it heavy.

Here are the numbers on the model card. On agentic coding evaluations, where the AI reads files and runs commands on its own while fixing code, it scores 65.5 on DeepSWE. Its comparison point, Qwen3.8-27B, scored 42.5. It also leads on Terminal-Bench 2.1, 80.9 to 73.0. On SWE-Bench Pro, though, it trails 60.3 to 61.7, so Qwen comes out ahead there. I actually found it refreshingly honest that the card doesn't claim a clean sweep. The context length, meaning how much text it can take in at once, is 512K tokens, and the model has 600 likes on Hugging Face.

Now for the main story. The model card is written on the assumption of tensor parallelism across four GPUs, where the model is split up and computed on several GPUs at once. Most people don't have that at home. Even so, people determined to fit it on a single card got to work within days of release.

"HyperNanbeige-33B" fits it on one 24GB GPU

The first one I want to highlight is HyperNanbeige-33B from independent developer akhtyamovpavel. It's a GGUF, a file format for running models on local machines, built to fit on a single 24GB card such as an RTX 3090 or 4090. I couldn't confirm the date the repository was created, but it appeared after the model's October 24 release. It has just one star so far, so almost nobody has found it yet.

The approach is interesting. Quantization, which lowers the numerical precision of a model to shrink the file, isn't applied evenly across the whole model. The model contains 366 tensors (blocks of numbers). Only 16 of them, the ffn_gate and ffn_up tensors in eight middle layers, are cut aggressively down to IQ2_XXS, a roughly 2-bit level. The rest stay at IQ4_XS to Q8_0, keeping much of their precision. The idea is to cut only where the model seems least likely to break 🛠️ The result is a 16.7GiB model averaging 3.82 bits per weight.

It uses 23,342MiB of the card's 24,576MiB of VRAM (GPU memory). Nearly full. The KV cache uses q8_0 for both K and V and holds 57,344 tokens. As for speed: prefill, where the prompt is read in, runs at 900 to 1,115 tokens per second. Decoding of reasoning-style output, where the model thinks as it writes, runs at 31 to 34 tokens per second. For structured output like code or JSON, it climbs to 45 to 87 tokens per second. With four concurrent requests, combined throughput is 54.4 tokens per second. The developer even published a configuration for using it from OpenCode, a coding agent that runs in the terminal, which is a nice touch.

The developer lists three tricks that made a difference. First, setting both K and V in the KV cache to q8_0. Matching them enables the FlashAttention kernel, a dedicated routine for faster attention computation, which reportedly roughly doubled decode speed. Second, N-gram speculative decoding, which predicts what comes next from the preceding sequence and verifies several tokens at once. It works especially well on output like code, where the same patterns keep repeating. Third, --no-reasoning-preserve, which discards old reasoning blocks to free up context. None of these make the model smarter. They're about tuning how the model is run. It feels a lot like harness engineering, designing the environment and machinery that runs an AI, and I find that part addictive.

The "honest numbers" from someone who fit it on a 12GB RTX 3060 at 2 bits

One person went even further: Doğukan. Every GGUF in circulation was over 19GB and wouldn't fit on a 12GB card, so they made their own: a 10.8GB mixed-precision 2-bit build. It uses an imatrix, which runs real text through the model to measure which weights matter most and protects them during quantization. To preserve reasoning ability, layers 6 through 12 were kept at higher bit widths. They also fixed the BOS token (the marker for the start of a sequence) configuration along the way.

The reason I trust this report most is that the author lays out the bad results too, under the label "honest numbers." The KV cache is q4, and context maxes out at 24,576 tokens. With an empty context it produces 20.8 tokens per second. After 22K tokens of input, that falls to 11.4 tokens per second, close to half. Reading in those 22K tokens runs at about 410 tokens per second, so feeding it long documents also means longer waits. And offloading half the model to the CPU, for the part that doesn't fit on the GPU, gives about 4 tokens per second. That's painfully slow even for conversation 🤔

The author's conclusion is blunt: at this context length and speed, it's a poor fit for agentic work. Agents keep piling file contents and command output into the context. 24K tokens fills up quickly, and the more you pile on, the slower it gets. The model card bills it as strong at agentic coding, yet squeezing it into 12GB takes away its biggest selling point. That was what gave me the most to think about.

Still, getting a 33B dense model to actually run on a 12GB card is an impressive feat. If all you want is answers to short questions, 20 tokens per second is perfectly readable. And because the author drew the line between what works and what doesn't and published it, the next person to try won't waste time going in circles.

The headline numbers may be real, but it's fragile on local hardware

So how did everyone else react? According to Startup Fortune, the release thread on r/LocalLLaMA reportedly passed 140 points quickly. The same article reports that the model beats Qwen on Automation Bench, 54.2 to 38.3, while Qwen leads on GPQA Diamond (graduate-level science questions), 91.3 to 89.5. With H100 rental prices up about 40% in five months, the outlet credits the model for being self-hostable on a single 24GB consumer card. It also cites users saying results on creative tasks are mixed.

I'll lay out the failure reports too. One r/LocalLLaMA user reportedly said the model couldn't solve their personal test prompt in one shot, describing the output as "terrible" and "garbage." A researcher familiar with earlier Nanbeige models reportedly said the benchmarks are probably broadly accurate, but urged caution: the model is fragile and at risk of falling into "doom loops," repeating the same steps endlessly. In setups with weak loop engineering, meaning the design of when an agent's repeated steps should stop, that hurts. Other early comments say its knowledge stops at June 2024 and that, despite its Chinese-language claims, its multilingual ability is weak.

Another important piece is OrcaRouter's analysis. It reports that using the full 512K context would take 111.7GB for the KV cache alone, more than the 66.3GB of weights. Capping context at 32,768 tokens brings the KV cache down to about 6.5GB. In other words, 512K is a spec-sheet number, and the context you can actually use on a home GPU is far shorter. It also cites community measurements showing that standard 4-bit conversion cuts token agreement with the original model (how often it returns the same tokens for the same input) by 7–10%. That's a bigger drop than typical models see. Cutting a quantization-sensitive model down to 2 bits means every trick for fitting it into 24GB or 12GB is a direct trade-off against quality. The piece also notes that, at the time of writing, there were no third-party evaluations, and only one person had published results from actually running it.

OrcaRouter frames the model as a teacher model for transferring knowledge to smaller models (distillation), not as something meant to go straight into production. There are also reportedly people skeptical of a recruiting company releasing models at all. My take: the official numbers may not be a lie. But they describe running it on four GPUs at close to full precision. The moment you cram it onto one card, you're giving up context and quality in exchange for speed. If you try it, the quickest way to find out is to check first whether your own task prompt works in one shot 💡