Compression is the new GPU upgrade: what NVIDIA’s Puzzle model means for self-hosted AI
NVIDIA shrank Nemotron-3-Super from 120 to 75 billion parameters, with no measurable loss on benchmarks. The result: double the capacity on the same hardware, and eight times more concurrent 1M-token requests per H100. We explain how Puzzle compression works, and why it makes self-hosted AI increasingly viable.
On July 6, NVIDIA released a model with a name only an engineer could love: Nemotron-Labs-3-Puzzle-75B-A9B. Behind the name hides something we consider more important than yet another model launch: clear evidence that compression is changing the economics for anyone who wants to run AI on their own servers.
The numbers worth pausing at
NVIDIA took its own Nemotron-3-Super, with 120.7 billion total parameters and 12.8 billion active per token, and compressed it to 75.3 billion total and 9.3 billion active. The interesting part is not that the model got smaller. The interesting part is what didn’t happen: benchmarks barely move. AIME25 goes from 89.7 to 89.9. MMLU-Pro from 82.4 to 82.2. Long context (RULER at 1 million tokens) actually goes up, from 92.2 to 93.2.
Meanwhile, capacity changes like this:
- Roughly double the server throughput on the same node, at the same per-user latency constraints.
- A single H100 goes from handling 1 concurrent request with 1M-token context to 8.
In other words: same hardware, eight times more heavy concurrent users. It is a hardware upgrade you download instead of buy.
What NVIDIA actually did, explained simply
The method is called Iterative Puzzle, and the core idea is that large models are unevenly important on the inside. Some layers do heavy lifting. Others can be slimmed down aggressively without anyone noticing. Instead of cutting uniformly, Puzzle learns where the model tolerates cuts:
- Non-uniform expert pruning. The model is a Mixture-of-Experts, where each token only activates a few “experts”. NVIDIA reduced both the size of the experts and the number activated, but differently from layer to layer: sensitive layers keep their capacity, robust layers are slimmed hard (from 22 active experts down to between 4 and 18, depending on the layer).
- Distillation. After each cut, the smaller model is trained to imitate the original: tens of billions of tokens of “answer keys” from the parent, including dedicated training rounds at very long context.
- Targeted RL repair. The skills most vulnerable to compression are coding and agentic work. They get a dedicated round of reinforcement learning to recover the loss.
- Quantization. Finally the weights are packed into a 4-bit format (NVFP4) for the latest GPU generation, with an FP8 variant for older cards.
None of these parts is new on its own. What is new is that the entire chain is now systematized, documented and reproducible, and that the result is released openly, with an open license for commercial use and open training data.
Why this matters in Norway
We have written before about why your words should stay in Norway: American cloud services are subject to the CLOUD Act, and after Schrems II, data flows out of Europe are a legal question, not just a technical one. The strongest counterargument against AI on your own infrastructure has always been economics: top-tier models required GPU fleets few Norwegian businesses could justify.
That is the calculation compression changes. When a 120B-class model can be run at 75B cost, and every GPU serves eight times as many heavy requests, the price per user drops accordingly. Open, compressed models with million-token context, tool use and agentic capabilities are now something a mid-sized Norwegian business can operate itself: on rented Norwegian hardware, in its own server room, or with someone who runs it for them. The data never has to leave the country.
The honest caveats
A proper field note also says what the press release does not. The NVFP4 format only delivers its full effect on the latest GPU generation (Blackwell). On H100, the FP8 variant is the realistic choice. 75 billion parameters is still serious hardware. This is not hobby-machine territory. And benchmarks are not your workday: a compressed model should always be tested on your tasks, your terminology and your security requirements before going to production. Compression can weaken exactly the capability you depend on, even when the average looks unchanged.
This is where we come in
This is the core of what we do at Skjld Labs. We already run open models and agents on our own infrastructure in Sandefjord. It is the engine behind Skjld Diktat, and the same craft behind the AI systems and agents we build and host for our clients. We follow the model releases, evaluate them against real workloads, and know what it takes for an open model to pass an audit, not just a demo.
Wondering whether self-hosted AI makes financial sense for your business: what it costs, which model fits, and what operations and security require? Book a free call, or read more about our AI advisory services. We will tell you what we would build first, and what is overkill.
Sources: the model card on Hugging Face and NVIDIA’s technical reports on Puzzle and Nemotron 3.