Skip to content
Skjld Labs

No AI model is best at everything: that's why we test them on our own tasks before deployment

We put five local language models through five real Norwegian work tasks and had them judge each other blind, without a single token leaving the building. Each model revealed its own strength profile. And the most important finding wasn't about the models, but about the knowledge required to use them well.

8 min read Laget med AI-hjelp
Overview of five local AI models with their strength profiles, from Skjld Labs’ model test

Model cards and benchmarks tell you how a language model performs in English, on generic tasks, on its best day. They don't tell you how it handles Norwegian professional language, your format requirements, or your hardware.

We run our model fleet locally: four models behind a LiteLLM proxy, plus the large Qwen3.5-122B. Before any model gets a job with us, it has to prove itself on our tasks. This time we're sharing the method and the numbers, and what the numbers showed was not one winner, but five distinct strength profiles. The most important finding wasn't about the models at all.

The setup

Five open-weight models, all on our own hardware:

Gemma-4-26B: 26B (4B active) · MoE
Qwen3.6-35B: 35B (3B active) · MoE
Nemotron-3-Puzzle-75B: 75B (9B active) · hybrid MoE (Mamba-Transformer, NVFP4)
Qwen3.5-122B: 122B (10B active) · MoE
Agents-A1: 35B · Qwen-based MoE agentic model (InternScience, Apache 2.0)

Five real tasks, in Norwegian: structured insight extraction from podcast and news transcripts, grounded Q&A with mandatory verbatim quoting, weekly insight synthesis, business lead qualification with reasoning, and first-contact strategy briefs. The same frozen task set for every model. More on why that phrasing matters, in the checklist. All company details are anonymized here.

And judging without calling a cloud: every answer was anonymized and blind-scored by three models that didn't write it: five axes (factual fidelity, insight value, instruction-following, Norwegian language, actionability), median as the verdict. No judge knew who wrote what. None judged itself. The panel agreed remarkably well on ranking (rank correlation 0.69-0.89 between judges), but is not human-calibrated, so read the numbers as a ranking, not a report card.

The numbers

Panel score 0-100, median of three blind judges. Order: Gemma-26B / Qwen-35B / Nemotron-75B / Qwen-122B / Agents-A1. Bold = best.

Insight extraction: 41 / 43 / 39 / 51 / 36
Grounded Q&A: 56 / 43 / 37 / 53 / 41
Norwegian synthesis: 64 / 55 / 60 / 72 / 45
Lead qualification: 75 / 97 / 81 / 82 / 98
Strategy brief: 100 / 93 / 99 / 100 / 99
Tokens/second: 39 / 43 / 25 / 130 / 43

Five models, five tasks, and five distinct strength profiles.

The strength profiles

Qwen3.5-122B: the precisionist. Won the extraction and the synthesis, second on grounded Q&A, full marks on strategy briefs: clean JSON with no decoration, verbatim quotes, fewest fabrications. And here the first intuition breaks: the largest model was also by far the fastest (130 tokens/second): the MoE architecture activates only 10 of its 122 billion parameters per token, and our serving stack exploits that well (more on why parameter counts don't predict speed under the operational lessons). If you need one model for structured factual work at volume, this is the profile to look for.

Agents-A1: the decider. A 35-billion-parameter MoE agentic model from InternScience (Qwen-based, Apache 2.0), trained specifically for long agent trajectories, averaging 45,000 tokens per training trajectory, with six agentic domains distilled into one model via multi-teacher distillation. Their ambition is frontier-level agentic performance from a 35B model, and with us it topped the qualification task (98 of 100): clear verdicts, precise product choices, reasoning that held up against the facts. But flip the page: on grounded analytical work it was the weakest in the field (36 on extraction, last overall in the insight track). A specialized model is specialized in both directions: the profile decides where it belongs, not the average. And only testing reveals which way the point is facing.

Qwen3.6-35B: the gatekeeper. Right behind the decider (97 on qualification), with the same strict decision discipline: where others said "maybe", it made the call, and dropped two thirds of the candidates. Zero format errors throughout the test. When a human's time is the downstream bottleneck, a strict gatekeeper is a strength, not a weakness.

Gemma-4-26B: the communicator. Won grounded Q&A and shared the top spot on strategy briefs (100): the most natural Norwegian prose in the field, text with direction, an addressee and a concrete angle. Also the most optimistic assessor: it would have pursued half the candidates, a trait you want in creative work and must compensate for in gatekeeper roles. Among the smallest in the field, and second overall on the insight work: the right model for the right job beats raw size.

Nemotron-3-Puzzle-75B: the thinker, and the test's lesson. A hybrid Mamba-Transformer MoE reasoning model (9B active parameters, compressed from 120B with NVIDIA's Puzzle framework) built for deep, multi-step analysis with thinking mode on, which the test deliberately disabled, for a like-for-like comparison. Outside its element it delivered mid-field where the tasks resembled reasoning (60 on synthesis, 99 on strategy briefs) and weakest where they demanded grounded rapid work. That is not a verdict on the model. It is the point of testing: the configuration a model ships with can determine where it lands. One qualification belongs here: thinking mode wasn't measured, so we don't know how much it would recover. That requires a separate test with thinking time in the budget.

The temperaments deserve their own overview, because they are a decision you inherit whether you want to or not. Same twelve leads, five distributions (pursue / maybe / drop):

Gemma-26B: 6 / 5 / 1
Qwen-35B: 4 / 0 / 8
Nemotron-75B: 5 / 7 / 0
Qwen-122B: 3 / 9 / 0
Agents-A1: 4 / 0 / 8

Gemma is the sales optimist. The decider and the gatekeeper are the two strict ones: identical distributions, clear decisions. And the two "maybe"-heavy models push the decision back to a human in practice: nine of twelve "maybes" isn't caution, it's unpaid operator time.

The two operational lessons

The thinking tax. Our first test run failed nearly half the calls against the reasoning models: they spent the entire token budget on internal thinking and delivered an empty answer, at full price. One configuration line (explicit thinking-mode control) separated useless from excellent. This is not in the model card.

The spec sheet doesn't tell you the throughput. Two of the models are MoE with nearly identical active parameter counts, 9 and 10 billion, yet delivered 25 versus 130 tokens/second on our setup. Quantization format, serving stack and kernel support for your particular hardware matter more than both the parameter count and the architecture column. Calculate cost as measured throughput on your own hardware, never as spec.

Gemma-4-26B: 39 tok/s · mean latency 7.0 s
Qwen3.6-35B: 43 tok/s · 7.0 s
Nemotron-3-Puzzle-75B: 25 tok/s · 13.4 s
Qwen3.5-122B: 130 tok/s · 2.4 s
Agents-A1: 43 tok/s · 6.9 s

The right model for the right job applies to procurement too

The strength profiles point to a question that is settled long before any prompt is written: what kind of hardware do you actually need? It's easy to spend half a million kroner (~$50,000) on a GPU server that can run the largest open models. But the smallest models in the test fit on a compact workstation in the 50,000-kroner (~$5,000) class, such as the Dell Pro Max GB10. That covers the 26B and 35B class, including Agents-A1.

Best model per price class and task (small box ~$5k vs GPU server ~$50k):

Insight extraction: Qwen-35B 42.7 vs Qwen-122B 51.3, small box delivers 83%
Grounded Q&A: Gemma-26B 56.0 vs Qwen-122B 53.3, 105%
Norwegian synthesis: Gemma-26B 64.0 vs Qwen-122B 72.0, or 89%
Lead qualification: Agents-A1 98.3 vs Qwen-122B 82.0, 120%
Strategy brief: Gemma-26B 100.0 vs Qwen-122B 100.0, 100%

(The percentages are based on panel scores that aren't human-calibrated. Read them as orders of magnitude. Price classes are approximate.)

On extraction and synthesis, the big server delivers a real but moderate gain: the small box achieves 83-89 percent of the quality at roughly a tenth of the hardware price. On grounded Q&A, qualification and strategy briefs, the small models matched or beat the large one, and on qualification it was precisely the specialized 35B agentic model that beat everything. The point is not that small boxes always suffice. It's that the price-performance curve is task-dependent and can't be read off a spec sheet. For many businesses, a box in the $5,000 class covers most everyday tasks. Which tasks actually need the big iron, you only find out by measuring.

Our checklist before a model gets a job

  1. Test on your own tasks, in your own language: benchmarks don't measure Norwegian professional text or your format requirements.
  2. Test in the right mode: thinking mode, temperature and token budget are part of the model choice, not details to sort out later.
  3. Freeze the task set: if models are to be compared over time, they must face exactly the same tasks. (Yes, we learned this along the way: the first version of the test built its tasks from live production data, and the set drifted. The entire study was re-run.)
  4. Measure cost as tokens/second on your own hardware: neither total nor active parameter counts predict throughput.
  5. Choose the temperament consciously: optimist, gatekeeper or indecisive. Your pipeline inherits the model's personality whether you chose it or not.
  6. Let the models judge each other, blinded: a local judging panel costs electricity and gives you rankings without sensitive data leaving the building.

Why this matters for your business

The entire experiment, generation and judging, ran on our own hardware, on real data. No cloud provider saw a single token. The bill was electricity.

That is the real payoff of sovereign AI: not just that the data lives at home, but that you can experiment freely with it: test, measure and adapt models against your actual tasks before they're given responsibility. The knowledge of how to do exactly that is the difference between a model that disappoints and a fleet that delivers. A competent small team with its own infrastructure does a test like this in a day, and re-runs it the next time a promising model appears, on exactly the same tasks.

Method in brief: 32 frozen tasks × 5 models = 160 generations via a local gateway, temperature 0.2, thinking off. Blind judging: three non-author models per answer, five-axis rubric, median (449 valid judging calls of 450). Limitations: 30 judged tasks per model, an LLM panel as judge (not human-calibrated, and three of the five models are in the Qwen family, shared bias possible), the reasoning model's thinking mode not measured, and performance measured under normal load.

From article to practice

Want to put this to work?

Our products. Svaragent, Skjld Diktat, Skjld Cloud and Skjld Kurs, the tools we have built and use ourselves.

Read about Our products Book a free call
From our workshop

Read on.

Newsletter

AI in practice, one email a month.

What we've tested, one tip you can use, and what's coming next.

You'll get the Skjld Labs newsletter once a month, and can unsubscribe at any time. Privacy

First step

Thirty minutes. Free.

A no-obligation conversation about what you are facing. We will say so if AI is not the right answer. We are happy to meet you at Langes gate 8 in Sandefjord, or on video.