Poor Man's Model Selection: Not Every Model That Boots Is Useful

Getting a model to load is the first test, not the last. Whether it’s actually useful depends on memory, quantization, bandwidth and the way you call it — none of which show up in the model card.

In part one I built the cluster and gave each machine a role by guesswork. This part is where I stopped guessing: the four questions that picked every model, and the benchmarks — 140 Exercism tasks through the same agent scaffold they’d actually work in — that settled the roster.

In the homelab world, “it fits in VRAM” has roughly the same energy as “it compiles.” Technically true, and almost no information about whether the thing is any good.

I spent part one of this series getting three machines to serve models through one API. That was the easy half. The hard half was picking which model each machine should run — and learning, mostly through humiliating experiments, that “does it start?” and “is it useful?” are two completely different questions.

This post is that part two. It covers the memory equation nobody puts on the model card, why quantization is a deal with the devil, the moment my most interesting model turned out to be too slow to talk to, and the benchmark runs that finally let me decide instead of guess.

The memory equation is bigger than the weights

The most important shift in how I think about local LLMs came from a single realization: when people say “the model fits in memory,” they’re usually only counting the weights. The file is 6.7 GB, so it should run on my 16 GB card — but the file is not the whole bill.

In practice, what a running model actually needs is closer to this:

memory ≈ model weights + KV cache + context + buffers + system

That approximation was enough to make every decision in this cluster:

Weights decide whether the model can load at all.
KV cache and context decide how much information the model can hold onto during a long session — or while working through a repository.
Buffers, backend and operating system decide how much memory is actually left once the server is up.

A model can load perfectly fine and still fail the moment you start a real task, because there’s no room left for context. Or it can load with room to spare and answer at a speed that makes an interactive agent loop physically painful. The file size tells you almost none of this.

Quantization: Q4 is the starting point, not the answer

Quantization shrinks the weights at the cost of precision. Q4 is a reasonable starting point — usually the sweet spot between file size and quality — but it’s not a universal solution, and it’s certainly not a reason to stop checking.

The boundary was made painfully visible by Bonsai on Phobos. The Q2_0 quant let me run a 27B-parameter model from a file of about 6.7 GB — but a smaller file didn’t mean comparable quality. In a quick smoke test Bonsai looked great. Given a longer Exercism set, its effectiveness dropped sharply. I gained a huge context window and the ability to run a bigger model, and I paid for it with reasoning precision.

The lesson I keep coming back to: quantization doesn’t make a model smaller for free. It moves the tradeoff. On the same machine you can pick Q4 when quality and stability matter, or drop lower when fitting a bigger model matters more — as long as you accept that the model’s answers will get worse, not just slightly, but in the specific ways that matter most for real work.

The context window is a bill you pay

A declared context window doesn’t mean the memory is spent on the model. The bigger the context you allow, the more memory goes to the KV cache — the stored keys and values the model needs to re-use earlier tokens. That cost grows with context length and depends on the model’s architecture, its layer count, the cache’s data type, and variants like MTP (more on that later).

So setting --ctx-size to something huge is not free. With a long context the model may load fine, then run out of memory mid-generation — or worse, start swapping into slower RAM and see generation speed collapse.

There’s a lever here: changing the KV cache type from a high-precision format down to q8_0 or lower cuts memory usage. For smaller models, my tests suggest the quality cost is barely measurable — which makes it one of the best optimizations available on small hardware. Mars runs its KV cache as q8_0, Deimos as q4_0, and both would struggle to hold their context windows otherwise.

In Olympus, I sized context together with each worker’s role:

Machine	Model	Context	KV cache	Speed
Mars	Qwen3.6 35B A3B (Q4_K_S)	128K	q8_0	~61 tok/s
Deimos	Ornith 1.0 35B (Q4_K_M)	64K	q4_0	~28 tok/s
Phobos	Bonsai 27B (Q2_0)	160K	—	~10 tok/s

Mars could afford both large context and fast generation. Deimos got a smaller window but strong code quality. Phobos got a large window mostly because its generation was too slow for an interactive agent loop — its context is the whole point of its existence. The same parameter means something different on every machine.

Bandwidth: why “fits in memory” is not enough

Once a model is loaded, generating each new token becomes a problem of memory bandwidth. The model has to repeatedly read large chunks of weights and cache — speed stops depending on core count or whether the model formally fits, and starts depending on how fast data can be moved around.

Phobos demonstrated this better than any slide deck could. Its Radeon 890M shares system DDR5 with the CPU, so the processor and GPU compete for the same bus. A big Bonsai 27B could run with a context up to 160K tokens — but it generated at around 10 tokens per second. The limit wasn’t space for the model file. It was the cost of shipping data around during generation.

I’d already seen the pattern before settling on Bonsai. A 27B Qwen on Phobos fit in memory with roughly 16K of context but crawled at about 13 tok/s; a smaller 9B hit only ~17 tok/s — not enough of a difference to justify giving up the bigger model’s quality and context. Both were far too slow for a comfortable multi-step agent loop.

So Phobos stopped being a coder and became something else: a specialist for large inputs and short answers. Not “too weak” in some vague sense — limited in a very specific way. It ingests big contexts happily; it just shouldn’t be asked to generate for very long.

Offloading and MoE: how a “35B” model lives on a 16 GB card

Offloading lets part of the model stay in RAM instead of demanding everything live in VRAM. That opens the door to models that would never fit on the card — at a price: data has to cross a slower memory path, so generation slows down noticeably.

For Mixture-of-Experts models, the picture gets interesting in a different way. Total parameters and active parameters per token are two completely different numbers. Qwen3.6-35B-A3B has 35B parameters total, but only about 3B are active when generating a single token — which is exactly why it feels responsive on a 16 GB card despite the intimidating name.

The catch, as always: the rest of the model doesn’t vanish from your memory requirements. All the expert weights, the router, the cache, and the offloading implementation still have to be somewhere. Total parameters still determine how much memory the model eats; active parameters determine how fast it thinks.

My conclusion from the cluster is simple: offloading isn’t a magic trick for a free bigger model. It’s a tool for moving the tradeoff — less demanding VRAM, more dependence on RAM and bandwidth. And it’s good enough to make genuinely useful models fit even a modest 12 GB VRAM / 32 GB RAM box, which is worth remembering.

MTP: when the model answers with silence

I also tested models using Multi-Token Prediction. The idea is attractive: predict more than one token at a time, verify, and speed up generation. In practice, MTP brings extra memory requirements and a dependency on the whole chain — backend, server, and calling format all have to support it.

My problem wasn’t that MTP is universally useless. It was that the specific Qwopus Coder MTP variants returned empty responses in the little-coder scenario that requires function calling. content_len=0. Not a wrong answer — no answer at all. Both reasoning modes, all five tasks.

The speedup was also modest: on my hardware, roughly +2 tokens/s for a real chunk of extra RAM. That was enough to retire those variants from agent duty in Olympus.

I don’t treat that as a universal verdict against MTP. It was a side lesson about the compatibility of the whole chain: a model, a backend, a server, an API and a scaffold have to work together. A model that generates beautiful prose in a chat window can be completely broken in an agent loop — and only one of those matters here.

The four questions

By this point, model selection stopped being a contest for the biggest parameter count. Every candidate had to answer four questions, in order:

Does the model fit in the available memory?
Is there enough room left for context and cache?
Does the bandwidth allow the intended role?
Does the model work correctly in the actual agent workflow?

Only answering all four told me whether a model earned a place in the team. The fact that the server answered the first prompt was the beginning of the test, not the end of it.

Benchmarking instead of guessing

I didn’t want to pick models based on file size, a few flashy answers, or the marketing section of a model card. I needed to test them in the same environment they’d actually work in — with tools, file edits, tests, and the constraints of a local workflow.

I used little-coder as the shared scaffold: it runs a model in task mode and lets it read files, edit code, and run tests on its own. The test was 140 Python tasks from Exercism, run sequentially. For Qwen3.6 I additionally checked two modes: reasoning on and reasoning off. The Qwen full-suite runs are dated 2026-07-22, and all speed figures are measurements from my specific hardware and config, not benchmarks from a model card.

The methodology mattered more than I expected. At one point I ran part of the suite through direct API calls instead of the scaffold. The results looked interesting — interesting enough to make me suspicious. They weren’t comparable with the scaffolded runs, so I marked them invalid and threw them out. That was a good lesson: a benchmark measures not just the model, but the entire chain — scaffold, prompt, tools, timeouts, result parser, and server configuration. Change any link in the chain and you’re measuring something different.

The results, and what they don’t mean

Model	Machine	Test scope	Result	Verdict
Qwen3.6 35B A3B vanilla	Mars	140 Exercism, reasoning off	126/140 — 90.0%	main local coder
Qwen3.6 35B A3B vanilla	Mars	140 Exercism, reasoning on	120/140 — 85.7%	reasoning only for selected harder tasks
Ornith 1.0 35B	Deimos	5-task smoke + Pi subagent	5/5 — 100% with scaffold	code-quality specialist
Bonsai 27B (Q2_0)	Mars / Phobos	5-task smoke	5/5	works, but a smoke test isn’t a role
Bonsai 27B (Q2_0)	Phobos	20 Exercism	9/20 — 45%	large context, short simple tasks, no repair loop
Qwopus Coder MTP	Mars / Phobos	5-task smoke	0/5 — empty responses	rejected in this tool-calling scenario

This table needs an important caveat: not all rows measure the same thing. Qwen’s 90% comes from the full 140-task suite, run on 2026-07-22; the Bonsai 20-task run on Phobos is dated 2026-07-24. The 5/5 rows are short smoke tests — they say the model passed a few chosen cases, but they can’t be compared with a full benchmark. I don’t treat these numbers as a single ranking, and neither should you. (Deimos’s Ornith, for the record, also carries strong external numbers — ~75.6% on SWE-bench Verified — which is why it earned the “code-quality specialist” title despite only a smoke test locally.)

Reasoning: more tokens, not always better answers

The most interesting result wasn’t a pass rate. It was that Qwen3.6 scored better with reasoning off: 90.0% versus 85.7% with reasoning on. That doesn’t mean reasoning is useless. With reasoning enabled, the model rescued a handful of tasks that need multi-step planning, parsing, or searching. But it also burned time and context on tasks that the plain mode solved without breaking a sweat.

The category breakdown makes the pattern visible:

Category	reasoning off	reasoning on
string	100%	100%
parsing	90%	100%
algorithm	79%	83%
OOP / state-machine	85%	70%
interpreter / compiler	25%	50%

Reasoning-on pulled ahead where the work was genuinely multi-step, and fell behind on well-structured problems where overthinking is just expensive noise. Both modes failed the same seven exercises no matter what — the interpreter/compiler and complex-OOP territory that even the bigger models couldn’t crack.

The practical conclusion: reasoning is not an “always on” setting. For short CRUD, simple transformations, and speed-sensitive work, plain mode wins. For a genuinely hard algorithm, I can accept the higher cost and flip reasoning on — remembering that longer reasoning can itself become a constraint.

Failures were as informative as results

The empty responses from Qwopus MTP in a function-calling test told me the model might be perfectly good somewhere else — just not in the chain I wanted to build. And Bonsai passed its short smoke test but couldn’t self-correct over a longer set. Both changed how I thought about the roster. I stopped asking “which model is best?” and started asking “which model is good enough for this specific role, at this speed, this memory footprint, and this way of calling it?”

That’s the question that produced the final split:

Mars / Qwen3.6 35B — the default worker for most tasks and the interactive agent loop;
Deimos / Ornith 35B — the quality specialist for harder modules;
Phobos / Bonsai 27B — the large-context endpoint for big inputs and short outputs;
Iris — the cloud planner and independent reviewer;
models that can’t survive a specific scaffold stay out of the active roster, even if they generate correct text in some other mode.

Phobos: the best lesson about matching model to hardware

I lost the most time not on Mars or Deimos, but on the smallest machine. The HP OmniBook with a Radeon 890M and 32 GB of unified memory looked wonderfully flexible: all 32 GB shared between CPU and GPU, so in theory I could run models bigger than anything on a 16 GB card. It took me embarrassingly long to learn that capacity and bandwidth are two different limits.

The memory bus is shared, so a model can load correctly and still generate slowly — every new token means re-reading large chunks of weights and cache. Phobos wasn’t weak in a simple sense. It was limited in a specific sense: excellent at absorbing large inputs, bad at generating for a long time.

So Bonsai 27B Q2_0 became its model — a 27B parameter file of about 6.7 GB, with a context of up to 160K tokens, generating at about 10 tok/s. And the role changed with the hardware:

exploring a larger codebase;
dependency and architecture analysis;
code review with a big diff;
SDD init and explore phases (more on that in part three);
one-off questions that require reading a lot of context and returning short conclusions.

The benchmark showed this boundary very clearly. Bonsai passed all five simpler smoke tasks. Extended to 20 Exercism tasks, it passed 9 — 45%. And here’s the kicker: three additional attempts after an error message didn’t recover a single one. All nine passes came on the first attempt. The model could write a correct solution to a straightforward task, but it was a poor candidate for iteratively debugging complex code.

These results aren’t contradictory — they measure different questions. The smoke test asks “can this model do a few simple tasks?” The 20-task test asks “can I hand it varied work and expect self-correction?” The answer to the first was yes. The answer to the second was no, not in this configuration.

So Phobos became an API endpoint, not another autonomous subagent. That’s one of the more important decisions in Olympus: not every machine has to do everything. Bonsai got the consultant role for large-context questions, while faster workers — and Iris, when the cloud was needed — took over implementation and iterative fixes.

The takeaway is worth framing carefully: a model that fits in memory is not yet a well-matched model. A well-matched model is one whose limits fit the kind of task. Treat a small model not as a general-purpose agent from a given domain, but as a precise operator with a well-described process — and surprisingly small models can deliver good results.

The takeaways

“It fits in memory” answers the wrong question — memory is weights + KV cache + buffers + system, and bandwidth is a separate limit entirely.
Quantization and KV cache type are real budget levers — Q2_0 fits a 27B model into 6.7 GB, but you pay for it in reasoning; dropping KV cache precision is nearly free on smaller models.
A benchmark measures the whole chain, not just the model — same model, direct API calls vs. scaffold, and you get non-comparable results. Mark the invalid runs and move on.
Reasoning is a tool, not a mode — 90% with it off, 85.7% with it on: it rescues hard tasks and wastes tokens on easy ones.
A smoke test is not a role — 5/5 said “can run simple tasks”; the 20-task run said “cannot self-correct.” Both are true, and they map to different jobs.

What’s next

The benchmark didn’t crown a winner. It produced a division of labor — which turned out to be the whole point. In part three I cover what turns these endpoints into an actual team: assigning roles, the Spec-Driven Development pipeline that moves a task from idea to archived change, the routing skill that decides which model works on which phase, and why the correct answer to “which model should I use?” is usually “it depends on the role.”

Meanwhile, if you’re building a similar setup on your own hardware, I’d genuinely like to know which limit bit you first: memory, bandwidth, or model quality. My inbox is open.