Strix Halo Model Bench
Local LLM benchmark · Lemonade 2026.40 · llama.cpp Vulkan · AMD Strix Halo · October 2026

Strix Halo Model Bench

Every chat model on one local Lemonade server, run on the same 70 general questions and a 21-question hard set of practical coding, logic and fixed-income risk calculations. Each answer was graded 1–100 against a written rubric; code answers were also executed against test cases.

52models tested
13failed to load
4,484graded answers
33models on the hard set
91distinct questions
Summary

What to run

CodingQwen3-Coder-Next

Hard coding + logic 99.7, all 21 hard answers finished within budget. Qwen3-Coder-30B is the fast option at 97.5.

Fixed incomegemma-4-31B

Best risk calculations (97.5): price, duration, DV01, convexity, loan EL and WAL. gemma-4-26B-A4B scores 96.2 and is faster.

General purposegpt-oss-120b

Intelligence 97.9 on the main set and 99.0 on hard coding + logic. Only slightly behind gemma-4-31B (98.1) overall.

Small and fastgemma-4-E4B

5.9 GB, about 51 tok/s, intelligence 95.6. The highest combined speed and quality score of all models.

For the team meeting

Proposed decisions and open questions

Proposed decisions

  1. Coding default: Qwen3-Coder-Next. Use Qwen3-Coder-30B when latency matters more than the last few points.
  2. Finance and fixed-income work: gemma-4-31B. Use gemma-4-26B-A4B as the faster alternative. Qwen3-Coder-30B is weak here (73).
  3. Serving settings: stop using temperature 0 for thinking models. Ship each family's recommended sampling preset, at least 16k output tokens and 32k context for Qwen3.x, Ornith and Nex.
  4. Roster cleanup: remove the 13 models that fail to load. Deprioritise Qwimi-9B and MiMo-Distill-9B, which stay weak even with 32k and sampling.
  5. Uncensored builds: use them only where the use case needs them. Prefer gemma-4-31B heretic or Ornith-35B abliterated, which matched their base models.

Open questions

  • Should we re-run the full hard set with recommended sampling? Only the over-budget answers were re-run, so the 32k numbers mix two settings.
  • One sample per question gives roughly ±5 points of noise on subjective items. Is it worth 3–5 samples on the shortlist to measure variance?
  • One LLM did all the grading. Should someone hand-check a sample, especially the fixed-income and planning answers?
  • Not covered: long-context retrieval, tool calling, structured JSON output and multi-turn chat. Which of these matters for our workloads?
  • The speed numbers are for one user on one machine. Do we need concurrency and throughput tests before choosing a serving model?
Main set

Speed vs quality

Each dot is one model: decode speed on a log scale against main-set intelligence. Dot area follows file size. Hover a dot for details.

● filled: base model○ ring: uncensored / abliterated variantcolour: summary score
Main set · 70 questions · 8k output cap

Overall ranking

Summary = 0.7 × Intelligence + 0.3 × Perf index. Perf index combines decode tok/s (50%), time to first token (25%) and cold load time (25%) on fixed log scales. Click a column to sort.

Main set · programming + logic

Coding & logic ranking

Practical = 0.7 × Code+Logic + 0.3 × Perf index. Ignores general knowledge and business writing.

Main set · real time and answer length

Efficiency ranking

Efficiency = 0.6 × Intelligence + 0.25 × time score (median seconds per answer, log 5 s → 300 s) + 0.15 × size score (median output tokens, log 150 → 6000). Useful answers per hour = sum of score/100 over total answer time.

Hard set · 21 questions · 32k context · 16k output cap

Hard set: coding, logic, fixed income

8 coding tasks (bug fixing, SQL window functions, a thread-safe queue, an expression parser, asyncio, pytest, Dijkstra, a 5 GB CSV), 5 logic puzzles and 8 fixed-income risk calculations from projected cash flows. The 32k columns replace answers that ran over budget with their re-run grades.

Recommended sampling · 32k output budget

Re-runs

Only answers that hit the output cap were re-run, so these scores mix greedy and sampled answers. They show what a model does with enough room, not a clean second benchmark.

Qwen3.8-27B main-set cut-offs at 32k

Base vs uncensored / abliterated

Uncensored variants

Δ = variant minus base, in score points. Changed answers = questions where the score differs by 30 points or more. Hard 32k columns use the re-run scores.

Answers that hit the output cap

Over budget: loop, cycling or real limit

Loop: verbatim repetition. Cycling: keeps re-checking or announcing the answer without progress. Budget: still making new progress when cut off, so more tokens would help. Classified from the last 12k characters of each answer.

How it was measured

Method and caveats

Runs

  • One model loaded at a time; questions sent sequentially over the streaming chat API.
  • Main set: 70 questions in 7 areas (finance, programming, math, logic, marketing, planning, general knowledge), 8,192 output tokens, temperature 0.
  • Hard set: 21 questions, context 32,768 and 16,384 output tokens, temperature 0, on the strongest models and every base/uncensored pair.
  • Re-runs: context 49,152, 32,768 output tokens, each family's recommended sampling (for example Qwen 0.6 / top-p 0.95 / top-k 20, gpt-oss 1.0 / 1.0, gemma 1.0 / 0.95 / 64).
  • Speed: decode tok/s and prompt time from server timings, TTFT measured at the client. Load time is the cold load; repeated loads are faster because of file caching.

Grading

  • Every answer graded 1–100 by Claude against a written rubric with anchors at 100 / 70 / 40 / 10. Expected numeric answers were verified in Python first.
  • Coding answers for the queue, parser and Dijkstra tasks were executed against test cases; the queue stress test ran three times to catch races.
  • An answer cut off with no final text scores 5; one cut off mid-answer scores 10. Any answer that hit the cap is capped at 20.
  • Thinking models are charged for their reasoning tokens in time and size, because that is the time you wait.