Field Manual No.13 · 27 August 2026
Qwen 3.8 vs Ornith vs Claude
A 27B model on a laptop matched Opus on hard coding and tied it on animated 3D. Here is exactly where it didn't.
Two open-weight models running on a single RTX 5090 Laptop GPU, measured against Claude Opus 4.6, 4.8 and 5 through the API. Same prompts, same graders, same day. Every number below was produced on this machine. No vendor-reported figure is mixed into any result.
Where each model actually wins
| Model | Coding | Structured | Instruction | Retrieval | Total |
|---|---|---|---|---|---|
| claude-opus-4-8 | 100% | 100% | 100% | 100% | 100.0% |
| claude-opus-5 | 100% | 100% | 100% | 100% | 100.0% |
| claude-opus-4-6 | 100% | 85% | 95% | 100% | 96.7% |
| qwen3.8-27b | 100% | 50% | 50% | 100% | 83.3% |
| ornith-1.5-35b-a3b | 83% | 65% | 45% | 100% | 76.7% |
Qwen matches Opus on hard coding, at 100%. That is not a soft task: RFC 4180 CSV parsing with embedded newlines and doubled quotes, topological sort with cycle detection and alphabetical tie-breaking, and a binary search with a subtle lower-bound bug. All three, all cases, on a laptop.
Retrieval is also at parity: every model found three facts buried at different depths in a 180-line log and summed them correctly.
The gap is elsewhere: multi-constraint instruction following (45-50% vs 95-100%) and multi-step arithmetic aggregation (50-65% vs 85-100%).
Per-task results
| Task | What it traps | opus-4-6 | opus-4-8 | opus-5 | qwen | ornith |
|---|---|---|---|---|---|---|
| csv-rfc4180 | quoted commas, embedded newlines, doubled-quote escapes, CRLF | 100% | 100% | 100% | 100% | 100% |
| toposort | cycle detection, implicit nodes, alphabetical ties | 100% | 100% | 100% | 100% | 50% |
| debug-binsearch | non-integer mid, wrong initial hi, empty array | 100% | 100% | 100% | 100% | 100% |
| json-regroup | group, sum, round, and pick top product by revenue not volume | 85% | 100% | 100% | 50% | 65% |
| instruct-multi | 6 simultaneous constraints incl. exact word counts and a letter ban | 95% | 100% | 100% | 50% | 45% |
| needle-compute | 3 facts at depths 14/78/153 in 180 lines, then arithmetic | 100% | 100% | 100% | 100% | 100% |
The json-regroup trap is worth calling out: the top product by revenue
in Germany is Gadget (149.50), not Widget (99.95, but three units across two orders). Opus 4.6 picked
Widget. So did an earlier version of this grader. See the harness-bug section.
The bright orrery
The first 3D task was a realistic solar system on black. It graded fine but was almost unreadable: a point light at the sun leaves every planet half-dark, and at small sizes five panels of black space look identical. This is the same subject rebuilt for legibility and made harder: a violet-to-magenta gradient backdrop, four emissive planets, tinted orbit rings, a tilted torus ring on the gas giant, a nested moon, and a comet on an elliptical path trailing thirty fading segments. Identical prompt to all five, no edits to any output.
Don't take the video's word for it. Each panel below is that model's own unedited HTML. Press run and it executes live in your browser.
Open the full-screen live instrument, all five at once →
All five produced a working, animated orrery that satisfies every stated requirement. On the measurable criteria there is nothing between them: 16/16 structural checks each, all render a real scene, all provably animate.
Which one looks best is a subjective call and we are not scoring it. Judge the video yourself. For what it is worth, we think Qwen3.8-27B, the model running on the laptop, is the most visually polished of the five, but that is an opinion, not a measurement, and reasonable people will disagree.
| Model | Attempts | 1st attempt | Final | Static | Colours |
|---|---|---|---|---|---|
| qwen3.8-27b | 1 | 100% | 100% | 16/16 | 268 |
| claude-opus-5 | 1 | 100% | 100% | 16/16 | 266 |
| claude-opus-4-8 | 1 | 100% | 100% | 16/16 | 159 |
| claude-opus-4-6 | 1 | 100% | 100% | 16/16 | 132 |
| ornith-1.5-35b-a3b | 2 | 60% | 100% | 16/16 | 193 |
Ornith is scored 60% on this task, not 100%. Its first attempt rendered a blank frame, and first-attempt reliability is part of what a model is being judged on. The 100% column records that a second sample of the same prompt did succeed; it is not the score.
Across both 3D tasks, the original solar system and this orrery, Qwen3.8-27B was the only model to add anything that was not asked for. In the orrery it shipped an on-screen title card (“Museum Model · Orrery No. 04”), corner framing, and two interaction handlers. In the earlier solar-system task it added hand-written orbit/zoom/pause/reset camera controls, orbital path rings, and a legend listing each planet with the radius and period taken from the prompt.
| Model | UI chrome | Interaction handlers | Output size |
|---|---|---|---|
| qwen3.8-27b | 14 refs | 2 | 20,456 B |
| ornith-1.5-35b-a3b | — | — | 13,490 B |
| claude-opus-5 | — | — | 6,188 B |
| claude-opus-4-6 | — | — | 5,161 B |
| claude-opus-4-8 | — | — | 4,700 B |
Both readings are defensible. The Opus models met the stated requirements exactly and stopped. Qwen went further, which reads as capability here, and as scope creep against a prompt that said “output only the complete HTML file”. That second reading is consistent with Qwen’s weak instruction-following score (50%) in the main suite: the same tendency that makes its 3D work look richer is what costs it marks when a spec must be followed literally.
Ornith’s first output passed all sixteen structural checks and rendered nothing. One line was responsible:
new THREE.MeshBasicMaterial({ color: 0xFFD34E })
sun.material.emissive = new THREE.Color(0xFFD34E); // MeshBasicMaterial has no .emissiveMeshBasicMaterial is unlit and has no emissive property. Assigning one
makes Three.js read uniforms.emissive.value on a shader that never declared it:
TypeError: Cannot read properties of undefined, thrown every frame, aborting the render
before any geometry draws. Deleting that one line takes the output from 2,286 bytes (blank) to 24,830
(full scene). Bisected and confirmed, not inferred.
The mechanism matters more than the bug. The prompt asked for an
“emissive” sun. Ornith picked MeshBasicMaterial, a defensible unlit
choice, then bolted .emissive on to satisfy the word. And the check meant
to catch exactly this, emissive used ≥ 3, counted the occurrences and passed it.
A checklist can be satisfied by vocabulary. Only rendering catches it.
The first replacement sample also passed the old render gate (a 30 KB PNG whose frames differed between timestamps) while actually being flat black regions and a stray magenta polygon. Byte size and frame-hash difference are not a render check.
The gate now decodes each frame to raw RGB and measures distinct colours, dark fraction and dominant-colour share. It separates cleanly: the four working renders score 132-268 distinct colours; the two broken ones scored 55 and 57. Both failing outputs ship in the package as evidence.
Blind planning vs interactive tool use
These are two different capabilities and conflating them is the most common way this kind of comparison goes wrong. Blind asks for a complete command sequence up front with no feedback. Interactive runs a real loop: one command per turn, an observation returned each time, up to 25 turns, with one retry allowed on a malformed reply.
| Model | Blind planning | Interactive loop | What the traces show |
|---|---|---|---|
| claude-opus-4-8 | 100% | 100% | Optimal or near-optimal every rep. |
| claude-opus-5 | 100% | 99% | Delivers in 10 turns; recovers cleanly from the wall. |
| claude-opus-4-6 | 20% | 88% | Blind, it routes through the wall at (3,2) and lands one square short. Given feedback, it recovers. |
| qwen3.8-27b | 0% | 0% | Walks onto the crate square and past it without picking up; later issues MOVE S twelve times into a grid edge without adapting. |
| ornith-1.5-35b-a3b | 0% | 0% | Navigates competently and maps walls correctly, but stops at (2,2) when the crate is at (2,1). |
This is the clearest capability gap in the whole comparison, and it is not a formatting problem. Both local models emit perfectly-formed commands and correctly interpret “Blocked: wall”. What they fail at is tracking their own coordinates across turns. Opus 4.6's blind score of 20% versus 88% interactive is the cleanest demonstration that feedback and planning are separate skills.
One caveat: this is a single spatial world. It is evidence about state tracking, not a general verdict on tool use. A model that fails here might still drive a well-designed tool API where the environment reports state explicitly.
What local buys you
Capability is only half the comparison. These are measured decode rates on the same laptop, tuned per the companion setup guide.
| Model | Baseline | Tuned peak | 256K ctx | VRAM @256K | Notes |
|---|---|---|---|---|---|
| qwen3.8-27b | 40.8 | 141.6 | 81.2 | 23,504 MB | Peak is JSON output at n-max 10. Code peaks at 96.8. |
| ornith-1.5-35b-a3b | 144.3 | 144.7 | 123.6 | 21,298 MB | MoE, ~3B active. Needs no tuning; MTP unavailable. |
Ornith's untuned baseline (144.3 tok/s) beats Qwen after every optimisation available (141.6), uses 2.2 GB less VRAM at full context, and holds 123.6 tok/s at 256K where Qwen drops to 81.2. It activates roughly 3B parameters per token against Qwen's 27B dense.
But Qwen wins the capability suite 83.3% to 76.7%, wins coding 100% to 83%, and ties Opus 5 on animated 3D where Ornith placed third. Speed and capability point at different models here.
Creative writing
Creative quality can only be judged, and the judge here is Claude while three of the five entrants are Claude models. That is a structural conflict and no headline number should rest on it. Constraint adherence below is machine-checked and trustworthy; the craft column is a judgement call, shown so you can disagree with it.
| Model | Constraints (machine) | Craft (judged) | Machine-checked detail |
|---|---|---|---|
| claude-opus-4-8 | 100% | 5/5 | all constraints met |
| claude-opus-5 | 100% | 5/5 | all constraints met |
| claude-opus-4-6 | 100% | 4/5 | all constraints met |
| qwen3.8-27b | 70% | 3/5 | word count overrun |
| ornith-1.5-35b-a3b | 55% | 3/5 | banned word used; final sentence wrong length |
The constraint column tracks the instruction-following result from the main suite, which is machine-verified and where the local gap is real and large. The craft ranking happens to agree with it, but agreement is not independence, and it should not be read as corroboration.
Harness bugs found and fixed
Every one of these produced plausible, publishable, wrong numbers. They are listed because a benchmark that silently mis-grades is worse than no benchmark, and because several of them inverted a result.
| # | Bug | Effect on the numbers |
|---|---|---|
| 1 | Expected answer was wrong | Inverted a result. The grader asserted Widget as Germany's top product; it is Gadget (149.50 vs 99.95). Opus 4.8 and 5 answered correctly and were marked down; Opus 4.6 repeated the error and was rewarded. |
| 2 | Thinking disabled on the API | Corrupted Opus output with mid-sentence fragments and stray CJK characters. Opus 4.8 instruction score 13% → 91% after switching to adaptive thinking at low effort. |
| 3 | Agent env omitted the crate location | Turned planning into unsensored blind search. Opus 4.6 0% → 89%, Opus 4.8 40% → 100%. |
| 4 | Stale screenshot read | PNGs from a previous run were hashed, so an animated scene scored STATIC. Opus 4.8 3D 76% → 96%. |
| 5 | Prompt echo graded as output | The line counter saw 10 lines for a 4-line answer; the code extractor matched the fence marker inside the prompt text itself. |
| 6 | Unclosed code fence | Ornith opens a javascript fence and never closes it. Its coding score went 48% → 81%. |
| 7 | Token budget starved by thinking | max_tokens: 24 with thinking on truncated every reply to two characters. All models scored 0%. |
| 8 | Health endpoint lies during load | /health returns 200 while chat still returns 503. Local reps fired into a cold server and scored as errors. |
| 9 | Missing software GL backend | Without --use-angle=swiftshader, headless Chrome silently produced blank PNGs for every WebGL render. |
| 10 | Counting geometries not meshes | Models legitimately reuse one sphere geometry across several planets; all five failed a check they had actually satisfied. |
The objective tasks are the defensible core: unit tests in a sandboxed VM, JSON schema checks, a deterministic simulator, and render-verified animation. They contain no model judgement and can be re-run by anyone. If you want to break this comparison, start there: read the graders, not the summary table.
The weakest claims, stated plainly: creative craft is judged by an interested party; the agentic result rests on a single spatial world; and each model was run once per task except the agent loop (3 reps) and the speed measurements (2-3 reps).
Reproduce it
node bench/run2.mjs api # 6 hard tasks vs Opus 4.6 / 4.8 / 5
node bench/run2.mjs local # same tasks vs local GGUF models
node bench/agentbench.mjs all # interactive agent loop, 3 reps each
node bench/threed2.mjs # animated 3D + headless render verification- Fairness settings: local models run with reasoning off; Opus runs adaptive thinking at low effort. Disabling thinking entirely on Opus corrupts its output, so it is not a valid analogue.
- Coding tasks are JavaScript because Node is the only real runtime on this box and objective grading requires execution. This mildly favours JS-strong models.
- No vendor-reported numbers appear anywhere in this document. Published benchmark scores use different harnesses and, in at least two cases checked during this work, refer to different-sized models than the ones being named.
What the vendors claim
Everything in this section is reported by the model developers or scraped from third-party aggregators. It is included because readers will look for it, and excluded from every score in this document because it cannot be compared like-for-like. Three specific reasons:
- Different harnesses. Ornith's own card reports Terminal-Bench 2.1 at 67.8 with the Terminus-2 scaffold and 68.5 with Claude Code: the same model, same benchmark, different number depending on the wrapper.
- Self-reporting. The Qwen and Ornith figures come from their own model cards. The Claude figures come from third-party leaderboards. Those are not the same kind of claim.
- Model-size confusion is rampant. Ornith-1.5's headline "beats Opus" coverage refers to the 397B sibling, not the 35B tested here. We nearly repeated that error twice while researching this piece.
| Benchmark | Qwen3.8-27B | Ornith-1.5-35B | Opus 4.8 | Opus 5 |
|---|---|---|---|---|
| SWE-bench Verified | — | 79.0 | 88.6 | 97.0 |
| SWE-bench Pro | 61.7 | 59.6 | 69.2 | — |
| SWE-bench Multilingual | — | 71.4 | — | — |
| Terminal-Bench 2.1 | 73.0 | 67.8 | — | — |
| LiveCodeBench v6 | 90.3 | — | — | — |
| GPQA Diamond | 89.2 | 89.2 | — | 94.1 |
| OSWorld-Verified | 84.3 | — | — | — |
This is the most useful thing in the section. On published SWE-bench figures, Ornith (79.0 Verified) looks stronger than Qwen. In our own coding tasks (RFC 4180 CSV parsing, topological sort with cycle detection, a subtle binary-search bug), Qwen scored 100% and Ornith scored 83%, with Ornith failing topological sort outright.
Both things can be true. SWE-bench measures patch generation against real repository issues; our tasks measure algorithmic correctness on adversarial edge cases. They are different skills. The lesson is that a leaderboard position does not predict behaviour on your workload, which is the entire argument for running your own harness.