Ferrox Labs · Field Notes · № 1

Better code than the frontier models wrote, at 1/29th of what they charge.

20 systems. 196 machine checks. Our gated pool matched the best on correctness, beat every frontier lane we profiled on maintainability and runtime, and did it at $0.0063 a task. The most expensive frontier model never even finished. It's not the model. It's a gate.

Better code. Faster code. A fraction of the bill. 4 leading lanes, formatting normalized before scoring; all 4 at 100% visible correctness. Objective re-score run; Anvil aggregates cover the 3 of 5 tasks that completed generation, timeouts disclosed in the repo. maintainability index (higher is better) Anvil 51.6 opus-4-8 47.5 gpt-5-6-sol 42.7 gpt-5-6-luna 44.0 measured runtime ms (lower is better) Anvil 103 ms opus-4-8 139 ms gpt-5-6-sol 157 ms gpt-5-6-luna 349 ms run cost USD (lower is better) Anvil $0.012 opus-4-8 $0.251 gpt-5-6-sol $0.453 gpt-5-6-luna $0.082 github.com/FerroxLabs/gatebench generated by assets/generate_charts.py from results/results-objective.json
The headline, in bars. 4 leading lanes, all at 100% visible correctness, formatting normalized before scoring. Anvil tops maintainability, runtime, and bill. Like every chart here, generated by a committed script from the raw results; regenerate it yourself.
10 systems scored 100%. Here's what each one paid. 5 tasks, 133 visible machine checks, measured spend per task. Fold run, every lane 5/5 tasks completed. correctness: all tied at 100% measured cost per task (USD) 0 50 100 $0.05 $0.10 $0.15 100 Anvil $0.0063 · baseline 100 gpt-5-6-luna $0.0160 · 2.5x 100 deepseek-v4-pro $0.0265 · 4.2x 100 opus-4-8 $0.0388 · 6.1x 100 fugu $0.0399 · 6.3x 100 gpt-5-6-terra $0.0427 · 6.7x 100 gpt-5-6-sol $0.0799 · 12.6x 100 claude-sonnet-5 $0.0807 · 12.7x 100 kimi-k3 $0.1744 · 27.5x 100 gemini-3-1-pro $0.1829 · 28.9x avg visible-gate % (133 checks per lane) linear scale; multiple vs Anvil on each bar github.com/FerroxLabs/gatebench generated by assets/generate_charts.py from results/results-full.json + results-pass2.json
Same finish line, different bills. All 10 lanes at 100% visible; measured spend per task, heat-colored by the multiple against the gated pool.

We spent months arguing that the AI industry optimizes the wrong number, and this month we finished the benchmark that settles it for us. 20 systems. 5 algorithmic tasks. 196 machine checks, 63 of them hidden from every builder. The system that produced the best code by the objective measures we ran, and matched the best on correctness, was not a frontier model. It was a pool of low-cost models wrapped in a machine gate, running a loop we call the gated climb. Its bill: $0.0063 per task. The frontier lanes that matched its correctness charged 2.5 to 29 times more. The most expensive frontier model in the lineup never finished at all.

Run the arithmetic at production volume and it stops being academic. 1,000 accepted-grade tasks a day costs about $6 on the gated climb. The frontier lanes that hit the same correctness bar cost $16 to $183. Per day. Same tasks, same checks, same finish line.

Quality is the part that surprised even us. This is not a budget option that's respectable for the price. Once we normalized formatting and measured substance, the gated pool's code had the highest maintainability and the fastest runtime of the 4 leading lanes we profiled, above the frontier models, with clean security findings. Better code. Fraction of the bill.

The number that decides an agent stack is not cost per token. It's cost per accepted task, weighted by what the accepted work is actually worth. Everything below is mechanism, war stories, and receipts.

Correctness vs. cost: 20-lane fold, 5 tasks, 133 visible checks Each point is one lane: avg visible-gate % (y) vs measured avg cost per task (x, log scale). Top-left wins. Excluded as flagged, not scored: fable-5 (router content-filter bug), hermes (empty output, $0 measured spend). 50 60 70 80 90 100 $0.001 $0.002 $0.005 $0.01 $0.02 $0.05 $0.1 $0.2 $0.5 measured avg cost per task (USD, log scale) avg visible-gate % Anvil $0.0063 claude-haiku claude-sonnet-5 deepseek-v4-pro fugu fugu-ultra gemini-3-1-pro glm-5-2 gpt-5-6-luna gpt-5-6-sol gpt-5-6-terra grok-4-5 kimi-k3 minimax-solo mistral-large openrouter-fusion opus-4-8 qwen-plus generated by assets/generate_charts.py from results/results-full.json + results-pass2.json
Top-left wins. Every chart in this piece is emitted by a committed script from the raw result files. Regenerate and diff them yourself.

§ 01The wrong number

Forget prompt engineering, loop engineering, and graph engineering for a minute. The taxonomy is real: prompts design the exchange, loops design the cycle, graphs design the organization, and each hype cycle moves the design problem up a level. But every rung of that ladder is a way to spend compute. Not one of them tells you when to stop spending. The loop needs an exit test. The graph's controller needs "good enough." The layer that decides has no name in any of the threads, and it's the only layer with a measurable ceiling.

The honest objective function is cost per accepted task: everything you spent, divided by the number of outputs that actually passed acceptance. Failures, retries, and evaluator time go in the numerator and add nothing to the denominator, which is how a stack that looks inexpensive by token prices turns out to be a money pit by the only number the business feels. Cost per token is what the pricing page shows you because it's the number that flatters the seller.

So the question that decides your stack is: what test does an output have to pass before you accept it and stop paying? That test is the gate. Gate engineering is the design of that acceptance test: what it checks, what it costs to run, how hard it is to fool, and what the system does when it fails.

A gate gets used two ways. You can score what comes back, or you can climb it. The gated climb is our method, the Anvil method, developed at Ferrox Labs: draft with a low-cost model, run the gate, repair the named failures, keep a change only when it strictly improves, escalate only where the low-cost pool is stuck, stop at green. Gate engineering is the discipline. The gated climb is our method within it. Anvil is our production implementation of the method. 3 terms, 3 layers, and we'll be precise about which one each claim belongs to.

On lineage, honestly

Gates are not new and we won't pretend otherwise. Counterexample-guided synthesis has iterated against verifiers since before LLMs. Test-suite-based program repair is a decade-old research field. Synthesize-execute-debug loops were published in 2020, and AlphaCode was filtering thousands of candidates through test gates in 2022. What we're contributing is narrower and, we think, more useful: the outcome-cost accounting, the strict-subset ratchet, per-check escalation instead of per-task escalation, and verification receipts, measured together as a system against the frontier. And a name for the discipline, because you can't budget for something that doesn't have one. "Prompt engineering" did that job once. We're proposing "gate engineering" does it here.

The claim, made falsifiable: your system is never reliably better than its gate. Outputs can exceed what the gate measures; the gate just can't certify the excess, which means you can't bank it. The gate is the assurance ceiling. Improve the loop, the graph, or the model and you reach that ceiling faster. Only improving the gate raises it.

§ 02The gated climb: the Anvil method

A blacksmith doesn't trust the hammer. The hammer is enthusiasm. The anvil is where the work gets tested: you strike, the metal answers, and the metal doesn't care how expensive your hammer was. We named the executor Anvil because that's the whole philosophy in one word. Don't admire the output. Bend it and see if it breaks.

Here's the method, and notice how boring the loop is. The intelligence lives in the gate.

The gated climb: how one task moves through the executor task + gate script PROBE: 1 low-cost model, 1 attempt RUN GATE green? yes free tooling polish, re-gate DONE most easy tasks end here: $0.0007 no pick target: first failing check with an untried model untried low-cost model? yes surgical repair: spec + candidate + check NAMES only no ESCALATE up the model ladder the only place a frontier model gets called RUN GATE on candidate better? better = score strictly up, or a tie with strictly fewer failures yes accept: new best no discard, remember model failed this check budget spent or plateau? the climb yes STOP: return best so far, report honestly no fake green, no infinite loop generated by assets/generate_charts.py (static method diagram, no data plotted)
The gated climb, end to end. Green exits are the economics; the blue rail is the climb; the honest stop is the guarantee. Every decision in this diagram appears in the traces below.

For the record, and for the paper to formalize, the whole method fits in 10 lines:

# the gated climb (the Anvil method), complete best = generate(cheapest_model, spec); score(best) while not green(best) and calls < budget: target = first failing check with an untried model if none: consolidate once from the best track-record model, then stop at plateau model = untried low-cost models by win record, else ESCALATE up the model ladder cand = repair(spec, best.text, target, other_failing_names) # check NAMES only if cand.score > best.score or (tie and cand.fails ⊊ best.fails): best = cand remember(model, target) # never re-ask a model a check it already fumbled return best # green, budget, or plateau. The loop always terminates.

The probe is the economics. One low-cost model gets one clean attempt, and the gate decides. Green means finished: no ensemble, no critic, no frontier call. Watch it live:

# toposort: gated climb, live trace probe[minimax] 18/18 SOLVED, simple task, no ensemble needed $0.0007 # expr_interp: the task the same probe model cracks alone (48%) probe[minimax] 16/29 → ensemble → best 21/29 cheap[deepseek] target: parse-precedence 25/29 ACCEPT cheap[deepseek] target: unary-minus 29/29 ACCEPT · green · frontier never called

$0.0007 for a solved task is what task-adaptive spending actually looks like. The system spends more only when the gate says it must, and on the second trace you can watch the spending logic operate: probe fails, a small ensemble of low-cost models takes a swing, then surgical repairs target one named check at a time until green. The frontier never got a phone call.

The ratchet rule sounds obvious and isn't, and we know because we shipped the wrong version first. Our first acceptance rule was "keep any repair with fewer failures." We watched it ping-pong, live, between two wrong answers. Version A failed checks 3 and 7. Version B fixed those and broke 4 and 9. Then version A came back. Equal counts, different failures, metered spend, zero progress. The fix is one line: the score must strictly increase, and a tie is accepted only when the new failure set is a strict subset of the old one. Fixed at least one thing, broke nothing. Note the precision, because it matters: a strictly higher score can still trade failures, so the tie rule is what carries the non-regression guarantee at the check level. That one clause is the difference between a climb and a coin flip.

The repair payload is deliberately starved. What goes to a repairing model: the spec, the current best candidate, the name of the target check, and the names of up to 8 other failing checks. What never goes: the gate's source, its expected values, its assertion logic. If a candidate comes back and the gate scores it worse, it's discarded without appeal, and the model that produced it is never re-asked about a check it already fumbled while that check still fails. Models that land repairs move to the front of the queue. Bookkeeping, not intelligence, and it's what stops the loop burning budget on a model that has proven it doesn't understand check 12.

Escalation is per check, not per task. Only when every low-cost model has had its shot at one stubborn check does the loop hand that check, that one check, up the escalation ladder to a frontier model. That's where the cost curve comes from, and it hands you something the graph-engineering threads call an open problem: credit assignment. When a run needs the frontier, the climb log names exactly which check needed it.

And the budget is a bound, not a boast. The whole climb is capped at about 12 calls, then it stops: green, budget, or plateau, always. We'll make the modest claim and not the grand one: a 12-call loop bounds how much a builder can learn about the gate's blind spots per task. It does not make the gate ungameable, and §06 is honest about what does and doesn't. What the cap certainly buys is the halting guarantee. The loop cannot thrash, because it isn't allowed to.

Law 1

The gate is the assurance ceiling.

Output can exceed what the gate measures, but you can't bank what nothing certifies. Invest in gate fidelity before loop cleverness or model pedigree.

Law 2

Ratchet, not retry.

Score strictly up, or same score with strictly fewer failures. Nothing else lands. Progress is monotonic by construction, at the check level, not just the score level.

Law 3

Escalate per check, not per task.

Frontier models are a scalpel for the one check the low-cost pool can't clear. That's the entire cost curve, and failure localization comes free.

§ 03The verdict in full

The design brief for the benchmark was density, not breadth. 5 tasks, on purpose: small enough that every task could carry an unusually deep gate. 133 visible checks builders could climb against, plus 63 hidden mutation checks no builder ever saw, the overfit detector. That's 196 machine-checkable verdicts across 5 tasks, which is a very different instrument from 1,000 tasks skim-checked by exact-match. Single-file Python is the friendliest terrain for machine gates, and we chose it for exactly that reason: prove the ceiling where gates are strongest, then earn the harder domains one gate at a time. We'd rather you read that as a design choice here than discover it as a caveat later.

Provenance, because the method is most of the argument: this began as a lab harness to test our own premise before we bet our stack on it. We wrote the tasks, we wrote both check sets, and our executor is one of the 20 lanes. The gates and the method were developed in the same lab, and we won't insult you by claiming a firewall we can't prove. What we can prove: every gate was validated against a known-good reference that must pass and a known-bad mutant that must fail, before it scored anything. The same gates scored every lane, frontier included. No builder saw the hidden checks. And the whole fold is public, so the strongest form of trust is available: run it yourself, or write a hostile gate and break us.

The obvious question about hidden checks: aren't they the visible ones reworded? Both sets ship in the repo, so measure the overlap. And the solo lanes dropped hard from visible to hidden, which only happens when the two sets catch different failures.

Scoring ran 3 ways: visible gate, hidden gate, and a 3-model cross-provider judge panel on a locked rubric. Then a 4th pass on the 4 leading lanes: normalize formatting with an auto-formatter so nobody wins on prettiness, then measure lint findings, security findings, cyclomatic complexity, maintainability, and actual runtime. One thing to be clear about before the table: the machine gates are deterministic and we trust them. The judge panel is heuristic, we include it because the industry uses judges, and one of this benchmark's findings is precisely how badly a judge can mislead. Deterministic gates and model judges are not the same kind of instrument, and this article never pretends they are.

Here's the whole experiment in 5 sentences. We took one low-cost model, the one our executor probes with first, and ran it bare: 87.6% on the 133 visible checks, 83.8% on the 63 hidden ones. Then we ran models of the same price class through the gated climb: draft, gate, repair the named failures, accept only strict improvement, stop at green. The pool finished at 100% visible and 98.4% hidden. On the task that cracked the bare model hardest, 48% visible and 27% hidden, the climb finished 100/100.

The hidden number is the one to stare at. A system that climbs visible checks will pass visible checks. That proves nothing; it's by construction. But climbing the checks it could see fixed failures on checks it could not. To be precise about what we measured: the whole method moved, gate plus climb plus escalation together. Splitting out each piece's contribution is ablation work, and it belongs to the paper we're preparing. What this benchmark establishes is the system-level result, and the system-level result is not close.

The complete fold, split by who cleared every visible check, cheapest first. Anvil is the cost baseline.

TABLE A · CLEARED ALL 133 VISIBLE CHECKS (10 lanes) · ordered by cost per task
#lanehidden %judge /10$ / task (× Anvil)
1Anvil (the gated climb)987.85$0.0063 · baseline
2gpt-5-6-luna1008.10$0.0160 · 2.5×
3deepseek-v4-pro997.27$0.0265 · 4.2×
4opus-4-8987.47$0.0388 · 6.1×
5fugu1008.73$0.0399 · 6.3×
6gpt-5-6-terra1008.35$0.0427 · 6.7×
7gpt-5-6-sol1008.88$0.0799 · 12.6×
8claude-sonnet-5988.10$0.0807 · 12.7×
9kimi-k31008.82$0.1744 · 27.5×
10gemini-3-1-pro1007.90$0.1829 · 28.9×

Every row on this table reached the same finish line. The bill is the only real difference: 6 lanes matched or edged Anvil on the hidden gate, and every one of them paid 2.5× to 28.9× more per task to do it.

TABLE B · FELL SHORT OF 100% VISIBLE (8 lanes) · ordered by correctness
lanevisible %hidden %judge /10$ / task (× Anvil)
glm-5-2991007.55$0.0191 · 3.0×
grok-4-599988.05$0.0263 · 4.1×
claude-haiku95896.17$0.0091 · 1.4×
minimax-solo (Anvil's probe model, bare)88846.72$0.0019 · 0.3×
fugu-ultra80796.82$0.1046 · 16.5×
mistral-large78666.35$0.0022 · 0.3×
qwen-plus61566.03$0.0230 · 3.6×
openrouter-fusion59588.75$0.3191 · 50.4×

Correctness isn't bought. The bottom row is the most expensive lane in the entire fold, 50.4× Anvil's bill, and it passed 59% of the checks. Note its judge score: 8.75, higher than 8 lanes that got everything right. That's finding 3, sitting in plain sight.

Flagged, not scored (2 lanes)

hermes, a mixture-of-agents product, was invoked on every task and never completed one: each run hung past our timeout with no output, including a trivial warm-up task, at $0.000 measured spend. That reads as a pipeline failure on its side of the wire, not a capability result, and we didn't diagnose a third party's outage on benchmark time. fable-5, the most expensive frontier model in the lineup, returned empty output behind a content filter on 4 of the 5 security-probing specs; on the 1 spec that ran, it posted a perfect gate score. Neither earned a zero. Neither got to compete. Both stay out of the tables so their absence is a fact, not a statistic. Full detail on the Fable case below.

Four ways to spend: same tasks, same checks The 20-lane fold grouped by approach: averages per group over the same 5 tasks and 133 visible checks. The gated pool is the only group that pairs a 100% visible average with a sub-cent cost per task. Outside these four groups: glm-5-2 (ungrouped) and the two flagged lanes, fable-5 and hermes (not capability scores). avg visible-gate % (higher is better) Gated pool (Anvil) · 1 lane 100.0% Frontier solo · 9 lanes 99.9% Fusion systems · 3 lanes 79.7% Solo, low-cost tier · 4 lanes 80.3% avg cost per task, USD (lower is better) Gated pool (Anvil) · 1 lane $0.0063 Frontier solo · 9 lanes $0.0742 Fusion systems · 3 lanes $0.1545 Solo, low-cost tier · 4 lanes $0.0090 generated by assets/generate_charts.py from results/results-full.json + results-pass2.json
Four ways to spend. The same tasks and the same checks, grouped by approach. The gated pool holds the frontier's correctness at the low-cost tier's bill.

Then there's the lane we had to flag instead of score. Fable 5 didn't make the cut, and "folded" means something specific: its output came back empty behind a content filter on 4 of the 5 specs, so there was nothing to evaluate and the lane is flagged, not counted as a zero it earned. The benchmark pokes at security behavior on purpose. Malformed input, path traversal, open-redirect tricks. The filter saw the content, not the intent. On the one spec that ran, it posted a perfect gate score. A safety filter is a gate too, and a miscalibrated one doesn't make anyone safer. It just leaves a very good model outside the arena.

§ 04What it wins you, exhibit by exhibit

One claim per chart. Every chart is generated by a committed script from the raw results; regenerate and diff them at will.

The price of a correct task

10 lanes reached 100% on the visible gate. Here's what each one paid to get there. This is the whole cost-per-accepted-task argument in one picture: when correctness ties, price decides, and it isn't a contest.

The price of a correct task: 10 systems at 100%, one at $0.0063 Every lane below cleared all 133 visible checks across the same 5 tasks. Bars are measured avg cost per task (5-task total / 5), cheapest first, with the cost multiple vs the gated pool. Same fold as the full 20-lane table. $0.05 $0.10 $0.15 Anvil $0.0063 · baseline gpt-5-6-luna $0.0160 · 2.5x deepseek-v4-pro $0.0265 · 4.2x opus-4-8 $0.0388 · 6.1x fugu $0.0399 · 6.3x gpt-5-6-terra $0.0427 · 6.7x gpt-5-6-sol $0.0799 · 12.6x claude-sonnet-5 $0.0807 · 12.7x kimi-k3 $0.1744 · 27.5x gemini-3-1-pro $0.1829 · 28.9x measured avg cost per task (USD) generated by assets/generate_charts.py from results/results-full.json + results-pass2.json
The same picture, drawn for analysis. The 10 lanes at 100% visible, cheapest first, multiples on every bar.

The quality, once prettiness stops counting

We took the 4 leading lanes, normalized formatting so nobody wins on style, and measured substance. The gated pool didn't just hold its own. It produced the most maintainable and the fastest-running code of the 4, with clean security findings. Beyond the frontier lanes on the quality axes we measured, not merely near them.

Objective substance: black-normalized, then measured (results-objective.json) Formatting equalized with black before scoring. Anvil aggregates cover its 3/5 completed cells (expr_interp and csv_parse timed out at generation in this run and are recorded as failures); the other lanes cover 5/5. Runtime is a per-task stress workload. Maintainability Index (radon, avg) → higher is better Anvil (gate-first) 51.6 opus-4.8 47.5 gpt-5.6-luna 44.0 gpt-5.6-sol 42.7 Measured runtime (ms, avg) → lower is better Anvil (gate-first) 103.4 ms opus-4.8 139.0 ms gpt-5.6-luna 348.7 ms gpt-5.6-sol 156.6 ms Run cost for the same generations: Anvil $0.012 · gpt-5.6-luna $0.082 · opus-4.8 $0.25 · gpt-5.6-sol $0.45 generated by assets/generate_charts.py from results/results-objective.json
Substance, formatting-normalized. Objective re-score run, 4 leading lanes. The gated pool's aggregates cover the 3 tasks that completed generation in this run; 2 generation timeouts are recorded as failures in the raw JSON, disclosed, not hidden. Costs in this run differ from the fold's ($0.012 vs $0.0063 per task): different runs, separately metered, both shipped.
lanecorrect (visible)maintainability ↑runtime ↓avg cost/task ↓
Anvil (gated pool)100%51.6103 ms$0.012
claude-opus-4-8100%47.5139 ms$0.251
gpt-5-6-sol100%42.7157 ms$0.453
gpt-5-6-luna97%44.0349 ms$0.082

The lift itself

Same price class of model, method on versus method off. Bare: 87.6% visible, 83.8% hidden. Through the climb: 100% and 98.4%. The worst solo crack, 48/27, finished 100/100.

The gate is the lift: same pool, ungated vs. gated minimax-solo is the gate-first executor's own first-probe model run bare, one-shot. Anvil wraps the same pool of models in the visible machine gate with a non-regressive climb. Averages over the same 5 tasks; the hidden gate is never shown to any builder. 0 25 50 75 100 87.6 100.0 visible gate % (133 checks) 83.8 98.4 hidden gate % (63 checks) ungated: minimax-solo, one-shot gated: Anvil, same pool + gate Worst solo crack: expr_interp 48% visible / 27% hidden. gated: 100% / 100%. Cost, all 5 tasks: ungated $0.0093 · gated $0.032 generated by assets/generate_charts.py from results/results-full.json
Method on, method off. The bare lane is Anvil's own first-probe model, one-shot. What we measure here is the whole method's lift; the per-component ablation is the paper's job.

Against the other multi-model systems

Fusion products attack the same problem from the opposite direction: blend many models' opinions and trust the consensus. Respect where due: fugu hit 100% on both gates, including a hidden score above ours, and it's the only rival that did. It paid 6.3× our bill for it. The rest of the category didn't survive contact: fugu-ultra at 80% for 16.5×, and openrouter-fusion at 59% for 50.4×, the most expensive lane in the entire fold with barely half its checks passing. Consensus is expensive and it isn't verification.

Multi-model vs multi-model: the gate is the difference Four systems that combine multiple models on the same 5 tasks: three fusion/ensemble products vs the gated-pool climb (Anvil). Fusion merges opinions; the gate verifies output against the visible machine contract and climbs until it passes. 0 25 50 75 100 100.0 98.4 Anvil $0.0063/task · baseline 100.0 100.0 fugu $0.0399/task · 6.3x vs Anvil 80.0 78.6 fugu-ultra $0.1046/task · 16.5x vs Anvil 59.2 58.4 openrouter-fusion $0.3191/task · 50.4x vs Anvil visible gate % (133 checks) hidden gate % (63 checks, never shown to any builder) hermes (MoA agent): empty output on every task, flagged, not plotted. generated by assets/generate_charts.py from results/results-full.json + results-pass2.json
Multi-model vs multi-model. Blend-and-trust against gate-and-verify, same tasks, same checks.

The judge that pretty code broke

Our judge panel is 3 models from 3 providers scoring against a locked rubric, about as hardened as an LLM judge gets. It still handed openrouter-fusion, 59% correct, a higher style score than 8 lanes that got everything right. Pretty code broke the judge. That finding is why the objective re-score exists, and it's why the machine gates, not the judges, are the method's foundation. If your gate is a model's opinion, your ceiling is a model's taste.

The bill, at volume

running 1,000 tasks/dayper task (fold run)per dayper month
Anvil (the gated climb)$0.0063$6.34$190
least expensive frontier lane at 100% (gpt-5-6-luna)$0.0160$16.02$481
most expensive frontier lane at 100% (gemini-3-1-pro)$0.1829$182.91$5,487

All 3 rows from the same run (the 20-lane fold), where every lane completed 5/5 tasks: the per-task figures carry no hidden failure costs. Model inference spend only; gate execution for these tasks is a subprocess that runs in milliseconds for effectively $0.

§ 05We built the fancy version. It flattered us.

Every AI engineer eventually builds the critic loop. Generate, critique, refine, repeat. It feels rigorous. It demos beautifully. So we built 2: a single post-green polish pass, and a full critic-guided refinement loop on top of it. Multi-round review, quality scoring, the works.

The judge panel loved the output. Scores went up.

The hidden checks did not move. The polish tiers cost multiples of the base climb and gave back nothing on the measurements that matter, and polish-stage rewrites are exactly where a regression can slip in behind a green visible gate. So every transform in our stack now has to re-pass the full gate before it's kept. A rewrite that drops a check is discarded. No appeal.

Then the genuinely annoying part. We ran a free deterministic formatter over the plain output, the kind of tool that changes zero execution behavior, and most of the judge's perceived quality gap closed on its own. We had been about to charge tokens for what a formatter does for nothing. We cut our own feature and kept the free one: after green, run the language's canonical format, lint, and security tooling, re-gate, keep only if nothing regressed. Deterministic. Un-gameable. $0.

§ 06Who watches the gate

If the gate is the assurance ceiling, the obvious attack is on the gate itself. Here's the trust boundary, drawn as a picture:

# what the repairing model receives # what stays on the trusted side the task spec the gate's source code the current best candidate every expected value "this check FAILS: unary-minus" the assertion logic up to 8 more failing check NAMES the hidden check set, all 63 nothing else crosses. ever. the scoring, the receipts, the logs

The model has to actually fix unary-minus handling, because there is nothing else to overfit to. Judge and generator stay separate for the same reason a student doesn't grade their own exam, and the strongest gates use no model at all.

Why models cheat at tests, and what catches it

We attacked our own gates, and we'll tell you what happened. A candidate that monkeypatches the test harness at import time can flip checks green. That's not a hypothetical: our own red-team probe did it. So the claim boundary is exactly this: our gates detect ordinary defects and known-bad mutations. They are not tamper-proof against a deliberately adversarial submission. Real containment, running candidate code with no path to the gate process at all, is an architecture problem we're solving in the runtime layer, not a prompt problem. Gates catch mistakes. Sandboxes catch malice. A production system eventually wants both, and pretending the first covers the second is how verification gets a bad name.

§ 07Gate the work, not just the code

The obvious objection to all of the above: code has tests, most work doesn't, so this is a coding trick. We think that's a failure of gate imagination, and this section is the part we most want stolen. Nearly every domain admits a gate. The craft is choosing the strongest gate the domain admits instead of the weakest one that's easy to build. Our working taxonomy, strongest to weakest:

tiergatedomainshow it checks
1 · Executablerun the artifactcode, SQL, web UI, agentic tool-use, infra, numeric mathtests pass, the query returns the right rows, the side effect actually happened
2 · Formalproof / schema / solverstructured output, logic, proofsschema validation, constraint solver; fails closed, near-free
3 · Referencediff vs. goldenextraction, translation, classificationfield-level ground truth, round-trip reconstruction
4 · Groundingevery claim entailed by a cited sourceresearch, RAG, long-formclaim decomposition → per-claim entailment → support score
5 · Consistencyagreement / round-tripcross-cuttinginexpensive pre-filter only, never the final word
6 · Judgemodel panel, locked rubricwriting, design, conversationcross-provider judges on an evidence-anchored rubric; the only tier that reward-hacks, so it's the fallback, not the plan

(This is the gate taxonomy. It's a different ladder from §03's escalation ladder, which orders models by price. 2 ladders, 2 jobs.)

A worked example, because the table undersells it. Take an agentic task: book the meeting room and email the team. The weak gate asks a judge model whether the transcript looks successful. The tier-1 gate ignores the transcript and checks the world: does the calendar API now return the booking, did the email land, does the end state match the spec? "The agent said it did something" is worth nothing. "The world changed" is worth everything.

Or research, the domain everyone writes off as unverifiable. Tier 4 says: decompose the draft into atomic claims, require every factual claim to cite a source, check each claim against the passage it cites, score the support, climb on the failures. Research isn't ungateable. It's grounding-gated, and once you see it that way, "you can't test prose" starts sounding like "we haven't written the gate yet."

This week we ran that conviction through the production executor. 3 tasks, 1 verb, only the domain label changed. We'll characterize the result honestly: these demonstrate the gates, the routing, and the loop's plumbing across domains. All 3 went green on the first probe, so no cross-domain climb was exercised on a working gate. The multi-task, deliberately-failing version of this is the paper's job.

proof taskdomaingate tiergate scorehiddenroundswall clock
release manifest (JSON)structured-gen2 · schema16/16n/a1115 s
cited research briefresearch4 · grounding14/14n/a164 s
toposort (code parity)code1 · executable18/1810/10161 s

Every final artifact re-scored independently outside the executor, same numbers. The code row matches the original benchmark lane exactly: 18/18 visible, 100% hidden.

And since "you can't test prose" deserves a body, not a hand-wave, here is the actual grounding gate the research brief had to pass. 14 machine checks. A deterministic grounding proxy: citation discipline, number-level grounding, required coverage. The full entailment version, where a model verifies each claim against its source, is the production path; this is the offline-reproducible subset.

# grounded_brief gate: the 14 checks, verbatim length-18-35-sentences no-headings-or-lists has-citations citations-valid-source-ids cites-all-three-sources all-numeric-sentences-cited all-numbers-grounded-in-cited-source no-content-free-citations covers-gate-lift covers-objective-substance covers-cost-advantage covers-limits-single-file covers-polish-rejection covers-not-tamper-proof # validation before we trusted it: reference brief -> 14/14 pass. # sabotaged mutant (one number altered, one citation removed) -> FAIL all-numeric-sentences-cited FAIL all-numbers-grounded-in-cited-source FAIL covers-objective-substance 11/14, exit 2. The gate caught exactly the sabotage.

Change one digit in the brief and the gate goes red on exactly the checks that guard digits. There's a joke buried in here too: the research task we gated was a brief about this very benchmark. The gate held our own claims to account. Honest work.

One run earned a footnote we're oddly proud of. On the first code-parity attempt, a broken gate harness (our bug, a Python loader issue, since fixed and revalidated) meant no candidate could ever pass. Watch what the climb did against a gate that could never go green:

# toposort attempt 1: the gate itself is broken. Nobody can win. Watch. probe[deepseek] red → convene ensemble ensemble ×2 red → surgical repairs, one named check at a time surgical ×5 red (one model timed out; skipped, budget preserved) ESCALATE sonnet-5 red ESCALATE opus-4-8 red consolidate ×1 red stop: plateau · 9 rounds · action: reject · no fake green, no infinite loop

9 rounds, the full escalation ladder, honest surrender. If you're going to trust a loop with your money, this is the trace to demand from any vendor: what does it do when winning is impossible? Ours stops and says so.

§ 08What this can't do yet

The limits, plainly, because half of gate engineering is deciding what you're not entitled to claim. The headline benchmark is 5 single-file Python tasks with unusually deep gates: dense, but narrow, and chosen because it's the terrain where machine gates are strongest. The cross-domain proofs are 1 task per domain, all first-probe green: they prove the gates and the plumbing, not cross-domain climbing. The comparison measures the whole method against bare models; it does not isolate which component does what, and anyone who wants that ablation is asking the right question. It's the centerpiece of the paper in preparation. The gates are operator-authored, which puts our blind spots inside the assurance ceiling. Goodhart pressure is permanent: any gate a system can see, some system will eventually satisfy in letter rather than spirit, which is why the hidden checks, the starved repair payload, and the call budget all exist, and why none of them adds up to tamper-proof. And yes: run it on SWE-bench is a fair demand. Dense micro-gates were this benchmark's design choice. Big-bench terrain is the next campaign.

§ 09Where this runs

We didn't build this for an article. We benchmarked it because we were betting our stack on it, and the stack is live.

Ferrox Factory, our agentic build system, has a lineage that matters here: it's the tool we used to build Wayland and Wayland Core. As of this month, the gated climb is its default executor. Every increment declares a domain, the Factory selects the strongest gate tier that domain admits, gateable work climbs, ungateable work goes to a judge council, and everything still faces the same fail-closed merge gate as any human diff. The factory that built our products now refuses, by default, to ship work nothing verified. 714 machine checks run green on the Factory itself before any of this shipped.

FluxRouter, our model router, runs the loop server-side. The design principle is the one we build everything around: don't make me think. Send an ordinary API request; the router probes low-cost, gates, climbs, escalates only where the gate demands, and returns output with a verification receipt: which checks ran, how many passed, how many attempts, and what the same result would have cost at frontier list prices. The open engineering problem there is the gate source. Caller-supplied checks are iron-clad. Server-derived checks are the server grading homework it also wrote, and they need holdouts and cross-checks before the receipt can be honest about its own confidence. That's the frontier of this discipline.

Wayland Core, our primary agent, runs the climb in its agent engine and holds the taxonomy's last tier: Crucible, its cross-provider council, is the hardened tier-6 judge panel everything falls back to when no stronger gate exists. Agent does the work, council judges the work, gate decides. Nothing in our stack ships on vibes anymore, and the 3 products enforce that on each other.

§ 10Receipts, and what's next

A claim like "your system is never reliably better than its gate" deserves a gate of its own. The full benchmark is public: task specs, both check sets, every result record including the generated code for all 18 objectively re-scored cells, and the scripts that emitted every chart above from the raw JSON. One command re-runs every gate against the shipped outputs and confirms the recorded scores reproduce:

$ git clone https://github.com/FerroxLabs/gatebench && cd gatebench $ python verify/verify_results.py 18/18 embedded-code cells checked: 18 verified, 0 mismatched. All recorded gate percentages reproduce from the embedded code.

A formal write-up is in preparation for arXiv: the ablation that splits the method into its parts, the accounting standard for cost per accepted task, and a versioned "Gate Card" spec so a gate's rigor can be cited the way a model card is. Until then the repo is the working paper, and it takes challenges: author a hostile gate, break our numbers, send the receipt. Somebody will. That's the discipline working exactly as designed.

The hype cycle will name something new next quarter. Meanwhile the gates for research are younger than the gates for code, the self-derived-check problem is open, and the cost-per-accepted-task number in your own stack is either going down or it isn't.

Ferrox Labs builds gate-first AI systems: Ferrox Factory (agentic delivery), FluxRouter (verified model routing), Wayland Core (agents with a judge council). Benchmark method, limits, and lane-by-lane reproducibility notes live in the gatebench repository, including which lanes reproduce with a public API key and which ship as static evidence. We'd rather you re-run it than believe us.