Benchmark report

01 At a glance

02 How to read this report

Two words this report cannot do without

Harness

The scaffolding a model is run inside on an agent benchmark: the loop that gives it tools, feeds back results and decides when to stop. The same model scores very differently under different harnesses, so a measurement is only meaningful if the harness is named. Where a publisher does not name one, this report says "as published" rather than inventing one.

Effort

The reasoning setting a model was run at, where the publisher exposes one, for example low, high or max. It changes both cost and score, so it is part of what was measured.

Two entities, and they are not the same claim

A system is one exact combination of model, harness and effort. It is the rigorous unit: two measurements are joined only when all three match, so a run whose harness was never published is never merged with one whose harness was.

A model is that model across every configuration measured. Rolling up to it is a declared operation with a stated rule, not a guess, and every model row names how many configurations sit behind it and which harnesses they used.

The rollup rule is stated, not implied: median observed configuration per benchmark (lower median, always a real measurement). Taking the best configuration instead would reward a model for having been measured more often, because the maximum of more draws is higher whether or not the model is better.

What has to be true before a score is published

A Ferrox Index score requires at least 2 qualifying benchmarks in each of agents, coding, reasoning. Systems below that bar appear on the category boards with their evidence count shown, and have no overall score.

The evidence flag on every row

• FULL Two or more qualifying benchmarks from genuinely different correlation families.
• THIN Enough benchmarks, but they come from one family, so they are correlated rather than independent. A category qualifies on 2 or more qualifying benchmarks. If those benchmarks all come from the same correlation family, the category is THIN: it is scored and ranked, and it carries a visible flag saying the evidence behind it is correlated rather than independent. Excluding those systems instead would hide most of the field behind a gate the reader cannot see.
• SINGLE One benchmark. Shown on the category board with its count, never given an overall score.

What the interval does and does not say

The interval printed beside an index is built only from the benchmarks that published one. Five of the benchmarks scored here publish no uncertainty at all, so for most rows the interval is a lower bound: the true uncertainty is at least this wide and can only be wider. Rows in that state are marked "at least". Publishing no interval at all would have thrown away the real information the other sources did provide.

How rows are ordered

Rows are ordered by the point estimate. The interval is printed beside it and is never used as the sort key. Sorting by the top of the interval instead would make a wider interval a rank bonus, and would reward a source for publishing a vaguer number.

03 The board

Models, rolled up across every configuration measured. 37 of 610 models clear the entry requirement.

Table scrolls sideways

Rule cost is how many normalised points this model would have gained had the board taken its best configuration instead of its median one. A large value is not a defect in the model and not a defect in the board: it means that model's configurations disagree with each other by a lot, so any single number for it is standing in for a wide range. Read a high rule cost as a warning that the row is less settled than its neighbours, and note that under the best-of rule those same rows would have been ranked higher on nothing but the spread of their own measurements.

Not separable marks a pair this evidence cannot order. It appears where one row has a measured preference score and the other has that category imputed, and the two sit within 1.78 index points. Section 09 gives the measurement behind that number. The band is applied one pair at a time and never chained into tie groups, because chaining is not transitive and would collapse rows tens of points apart into a single tie.

04 The strict board

The same rubric applied to the rigorous entity: one exact (model, harness, effort) tuple, with nothing rolled up. 2 of 1147 systems qualify.

What would fill it

Of 1147 systems identified, 2 hold enough evidence in all three required categories, 26 hold it in two, 245 in one, and 874 in none. 14 systems are a single measurement away from qualifying.

Every near-miss system has no harness recorded. A benchmark that publishes a harness column would create a new system rather than complete one of these, so only harness-free benchmarks appear as candidates here.

These counts are alternatives, not additions. Cybench and the METR time horizon evaluation would each qualify the same 11 systems, so running both buys nothing that running either does not. Pick one. That is what makes the near-empty strict board a measurement plan rather than a dead end.

This board is small because publishers do not report a harness consistently. Epoch's own SWE-bench Verified rows name no harness while DeepSWE's do, so those two measurements describe different configurations and are never merged. The number is published rather than hidden: a near-empty strict board is a fact about the state of public benchmark reporting, and it is the reason the model rollup exists as a second, separately labelled entity.

05 Category boards

Each classification is answered twice. First the ranking, on placement, which is the same ordering the board publishes: one snapshot gives one answer. Then every configuration measured, as evidence, so a reader can see how far a result moves when the harness or the reasoning effort changes. The second table is deliberately not a ranking.

Agents weight 35%

Ranked head to head: ordered by Bradley-Terry ratings fitted to every pairwise comparison between two models on the benchmarks they BOTH sat. This orders the category boards, and since 2026-08-29 it orders the Ferrox Index too: a category contributes its position in this fit, weighted by the rubric, never the raw composite it used to contribute. the figure shown is the observed win share: of every head-to-head comparison a model actually played, the share it won. The fitted rating orders the board; it is not published as a score, because under near-perfect transitivity its magnitude reflects the prior rather than a measurement. Showing 10 of 54 models with enough evidence in this classification. What it does not fix: a benchmark is not a fair coin. Two models separated by a hair and two separated by a mile both count as one win, so this measures who beats whom and not by how much. The margins are on the benchmark panels.

Every configuration measured

This table is evidence, not a ranking. It is sorted by score, and score is not comparable across rows: a row resting on one benchmark sits beside a row resting on three, and the benchmarks themselves differ in difficulty by up to 58 normalised points. Read it to see how much a configuration changes a result, and read the table above for where a model actually places.

367 systems with at least one qualifying benchmark: 20 FULL, 0 THIN, 347 SINGLE. 280 contributing measurements carry no measurement date. Showing 15.

clamped means the fixed anchor cut the value off at the top or the bottom of its scale, so that score is a floor or a ceiling rather than a point. It is the signature of an anchor that needs widening, and it is flagged rather than smoothed away. sources disagree means two benchmarks in the same correlation family, which are supposed to measure the same thing, returned materially different answers, so the interval was widened rather than allowed to shrink.

A model name can appear on more than one row. Those rows are not duplicates: they are different configurations of the same model, and the Configuration column names what differs. One model run under four agent harnesses is four measured systems with four different scores, and averaging them away would hide how much of a result comes from the harness rather than the model.

Coding weight 30%

Ranked head to head: ordered by Bradley-Terry ratings fitted to every pairwise comparison between two models on the benchmarks they BOTH sat. This orders the category boards, and since 2026-08-29 it orders the Ferrox Index too: a category contributes its position in this fit, weighted by the rubric, never the raw composite it used to contribute. the figure shown is the observed win share: of every head-to-head comparison a model actually played, the share it won. The fitted rating orders the board; it is not published as a score, because under near-perfect transitivity its magnitude reflects the prior rather than a measurement. Showing 10 of 54 models with enough evidence in this classification. What it does not fix: a benchmark is not a fair coin. Two models separated by a hair and two separated by a mile both count as one win, so this measures who beats whom and not by how much. The margins are on the benchmark panels.

Every configuration measured

This table is evidence, not a ranking. It is sorted by score, and score is not comparable across rows: a row resting on one benchmark sits beside a row resting on three, and the benchmarks themselves differ in difficulty by up to 58 normalised points. Read it to see how much a configuration changes a result, and read the table above for where a model actually places.

275 systems with at least one qualifying benchmark: 16 FULL, 1 THIN, 258 SINGLE. 212 contributing measurements carry no measurement date. Showing 15.

clamped means the fixed anchor cut the value off at the top or the bottom of its scale, so that score is a floor or a ceiling rather than a point. It is the signature of an anchor that needs widening, and it is flagged rather than smoothed away. sources disagree means two benchmarks in the same correlation family, which are supposed to measure the same thing, returned materially different answers, so the interval was widened rather than allowed to shrink.

A model name can appear on more than one row. Those rows are not duplicates: they are different configurations of the same model, and the Configuration column names what differs. One model run under four agent harnesses is four measured systems with four different scores, and averaging them away would hide how much of a result comes from the harness rather than the model.

Reasoning weight 25%

Ranked head to head: ordered by Bradley-Terry ratings fitted to every pairwise comparison between two models on the benchmarks they BOTH sat. This orders the category boards, and since 2026-08-29 it orders the Ferrox Index too: a category contributes its position in this fit, weighted by the rubric, never the raw composite it used to contribute. the figure shown is the observed win share: of every head-to-head comparison a model actually played, the share it won. The fitted rating orders the board; it is not published as a score, because under near-perfect transitivity its magnitude reflects the prior rather than a measurement. Showing 10 of 168 models with enough evidence in this classification. What it does not fix: a benchmark is not a fair coin. Two models separated by a hair and two separated by a mile both count as one win, so this measures who beats whom and not by how much. The margins are on the benchmark panels.

Every configuration measured

This table is evidence, not a ranking. It is sorted by score, and score is not comparable across rows: a row resting on one benchmark sits beside a row resting on three, and the benchmarks themselves differ in difficulty by up to 58 normalised points. Read it to see how much a configuration changes a result, and read the table above for where a model actually places.

398 systems with at least one qualifying benchmark: 265 FULL, 1 THIN, 132 SINGLE. 274 contributing measurements carry no measurement date. Showing 15.

clamped means the fixed anchor cut the value off at the top or the bottom of its scale, so that score is a floor or a ceiling rather than a point. It is the signature of an anchor that needs widening, and it is flagged rather than smoothed away. sources disagree means two benchmarks in the same correlation family, which are supposed to measure the same thing, returned materially different answers, so the interval was widened rather than allowed to shrink.

A model name can appear on more than one row. Those rows are not duplicates: they are different configurations of the same model, and the Configuration column names what differs. One model run under four agent harnesses is four measured systems with four different scores, and averaging them away would hide how much of a result comes from the harness rather than the model.

Preference weight 10%

Ranked head to head: ordered by Bradley-Terry ratings fitted to every pairwise comparison between two models on the benchmarks they BOTH sat. This orders the category boards, and since 2026-08-29 it orders the Ferrox Index too: a category contributes its position in this fit, weighted by the rubric, never the raw composite it used to contribute. the figure shown is the observed win share: of every head-to-head comparison a model actually played, the share it won. The fitted rating orders the board; it is not published as a score, because under near-perfect transitivity its magnitude reflects the prior rather than a measurement. Showing 10 of 97 models with enough evidence in this classification. What it does not fix: a benchmark is not a fair coin. Two models separated by a hair and two separated by a mile both count as one win, so this measures who beats whom and not by how much. The margins are on the benchmark panels.

Every configuration measured

This table is evidence, not a ranking. It is sorted by score, and score is not comparable across rows: a row resting on one benchmark sits beside a row resting on three, and the benchmarks themselves differ in difficulty by up to 58 normalised points. Read it to see how much a configuration changes a result, and read the table above for where a model actually places.

419 systems with at least one qualifying benchmark: 0 FULL, 93 THIN, 326 SINGLE. 512 contributing measurements carry no measurement date. Showing 15.

clamped means the fixed anchor cut the value off at the top or the bottom of its scale, so that score is a floor or a ceiling rather than a point. It is the signature of an anchor that needs widening, and it is flagged rather than smoothed away. sources disagree means two benchmarks in the same correlation family, which are supposed to measure the same thing, returned materially different answers, so the interval was widened rather than allowed to shrink.

A model name can appear on more than one row. Those rows are not duplicates: they are different configurations of the same model, and the Configuration column names what differs. One model run under four agent harnesses is four measured systems with four different scores, and averaging them away would hide how much of a result comes from the harness rather than the model.

06 Every benchmark behind the numbers

Who measured it, who moved it, on what terms, on what scale, and how many of its rows survived to enter a score.

Table scrolls sideways

Ingested and Scored are deliberately different numbers, and so is the row count in the source signature printed in section 11. A source file has some number of rows; some of those rows have no usable value or no identifiable model and never become an observation; and some observations are then refused entry to a score by the rights gate or the provenance gate. Three numbers, narrowing at each step. Section 07 accounts for the difference. Contamination is our recorded judgement of how likely a benchmark's tasks are to have appeared in training data. It is a risk flag, not a measurement, and a high rating is a reason to weigh that benchmark's contribution more carefully rather than to discard it.

Evaluator is who ran the benchmark. Transport is who we obtained it from. They are recorded separately and both are credited, because crediting only the aggregator would misstate who did the work. Attribution says how the evaluator is known: own means the transport ran the evaluation itself, row means the file carries a per-row source column so each row can be screened for vendor self-reports, and file means the file has no attribution column at all, so the evaluator is a reviewed assertion about the whole file and individual rows cannot be screened. Rows from a file-level source are counted and reported in the next section.

Correlation families

Benchmarks that measure the same thing are averaged inside their family before the category mean, so four variants of one task distribution count as one piece of evidence rather than four.

• swebench: swe_bench_verified, deepswe, swe_bench_pro, swe_bench_multilingual (4 benchmarks in this snapshot, 2 more declared in the contract and not yet admitted). These share the SWE-bench task distribution. The family contributes at most one effective bench-weight to Coding; members are averaged within the family before the category mean.
• arena: arena_text_style_control, arena_webdev, arena_agent (3 benchmarks in this snapshot). Same vote pool, different slices. Preference takes text and webdev; the agent config scores in AGENTS. webdev_arena_external is the same data via Epoch and is excluded.
• terminal: terminalbench, terminalbench_4 (2 benchmarks in this snapshot). Terminal-Bench counts in AGENTS only, never in Coding, even though its tasks are code-shaped (build spec v1.1). Terminal-Bench 2 (via Epoch) and Terminal-Bench 4.0 (via the benchmark repo) are different dataset versions of the same benchmark: correlated, never merged.
• math_competition: otis_mock_aime, frontiermath, aime, hmmt, imo_answerbench (5 benchmarks in this snapshot, 2 more declared in the contract and not yet admitted). Competition-maths family. frontiermath and frontiermath_tier_4 are the same benchmark at two difficulties: average them, never count both.
• tau2: tau2_airline, tau2_retail, tau2_telecom, tau2_banking (4 benchmarks in this snapshot). The four Tau2-Bench domains. One framework, one user simulator, one team, so they are correlated by construction: the family contributes at most one effective bench-weight to Agents and members are averaged within the family before the category mean. A model that sat all four holds four measurements and one independent proof, and is marked THIN accordingly.
• osworld: osworld (1 benchmark in this snapshot). OSWorld stands alone: a live desktop environment is not the same task as a terminal or a customer-service dialogue, so it shares no family with Terminal-Bench or Tau2-Bench.
• tool_use: mcp_atlas, tool_decathlon (2 benchmarks in this snapshot). Tool-calling suites. They measure one capability, driving an API surface, through different task sets, so the family contributes at most one effective bench-weight to Agents.
• cyber: cybench, cybergym (2 benchmarks in this snapshot). Security agent suites. Cybench was family-less, which quietly let it act as independent evidence beside any future security benchmark; it is grouped here so it cannot.
• browsecomp: browsecomp (1 benchmark in this snapshot). Live-web browsing, which no other benchmark here measures.

07 What was refused, and what is uncertain

An exclusion the reader cannot see is the same as no exclusion at all.

Refused entry to the score

Known weaknesses in what did get scored

08 Method

Weights

Agents 35%, Coding 30%, Reasoning 25%, Preference 10%. knowledge and long_context carry zero weight in this rubric version: they are ingested and displayed, never scored.

Required categories

agents, coding, reasoning. A model missing any of them has no index. preference is optional, and its absence is declared on the row.

Entry requirement

2 qualifying benchmarks in each required category, from at least 2 distinct correlation families before a category counts as FULL rather than THIN.

Normalisation

Every raw value is mapped onto 0 to 100 against fixed anchors, never against the current field. Normalising against the live field would move every published score whenever an unrelated model launched. Anchors are per benchmark wherever the scale differs, because two rating systems do not share a scale.

Recency

Measured on the date of MEASUREMENT, not the model's release date, with a default cutoff of 18 months. Filtering on release date would delete every comparison baseline from the board.

Uncertainty

Published intervals are used as published and never recomputed. Where members of a correlation family disagree, the band is widened by half their spread rather than being allowed to shrink, so disagreement between two measurements of the same thing shows up as less confidence instead of vanishing into a mean. A published interval of width zero is treated as saturation evidence, not precision evidence.

Ranking

weighted mean of each category's head-to-head position within its published field. Positions, not scores: a category counts for where a model finished against everyone who sat the same benchmarks. The composite the previous rule produced is kept on every row and printed beside the index.. Confidence-interval overlap is not used to group rows into tiers, because overlap is not transitive: a row can overlap its neighbour, and that neighbour overlap its own, while the two ends are clearly apart.

Rollup

median observed configuration per benchmark (lower median, always a real measurement).

Self-reports

A value whose source is a vendor technical report, system card, model card or launch announcement is recorded as CLAIMED and never enters a score.

09 What this report cannot tell you

Every limit below is measured, not estimated, and every one of them is a reason to read a specific number with more care.

The rubric is not calibrated

UNCALIBRATED - anchors and thresholds are placeholders until the 90-day backfill study runs. Every published number must carry this label. Anchors and thresholds are placeholders. Ordering within the board is far more trustworthy than the absolute value of any index.

Benchmarks of different difficulty are averaged together

A category score is the mean across its benchmark families, and those families are not equally hard. In Coding alone this report scores SWE-bench Verified (Epoch's own run), Aider Polyglot, DeepSWE, FrontierCode, ALE-Bench, SWE-bench Pro, SWE-bench Multilingual, whose normalised distributions sit at very different heights. Until the calibration study equalises them, being measured on a harder benchmark costs a model points that a reader would naturally attribute to weaker ability.

The missing preference category is biased, not merely absent

RE-MEASURED on 2026-08-29 when the index moved from composite category scores to head-to-head positions, because the previous band of 2.17 was derived in composite points and could not be carried onto a different scale. The measurement is the same one, run again: for the 13 ranked models holding both a qualifying preference position and all three required categories, compare the measured preference position with the position implied by the weighted mean of the others. THE BIAS DISAPPEARED. It was +21.66 composite points, 8 of 8 in the same direction. It is now -1.64 position points with an sd of 17.79 and only 5 of 13 positive, which is indistinguishable from zero. That is exactly what the previous version of this note predicted would happen: it argued the offset was an artefact of incommensurable anchors rather than a property of models, since preference is a Bradley-Terry rating normalised against placeholder anchors of its own while the other categories are pass rates against 0 to 1. Positions are scale-free, so the artefact had nothing left to live in. The hypothesis was recorded before the test and the test agreed with it. The band therefore now describes NOISE, not bias. Imputing preference moves a row by 17.79 position points one standard deviation, which at the 10% preference weight is 1.78 index points, so an ordering finer than that between an imputed row and a measured row is not something this board can assert. On the live snapshot the band contains 18 of 561 pairs, 10 of them with exactly one side imputed, against a median adjacent gap of 1.95 index points.

There is no longer an offset to correct, and that is the finding rather than a convenience. The previous version of this field declined to add back a +21.66 point offset on the grounds that it was an artefact of incommensurable anchors rather than a property of models: preference is a Bradley-Terry rating normalised against placeholder anchors of its own, 900 to 1600 for the text arena and 1000 to 1800 for the web development arena, which run on different scales and so cannot share one, while the other categories are pass rates against 0 to 1. Moving the index to positions removed the anchors from the comparison, and the offset went with them. The band is now a width and not a shift, so there is nothing to add back and adding anything would be fitting noise.

Measured on 2026-08-29 over 13 models. That is a thin sample, and it is re-measured as coverage grows.

Some rows cannot be screened

439 scored rows come from files that carry no attribution column. The evaluator is named for the file, and the file is reviewed, but no individual row can be checked. They are marked rather than dropped, because dropping them would remove real coverage and hide the fact that the source does not publish attribution.

Some measurements carry no date

1278 contributing measurements across the category boards were published without a measurement date. They are counted and shown rather than assumed current or silently dropped.

Absence of a model is not a judgement about it

A model can be missing because no one has published two independent measurements of it in a required category. That is a statement about public benchmark coverage, not about the model.

10 Corrections

Every figure here traces to a snapshot id and a source manifest id, both printed at the top of this report and repeated in the footer. Quote them when reporting a problem and the exact inputs can be replayed to the byte. The same snapshot is available as a machine-readable document, so any figure on this page can be checked against the object it was rendered from without taking our word for it.

Corrections go to ferroxlabs.com. A correction that changes a published number is described when it is applied, not quietly folded into the next snapshot.

• A number that is wrong because a source published it wrongly is corrected when the source corrects it, and the change is visible in the next snapshot.
• A number that is wrong because this engine handled it wrongly is corrected in the code, and the correction is described rather than quietly applied.
• Identity is never merged on a guess. If two identifiers look like the same model and no reviewed alias exists, they stay separate and are listed as unmapped.
• A benchmark is never removed from a published report to improve a result.

11 Attribution and licences

Required on any page that shows these numbers.

• LMArena leaderboard dataset (CC BY 4.0) CC-BY-4.0
  https://huggingface.co/datasets/lmarena-ai/leaderboard-dataset
• Per-row source URLs are published on every record page. facts, not expression
• Epoch AI, 'AI Benchmarking Hub'. Published online at epoch.ai. Retrieved from 'https://epoch.ai/benchmarks' [online resource]. CC-BY-4.0
  https://epoch.ai/data/ai-benchmarking-dashboard
• models.dev (MIT) MIT
  https://models.dev
• OSWorld leaderboard, XLANG Lab. Published at http://osworld-v1.xlang.ai/ [online resource]. none stated
  http://osworld-v1.xlang.ai/
• Terminal-Bench 4.0 leaderboard, Harbor Framework. Published at https://github.com/harbor-framework/terminal-bench [online resource]. Apache-2.0
  https://github.com/harbor-framework/terminal-bench

Sources in this snapshot