What happened
Epoch AI’s November 4, 2026 announcement describes a new statistical model for the Epoch Capabilities Index, which tracks model capability using benchmark results. ECI uses an item-response-theory framework to compare models that have taken different benchmarks of varying difficulty. Epoch adapts a four-parameter logistic model to continuous benchmark scores: each model has a capability estimate, and each benchmark has a difficulty and slope. Scores are rescaled so chance-level performance maps to zero and the highest achievable score maps to one. The model assumes a single capability dimension and uses benchmarks scored between 0% and 100%.
Previously, Epoch fit the model by minimizing squared error between predicted and observed benchmark scores, with a regularization term. It estimated 90% confidence intervals by taking 500 bootstrap samples of each model’s benchmark results and refitting the model. Epoch says models had a median of ten benchmark scores and as few as four, making bootstrap resampling unreliable as an estimate of uncertainty in this small-data setting. The earlier method also was not designed to compare models directly.
The new method samples a joint posterior distribution using Markov Chain Monte Carlo. Epoch reports each model’s ECI as the mean of its resulting distribution and the 5th–95th percentile range as a 90% credible interval. This also lets the researchers examine the distribution of the difference between two models and estimate how likely one is to be more capable. The model uses a beta likelihood, keeping scores within the 0–1 range and allowing expected noise to shrink near the limits. Epoch also gives each benchmark its own noise parameter, rather than treating all benchmark results as equally noisy.
For identification, the new model centers benchmark difficulties at zero and sets the geometric mean of benchmark slopes to one, using the full benchmark set rather than fixing the slope of a single benchmark. Epoch then transforms results to the familiar ECI scale by setting Claude 3.5 Sonnet to 130 and GPT-5 to 150. The authors describe flat priors for model capability and benchmark difficulty and hierarchical priors for slopes and noise.
The reported comparison covers 252 models. For the 176 models whose previous ECI was at least 125, the new and old estimates align closely: R² is 0.995, root mean squared difference is 1.1 points, and mean absolute difference is 0.9 points. Every new estimate in this group falls inside its previous 90% confidence interval; the new intervals are 13% narrower on average. The 76 models below 125 all move down, by 13.7 points on average. Epoch attributes part of that shift to removing the earlier regularization, which had particularly affected weak models with few, mostly saturated benchmark results.
What the archive adds
New Runtime’s earlier coverage examined Epoch’s automation report as an evaluation of actual research work performed for one organization (Can AI automate Epoch?, October 10, 2026). Its earlier coverage also described Epoch’s benchmark comparison placing Gemini 4 Argon near the frontier (Gemini is the closest it’s been to the frontier since February., October 24, 2026). This is a new methodological event on the broader story of how benchmark evidence is converted into model comparisons.
Trend signal
This supports the broad importance of making model evaluations interpretable, but it does not directly establish a tracked agent-system pattern. Epoch’s change concerns statistical treatment of benchmark scores, not verification of agent work products or a runtime capability. This does not yet fit a pattern New Runtime tracks.
Why now
Epoch says the previous bootstrap approach was not justified with so few benchmark scores per model, while the new model addresses uncertainty estimation and direct model comparison. The announcement presents the update as a methodological correction, not a response to a newly released model or a change in benchmark access.
Why it matters
A ranking is not just a list of point estimates: it depends on how scores from different tests are normalized, how much uncertainty is attributed to each result, and how that uncertainty is summarized. Epoch’s change makes the uncertainty model explicit and permits probability-based comparisons between model pairs. For teams choosing models, that can help prevent a small difference in a leaderboard score from being treated as decisive when the uncertainty overlaps.
The results also demonstrate that method changes can affect parts of a ranking unevenly. Epoch reports that frontier estimates remain close to their former values, while lower-scoring models move down substantially. A team using ECI historically should not interpret every change in a model’s score as new capability evidence; it should check whether the underlying index methodology changed. The anchor scores—Claude 3.5 Sonnet at 130 and GPT-5 at 150—preserve a reference scale, but do not make the revised and earlier estimates identical.
What happens next
- Teams comparing historical ECI scores will increasingly annotate methodology changes alongside model releases (6 months, 65%). Watch for: benchmark dashboards adding explicit methodology or version labels.
Limits and uncertainty
Epoch’s reported comparison is an evaluation by the organization that maintains ECI; it is not an independent replication of the new model. The post reports fit statistics and score shifts, but the announcement alone does not establish how useful the new pairwise probabilities will be in real-world model selection. ECI also compresses performance across benchmarks into a single capability dimension, an assumption that can conceal task-specific strengths and weaknesses. The announced anchor models set the scale and have no uncertainty in the new reporting, so interval estimates are relative to those anchors rather than absolute measures of capability.
Skill to build now
When using a model index in procurement or evaluation, record its methodology and version alongside the score. For ECI specifically, compare models using the reported uncertainty distributions where available, and retain task-level benchmark results for decisions where your workload differs from the index’s single capability dimension.
Reader takeaway
Epoch’s new Bayesian ECI is a change in how benchmark evidence is aggregated, not evidence that frontier models suddenly changed. The top of the index remains broadly stable, but weaker models move down and uncertainty is represented differently. Treat the methodology date as part of any comparison across time.