What happened
Artificial Analysis introduced Progress Index v1 as a methodology for estimating the general capability of language models on a scale intended to remain consistent over time. Its stated use is analysis of AI progress: comparing models released in different periods even when they were not evaluated on precisely the same benchmark suite. Version 1.0 is listed as launched in October 2026 and current.
The company distinguishes the new index from its Intelligence Index. The Intelligence Index is its primary measure for comparing models for current use cases, but its composition changes as new evaluations are added and older ones stop differentiating frontier models. Artificial Analysis says scores across versions of that index are therefore not intended as a long-run comparison. Progress Index v1 is intended to supply that missing longitudinal view, rather than replace the current-capability index.
The method uses item response theory (IRT). Each model receives one estimated capability value; each task receives an estimated difficulty and discrimination value. A task's difficulty is the capability at which a model is expected to score 50%, after removal of any guessing floor. Models and tasks are fitted jointly using a weighted log-loss objective with regularization and priors. Results from benchmarks that share models link older and newer evaluations onto a common scale, so a model need not have been tested on every benchmark in the dataset.
Artificial Analysis treats each effort variant as a separate model during fitting, then represents a model by its highest-scoring variant. It scales the resulting capabilities and difficulties so that o3-mini (high) equals 220 and GPT-5.5 (xhigh) equals 320. Input data comes from public Artificial Analysis benchmarks: a benchmark needs results from at least four models, and a model needs results on at least four benchmarks. Scores are normalized from zero to one; where guessing is possible, the guessing baseline is rescaled to zero. The methodology also imputes a zero on AA-LCR for models with context windows below 100,000 tokens.
What the archive adds
This is a new fact in a known evaluation story. Artificial Analysis launched the Decision Index v1.0 separated decision-model evaluation from general benchmarking; Progress Index instead separates longitudinal comparison from a changing frontier index. Vals AI’s Inference Provider Index addressed a different layer: endpoint cost and speed for agentic tasks.
Trend signal
The release modestly supports the assumption that organizations benefit from evaluation assets that remain usable while models change. The owned learning loop assumption concerns preserving traces and rubrics independently of any one model; Progress Index is not an organizational evaluation system, but it makes the same portability problem explicit at the benchmark level.
Why now
The timing follows a structural limitation in frontier scorecards. As models improve, benchmark operators add new evaluations and retire ones that no longer discriminate among leading systems. That makes a current leaderboard responsive, but breaks direct comparison between its historical versions. Progress Index v1 is Artificial Analysis’s attempt to retain a current index while adding a separate historical measuring instrument.
Connections to other events
The Decision Index release framed classification, tool calling, knowledge, and reasoning as a distinct task class rather than one aggregate score. Progress Index does not create another task category. Instead, it tries to make heterogeneous task results comparable across time through shared models and estimated task difficulty. The two approaches can coexist: one refines what is being evaluated, while the other refines how changing evaluations are connected historically.
Why it matters
Teams should resist reading a single benchmark score as a stable measure of progress when the underlying test mix has changed. Progress Index v1 offers a more explicit way to ask a different question: how does capability appear to move across generations when overlapping evaluation results are statistically linked? That can help research, strategy, and procurement teams distinguish a changing ruler from a changing model.
It does not, by itself, select a model for production. Artificial Analysis explicitly recommends its Intelligence Index for present-day use-case comparisons. Builders still need task-specific evaluations, deployment constraints, reliability measurements, price, latency, and safety controls. The useful takeaway is methodological: a historical chart should disclose how old and new tests were made commensurable.
What happens next
- Artificial Analysis will revise the index methodology or dataset as additional model and benchmark results accumulate (6 months, 70%). Watch for: a new version-history entry or changed inclusion rules.
- More evaluation products will separate historical-progress measures from current-use-case leaderboards (12 months, 60%). Watch for: methodology pages that state explicit cross-version linking or calibration methods.
Limits and uncertainty
The source describes methodology, not a validation study showing that the resulting scale predicts real-world task success, economic value, or production reliability. It does not provide the full fitted result table, uncertainty intervals, or an independent replication in the supplied material. Anchoring the scale to o3-mini (high) and GPT-5.5 (xhigh) sets its numerical interpretation, while the weighting, priors, benchmark coverage, and zero imputation rules remain consequential modeling choices. The index also uses only Artificial Analysis benchmarks, so its conclusions inherit the scope and quality of that evaluation corpus.
Skill to build now
Audit any internal model-progress chart this week. Record which tasks were used in each period, which models overlap between periods, and whether score changes could result from a changed test set. Where tests evolve, preserve a stable bridge set and report a current task-specific score separately from a longitudinal estimate. This does not require implementing IRT immediately; it requires treating benchmark continuity as a design requirement rather than an afterthought.
Reader takeaway
Progress Index v1 is a proposal for measuring model progress across changing benchmarks, not a new verdict on which model to deploy. Its core contribution is to estimate task difficulty and model capability together, using overlapping results to connect evaluations across years. For builders, the lesson is to keep present-day model selection and long-run progress tracking separate.
Open question
Will the index’s historical ordering and apparent rate of progress remain stable when more benchmarks, models, and independently reproduced task results are added?