---
type: "post"
id: "evaluate-ai-models-for-observability-o11y-bench-joins-2cecf9"
slug: "evaluate-ai-models-for-observability-o11y-bench-joins-2cecf9"
title: "Evaluate AI models for observability: o11y-bench joins Fireworks’ Specialized Intelligence Index and reaches v1.1"
description: "Grafana says o11y-bench, its open benchmark for observability agents, has joined Fireworks’ Specialized Intelligence Index and reached version 1.1."
retrieval_nugget: "Grafana says o11y-bench, its open benchmark for observability agents, has joined Fireworks’ Specialized Intelligence Index and reached version 1.1. The update checks whether agents’ claims and persistent changes match a running Grafana environment, giving teams a more grounded way to compare performance on operational work."
published_at: "2026-11-06"
updated_at: "2026-11-06"
record_date: "2026-11-06"
date_kind: "published_at"
topics: ["mcp"]
entities: ["Grafana Labs","o11y-bench","Fireworks","Specialized Intelligence Index","Grafana MCP server","Prometheus","Loki","Tempo"]
related_patterns: ["agent-economics-moves-to-completed-work"]
novelty_label: "new_fact"
what_changed: "Grafana Labs’ o11y-bench has joined Fireworks’ Specialized Intelligence Index and reached version 1.1."
prior_refs: []
source_url: "https://grafana.com/blog/evaluate-ai-models-for-observability-o11y-bench-joins-fireworks-specialized-intelligence-index-and-reaches-v1-1"
source_urls: ["https://grafana.com/blog/evaluate-ai-models-for-observability-o11y-bench-joins-fireworks-specialized-intelligence-index-and-reaches-v1-1"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:evaluate-ai-models-for-observability-o11y-bench-joins-2cecf9"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/evaluate-ai-models-for-observability-o11y-bench-joins-2cecf9/","markdown":"https://newruntime.com/posts/evaluate-ai-models-for-observability-o11y-bench-joins-2cecf9.md","json":"https://newruntime.com/posts/evaluate-ai-models-for-observability-o11y-bench-joins-2cecf9.json"}
markdown_url: "https://newruntime.com/posts/evaluate-ai-models-for-observability-o11y-bench-joins-2cecf9.md"
---

# Evaluate AI models for observability: o11y-bench joins Fireworks’ Specialized Intelligence Index and reaches v1.1

## Retrieval answer

Grafana says o11y-bench, its open benchmark for observability agents, has joined Fireworks’ Specialized Intelligence Index and reached version 1.1. The update checks whether agents’ claims and persistent changes match a running Grafana environment, giving teams a more grounded way to compare performance on operational work.

## Summary
Grafana says o11y-bench, its open benchmark for observability agents, has joined Fireworks’ Specialized Intelligence Index and reached version 1.1. The update checks whether agents’ claims and persistent changes match a running Grafana environment, giving teams a more grounded way to compare performance on operational work.

## What happened
Grafana Labs announced two changes on October 28, 2026: o11y-bench is now included in Fireworks’ Specialized Intelligence Index, and Grafana is releasing o11y-bench 1.1. The index brings together practitioner-built benchmarks across domains including healthcare, finance, cybersecurity and software engineering. The observability entry covers workflows such as querying telemetry, investigating incidents and managing Grafana dashboards. Grafana says teams can compare models on the index and use o11y-bench with Fireworks inference, including serverless models and models deployed in their own Fireworks accounts.

The benchmark runs agents against a Grafana stack containing synthetic but realistic telemetry. Tasks span six areas: Prometheus and PromQL; Loki and LogQL; Tempo and TraceQL; multi-step incident investigations; Grafana API workflows; and dashboard creation, editing and repair. Agents interact through the Grafana MCP server. There are 64 tasks. Grafana emphasizes that a syntactically valid query or a dashboard that renders may still be wrong: an agent must select relevant signals, use the right time range, connect evidence across telemetry types, and avoid incorrect changes to persistent state.

Version 1.1 makes grading more outcome-based and reproducible. Grafana says the evaluator checks whether an agent’s claims match facts in the environment, whether saved changes are correct, and whether the task requirements are met. Reported numbers are checked against reference values, while cited traces are read back from Tempo to test whether they support the agent’s claims. Dashboard edits are assessed through before-and-after diffs of saved state. If a ground-truth query is empty or fails, Grafana treats that as a grading error rather than letting it affect pass rates.

Each task is run three times. Grafana prioritizes Pass^3, the average score across the three runs, and also reports Pass@3, whether the model succeeded at least once. Saved facts allow results to be regraded without rerunning the agent, and the scenarios are designed to remain consistent over time. Grafana reports that a no-tools control failed all 64 tasks. Because version 1.1 updates grading and task calibration, Grafana says its results are not directly comparable with version 1.0; v1.1 establishes a new baseline. The benchmark’s tasks, environments, grading logic and results are open source.

## What the archive adds
New Runtime’s earlier coverage on [Zapier’s AutomationBench joining the same index](https://newruntime.com/posts/zapier-s-automationbench-joins-fireworks-specialized-intelligence-index-11f11f/) reported a benchmark for end-to-end business workflows, with its best model completing 51.3% of workflows. This is a new domain-specific benchmark and a new version, not a restatement of that result.

## Trend signal
This supports [agent economics moves to completed work](https://newruntime.com/patterns/agent-economics-moves-to-completed-work/): the benchmark focuses on whether an agent completes operational tasks correctly, not merely on plausible responses or model-call characteristics. Its repeated runs also make consistency part of the evaluation. The benchmark does not itself establish production performance or cost per completed task.

## Why now
Grafana frames the release around the difficulty of evaluating observability agents during incident work: correct-looking queries and explanations can still be unsupported by telemetry, and dashboard edits can leave faulty state. Version 1.1 is intended to make results more grounded and reproducible as models, harnesses and observability tools evolve.

## Why it matters
Teams evaluating agents for SRE or platform work need to test more than whether a model can produce valid PromQL or summarize logs. A useful evaluation needs to check whether the cited evidence supports its conclusions and whether its actions leave dashboards and other state correct. o11y-bench’s outcome checks address that gap in a public, inspectable task environment.

The three-run reporting also draws attention to reliability: one successful incident investigation is not the same as repeatable behavior. Teams can use Pass^3 alongside Pass@3 to distinguish consistent performance from occasional success. But v1.1’s new baseline means old and new scores should not be compared as if the evaluation were unchanged. For teams considering an observability agent, the practical value is a reproducible test set to examine and run against their own requirements—not a guarantee that benchmark performance will transfer to production telemetry.

## Limits and uncertainty
Grafana describes synthetic but realistic telemetry and benchmark tasks; it does not establish that results predict outcomes on any particular team’s production systems. The announcement provides no model-by-model scores in the text summarized here, so it does not support a ranking of candidates. Grafana also says version 1.1 results are not directly comparable with version 1.0. The benchmark is open source, but these claims are Grafana’s account of its design and reported control result; they are not an independent audit of benchmark validity.

## Skill to build now
Choose one routine observability workflow—such as investigating an alert across metrics and traces—and write down what would count as a correct outcome: the relevant time range, the facts an agent must cite, and any state it may change. That gives your team a concrete checklist for inspecting o11y-bench tasks or designing an internal evaluation that grades evidence and saved state, not just answer fluency.

## Reader takeaway
For observability agents, a credible benchmark should verify both the evidence behind an answer and the state left behind by an action. Grafana’s o11y-bench 1.1 builds those checks into an open, repeated-run evaluation, but its new scoring baseline cannot be read as a direct continuation of v1.0.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Mcp](https://newruntime.com/topics/mcp.md): Explore the mcp topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)