---
type: "post"
id: "vulcanbench-ranks-reflection-beam-far-below-frontier-models-8ecbd2"
slug: "vulcanbench-ranks-reflection-beam-far-below-frontier-models-8ecbd2"
title: "VulcanBench ranks Reflection Beam far below frontier models on Frontier v4 reconstruction"
description: "VulcanBench reports 115 runs of Reflection Beam 1.0 in its Frontier v4 coding evaluation, with the model passing six of 23 reconstruction tasks at its best effort level."
retrieval_nugget: "VulcanBench reports 115 runs of Reflection Beam 1.0 in its Frontier v4 coding evaluation, with the model passing six of 23 reconstruction tasks at its best effort level."
published_at: "2026-11-09"
updated_at: "2026-11-09"
record_date: "2026-11-09"
date_kind: "published_at"
topics: ["models"]
entities: ["VulcanBench","Reflection Beam","Mirror CLI","Reflection","Frontier v4","Muse Spark 1.3","GPT-6.1 Sol"]
related_patterns: ["ai-engineering-evaluations-are-splitting-into-task-specific"]
novelty_label: "new_fact"
source_url: "https://vulcanbench.com/models/reflection-beam.html"
source_urls: ["https://vulcanbench.com/models/reflection-beam.html"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:vulcanbench-ranks-reflection-beam-far-below-frontier-models-8ecbd2"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/vulcanbench-ranks-reflection-beam-far-below-frontier-models-8ecbd2/","markdown":"https://newruntime.com/posts/vulcanbench-ranks-reflection-beam-far-below-frontier-models-8ecbd2.md","json":"https://newruntime.com/posts/vulcanbench-ranks-reflection-beam-far-below-frontier-models-8ecbd2.json"}
markdown_url: "https://newruntime.com/posts/vulcanbench-ranks-reflection-beam-far-below-frontier-models-8ecbd2.md"
---

# VulcanBench ranks Reflection Beam far below frontier models on Frontier v4 reconstruction

## Retrieval answer

VulcanBench reports 115 runs of Reflection Beam 1.0 in its Frontier v4 coding evaluation, with the model passing six of 23 reconstruction tasks at its best effort level.

## Summary
VulcanBench reports 115 runs of Reflection Beam 1.0 in its Frontier v4 coding evaluation, with the model passing six of 23 reconstruction tasks at its best effort level. The result offers a concrete but narrow comparison point for teams assessing coding agents: performance varied sharply by effort setting, and the evaluation does not establish how Beam will perform on a team's own codebase.

## What happened
VulcanBench's Reflection Beam results page lists measurements for Reflection Beam across every reasoning level offered on Frontier v4. The tested model is identified as Beam-1.0-501B-A23B, or Reflection Beam 1.0, with 501 billion total parameters and 23 billion active parameters. It ran through Mirror CLI 0.1.1, described as Reflection's own coding agent in an early beta, using Reflection's API with analytics off.

The suite contains 23 binary-reconstruction coding tasks and reports five effort levels, from Low through Max. VulcanBench lists combined scores on judged runs of 39.51 at Low, 44.79 at Medium, 55.01 at High, 64.72 at Extra-high and 63.92 at Max. Across those levels, the model fully passed zero, zero, zero, four and six tasks respectively. The page also gives an alternative tally that counts runs leaving no code as zero: 18.90 at Low and 38.95 at Medium. The sweep comprised 115 runs in total, with 11 to 23 judged runs per effort level.

The page reports that the Extra-high run count was 23, the same number as the tasks in the suite; the other effort levels had 11, 20, 23 and 22 judged runs respectively. These counts matter because the headline scores are calculated over judged runs, while the alternative zero-inclusive figures are specifically supplied only for Low and Medium. The reported pass totals also show that a higher combined score did not mean all tasks were completed: even at Max, six of the 23 tasks passed.

The reported Extra-high score placed Reflection Beam 63rd among 75 model-and-effort columns on VulcanBench's board as of October 27, 2026. The page says that result was about 24 to 26 points below Claude Opus 5.5, GPT-6.1 Sol and GPT-6 Astra. It cautions that those columns used a different judge panel. For Reflection Beam, code quality was judged by Muse Spark 1.3 and GPT-6.1 Sol; VulcanBench says Grok 4.6's Cursor account had run out of usage, prompting the different panel.

The page reports runtime ranging from 0.4 to 22.4 minutes per task and token use from 0.09 million to 17.29 million per task. Reflection publishes no Beam price, so VulcanBench does not give a cost comparison. A separately published GitHub trace repository contains model calls, token use and timing, plus tool-step timing. The page notes that this sweep's adapter did not record which tools Beam called, making those traces thinner than traces for other models. VulcanBench describes the published result as an answer to which model is stronger on this suite; it says a team's own languages, codebases and task mix require measurement rather than inference from the public ranking.

## What the archive adds
New Runtime's earlier coverage introduced benchmarks for other bounded model tasks, including [LifeVal](https://newruntime.com/posts/openai-unveiled-lifeval-a-benchmark-grading-models-on-44478e/) on personal-assistant work (November 2, 2026) and [IWIBench](https://newruntime.com/posts/hcltech-researchers-release-iwibench-a-benchmark-for-models-08782f/) on model weight editing (November 5, 2026). This is a new result in that broader stream of task-specific evaluations, not a follow-up to either benchmark.

## Trend signal
This supports the general value of task-bounded evaluation, but it does not materially support a tracked pattern about agent architecture or economics. The results are specific to one reconstruction suite, a particular harness, and a distinct judge panel. This does not yet fit a pattern New Runtime tracks.

## Why it matters
A single leaderboard position can hide operationally important differences. In VulcanBench's results, Reflection Beam's best effort setting passed six tasks while the lower three passed none; the score alone does not tell a team whether those solved tasks resemble its own work. Runtime and token ranges also vary substantially, and no model price is available to turn usage into a cost-per-task comparison.

For builders, the page is most useful as a reason to evaluate the entire coding setup rather than the model name in isolation. Mirror CLI, task selection, effort setting, judge design and tool behavior all shape the result. Teams can use the published figures to decide whether a local evaluation is worth running, but should not treat a ranking from binary reconstruction as a proxy for general coding quality. The 115-run sweep also demonstrates the value of keeping task-level outcomes and execution traces alongside aggregate scores: those details expose where a system succeeds, how much time and token use it spends, and where the measurement itself is incomplete.

## Limits and uncertainty
The evaluation covers 23 tasks, and VulcanBench reports pass counts rather than evidence that this suite represents ordinary software maintenance or feature work. The page says code quality was judged by Muse Spark 1.3 and GPT-6.1 Sol, unlike most leaderboard columns, so the cross-model gap is not a fully like-for-like comparison. The reported traces do not identify which tools Beam called. Reflection's price is not published, and the results do not establish performance on other languages, repositories, or task mixes. The page also does not establish whether the early-beta Mirror CLI setup behaves like a later production release. Without a published price and a consistent judge across all compared columns, neither cost efficiency nor the exact size of Beam's gap to other models can be established from this sweep alone.

## Skill to build now
Build a small held-out evaluation from real tasks in your repository before changing a coding-agent route. Include representative fixes and reconstruction tasks, define tests that determine completion, and record pass rate, elapsed time and token use for each effort setting. That turns VulcanBench's narrow result into a decision your team can verify against its own workload.

## Reader takeaway
Reflection Beam passed six of 23 Frontier v4 reconstruction tasks at its strongest reported setting, but the number is a suite-specific signal, not a verdict on the model for everyday coding. Use it to motivate a local evaluation, and compare completed tasks, runtime and usage under your own tests before adopting or routing work to Beam.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Models](https://newruntime.com/topics/models.md): Explore the models topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)