---
type: "post"
id: "stackness-reanalysis-retrying-bare-gpt-6-luna-costs-684ccf"
slug: "stackness-reanalysis-retrying-bare-gpt-6-luna-costs-684ccf"
title: "Stackness reanalysis: retrying bare GPT-6 Luna costs about the same as harnessing it, and harness build cost was never measured"
description: "Stackness reanalysed a one-task coding-agent experiment comparing GPT-6 Luna with GPT-6 Sol."
retrieval_nugget: "Stackness reanalysed a one-task coding-agent experiment comparing GPT-6 Luna with GPT-6 Sol. The headline cost advantage of a task-specific behavioral harness becomes much less decisive when bare Luna retries are priced by successful outcomes, while the cost of creating and maintaining the harness was not measured."
published_at: "2026-10-28"
updated_at: "2026-10-28"
record_date: "2026-10-28"
date_kind: "published_at"
topics: ["codex"]
entities: ["Stackness","GPT-6 Luna","GPT-6 Sol","Codex","Naresh Jain","Specmatic","OpenAI"]
related_patterns: ["harness-architecture-outlives-model-choice","agent-economics-moves-to-completed-work","verification-bandwidth-is-the-scarce-resource"]
novelty_label: "new_fact"
what_changed: "Stackness adds a reanalysis: retrying bare Luna costs about as much as harnessing it, and the harness’s build cost was not measured."
prior_refs: ["memory:3374a02a-dee9-87a4-a3ee-b5e3b1a77fb7"]
source_url: "https://stackness.dev/blog/does-a-harness-make-a-cheap-model-as-reliable-as-a-frontier-model"
source_urls: ["https://stackness.dev/blog/does-a-harness-make-a-cheap-model-as-reliable-as-a-frontier-model"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:stackness-reanalysis-retrying-bare-gpt-6-luna-costs-684ccf"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/stackness-reanalysis-retrying-bare-gpt-6-luna-costs-684ccf/","markdown":"https://newruntime.com/posts/stackness-reanalysis-retrying-bare-gpt-6-luna-costs-684ccf.md","json":"https://newruntime.com/posts/stackness-reanalysis-retrying-bare-gpt-6-luna-costs-684ccf.json"}
---

# Stackness reanalysis: retrying bare GPT-6 Luna costs about the same as harnessing it, and harness build cost was never measured

## Retrieval answer

Stackness reanalysed a one-task coding-agent experiment comparing GPT-6 Luna with GPT-6 Sol. The headline cost advantage of a task-specific behavioral harness becomes much less decisive when bare Luna retries are priced by successful outcomes, while the cost of creating and maintaining the harness was not measured.

## Summary
Stackness reanalysed a one-task coding-agent experiment comparing GPT-6 Luna with GPT-6 Sol. The headline cost advantage of a task-specific behavioral harness becomes much less decisive when bare Luna retries are priced by successful outcomes, while the cost of creating and maintaining the harness was not measured.

## What happened
Stackness published its reanalysis on 23 October 2026 of an experiment by Naresh Jain, published on 13 October. The underlying test asked Codex, at low reasoning effort, to build a Spring Boot backend-for-frontend. The service had to call an Order API, publish to Kafka, and follow awkward requirements including returning HTTP 202 plus a polling link when creation timed out. Each run used a fresh Docker container. A result counted only when the service started and passed all 178 hidden acceptance tests.

Jain ran ten paired attempts for each model and setup. Bare GPT-6 Luna passed 3 of 10 runs. With a behavioral harness, Luna passed 10 of 10, matching bare GPT-6 Sol, which also passed all ten. Harnessed Luna was estimated at $0.273 per run, versus $0.406 for bare Sol. But it took 35 minutes per run rather than 6 minutes 51 seconds, and consumed 22.93 million tokens rather than Sol's 1.08 million. The reported figures are API-equivalent estimates based on recorded token use and model prices, not billed costs.

The behavioral harness in this experiment was a specific set of guides and sensors around an otherwise unchanged Codex setup. Guides included OpenAPI and AsyncAPI specifications, request and response examples covering failure cases and Kafka events, plus an API-development skill. Sensors included contract checks and API tests against contract-driven HTTP and Kafka mocks. Their reports were returned to the agent during the build. The checks did not use an LLM as a judge; the agent had to invoke the test command, and was instructed to repair code rather than weaken expectations.

The setup changed Luna's reliability, not Sol's observed pass rate. Stackness reports that bare Luna's failures clustered around protocol and integration details: wrong response codes for a timed-out creation, an unexpected Kafka header from Spring's JSON serializer, and an incorrect environment-variable name. The article interprets the harness as an external memory and attention aid that surfaces omitted requirements when a check fails. For Sol, adding the harness retained the 10-of-10 pass result but raised cost, token use, and elapsed time.

Stackness then recalculated cost per accepted application. At a 3-in-10 pass rate and $0.086 per bare Luna run, repeated independent attempts imply roughly $0.287 per accepted app, close to the $0.273 estimate for harnessed Luna. This is arithmetic on the published numbers, not a separately executed retry experiment. It also implies roughly 53 minutes of expected wall time per accepted app. Crucially, retries only help when an evaluator can distinguish a successful output from a failing one; here that role was played by the independent hidden test suite. The article also notes that neither the original experiment nor the reanalysis measured the engineering cost of building and maintaining the harness, and that cache composition and long-context price tiers could change the estimated economics.

## What the archive adds
This is a new fact on a known story. [Our HarnessTax analysis](https://newruntime.com/posts/our-harnesstax-analysis-examines-how-harness-choice-changes-80954d/) framed harness design as a model- and task-dependent control surface. [Cheap, but unreliable](https://newruntime.com/posts/cheap-but-unreliable-smaller-ai-models-are-tempting-057529/) argued that a behavioral harness can make a cheaper coding model dependable on bounded tasks when feedback is available. The reanalysis adds a sharper qualification: a reliable evaluator may make repeated cheap-model attempts economically comparable to harnessing, before harness construction is counted.

## Trend signal
This supports [Harness architecture outlives model choice](https://newruntime.com/patterns/harness-architecture-outlives-model-choice/), but with an important boundary. The experiment shows that specifications, checks, mocks, and feedback can alter a smaller model's completion rate dramatically. Yet the reanalysis also shows that the harness is not automatically the cheapest answer: its value depends on the task's feedback surface, latency tolerance, caching, and the cost of producing the harness itself.

## Why now
The timing follows an attractive but incomplete comparison: $0.273 for harnessed Luna versus $0.406 for bare Sol. Dividing each setup's run cost by its observed probability of acceptance changes the practical question from “which run is cheaper?” to “what does one accepted application cost?” That reframing exposes both the near-tie with retries and the unpriced harness investment.

## Why it matters
Teams evaluating coding agents should not treat a model price sheet or a single pass rate as the unit of decision. The usable unit is a verified completed task, including retries, human or automated verification, latency, and the work required to create the feedback loop. On a tightly specified service with independent acceptance tests, a small model may be viable without a heavy harness if failures can be detected cheaply and retried safely. But that is not a license to ship unaudited outputs: in this experiment, hidden tests made failed Luna runs visible. Where no equivalent evaluator exists, retrying merely creates repeated uncertainty.

The practical comparison is therefore three-way: bare frontier model, cheap model plus task-specific harness, and cheap model plus verified retries. Teams should run all three on their own representative work rather than generalize from this one Java service and ten-run sample.

## What happens next
- **Teams will compare agent setups on cost per verified completion rather than cost per run.** (3 months, 70%). Watch for: evaluations that publish pass rates, retry policies, and verification costs together.
- **Harness evaluations will begin to separate build and maintenance cost from per-run inference cost.** (6 months, 60%). Watch for: studies reporting engineering effort and reuse across tasks.

## Limits and uncertainty
The evidence covers one favorable, well-specified Spring Boot task, ten runs per configuration, and a particular Codex setup at low reasoning effort. It does not establish results for other languages, architectures, agents, models, tasks, or organizational review processes. The retry calculation assumes independent attempts and depends on a strong external acceptance suite; neither condition is guaranteed in production. Reported costs are estimates rather than invoices, and cache-read proportions were not published, so different cache behavior or context-length pricing could substantially alter them. Most importantly, the experiment did not measure the time or cost to build, validate, and maintain the task-specific behavioral harness.

## Skill to build now
Choose one recurring agent task and make its acceptance rule executable this week. Keep the check independent of the agent's own explanation: a contract test, fixture-based integration test, build-and-start check, or another observable criterion. Then record pass rate, elapsed time, token use, and the number of retries for a bare configuration and a guided one. This produces the information needed to decide whether a harness is buying reliability, merely buying reassurance, or costing more than verified retries.

## Reader takeaway
A behavioral harness made Luna reliable on this task, but its apparent cost advantage over bare Sol is not the whole story. Once success probability is included, verified retries of bare Luna were nearly as cheap in the reanalysis. The deciding asset is not just a cheaper model or a richer prompt: it is a trustworthy way to tell completed work from plausible failure.

## Open question
How much engineering effort does a reusable behavioral harness require, and across how many tasks can that investment be amortized before it beats verified retries?

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Codex](https://newruntime.com/topics/codex.md): Explore the codex topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)