---
type: "post"
id: "cheap-but-unreliable-smaller-ai-models-are-tempting-057529"
slug: "cheap-but-unreliable-smaller-ai-models-are-tempting-057529"
title: "Cheap, but unreliable. Smaller AI models are tempting for coding tasks because they cost less, and teams rarely choose a model on raw intelligence alone."
description: "Thesis: A behavioral harness can make a cost-optimized coding model more dependable on a bounded task, but its value depends on the model and the task’s feedback surface."
retrieval_nugget: "Thesis: A behavioral harness can make a cost-optimized coding model more dependable on a bounded task, but its value depends on the model and the task’s feedback surface."
published_at: "2026-10-20"
updated_at: "2026-10-20"
record_date: "2026-10-20"
date_kind: "published_at"
topics: ["models"]
source_urls: ["https://martinfowler.com/articles/are-harnesses-worth-the-cost.html"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:cheap-but-unreliable-smaller-ai-models-are-tempting-057529"
status: "published"
visuals: []
editorial_provenance: {"schema_version":"newruntime-editorial-copy-v1","content_status":"source_grounded_final","final_copy_sha256":"sha256:8c66809246e91942d33605d56fb7c88836bfc3e0b7faea6c61faf3398d3f81a3","reviewed_at":"2026-10-20T14:09:03.498Z","source_evidence_count":1,"verified_claim_count":3,"site_analysis_schema_version":"newruntime-site-analysis-v1","site_object_kind":"field_note","observed_fact_count":3,"implication_count":1,"watch_condition_count":1,"related_record_count":0}
analysis: {"schema_version":"newruntime-site-analysis-v1","object_kind":"field_note","thesis":"A behavioral harness can make a cost-optimized coding model more dependable on a bounded task, but its value depends on the model and the task’s feedback surface.","observed_facts":[{"text":"GPT-6 Luna passed all 178 external acceptance tests in 3 of 10 unharnessed runs and 10 of 10 harnessed runs.","source_urls":["https://martinfowler.com/articles/are-harnesses-worth-the-cost.html"]},{"text":"GPT-6 Sol passed in all 10 runs with and without the harness; the harnessed runs were substantially slower and more expensive.","source_urls":["https://martinfowler.com/articles/are-harnesses-worth-the-cost.html"]},{"text":"Harnessed GPT-6 Luna averaged an estimated $0.273 per run versus $0.406 for GPT-6 Sol without a harness, a reported 33% lower cost.","source_urls":["https://martinfowler.com/articles/are-harnesses-worth-the-cost.html"]}],"mechanism":"Machine-readable API specifications and examples guided development, while deterministic checks and test feedback acted as sensors during coding.","why_now":"Teams are weighing less costly coding models, and this experiment measures repeated-run reliability and cost rather than relying on a single demo.","implications":["A team evaluating smaller coding models should compare repeated task completion with and without task-specific feedback, not just model price or raw token use."],"evidence_boundary":"This is one experiment on one unusually detailed service brief; it does not show that harnesses will improve every model, task, or production outcome.","watch_conditions":["Whether similar repeated-run comparisons on less complete briefs and different software tasks preserve the reliability and cost advantage."],"related_records":[],"new_branch_reason":"No prior New Runtime record for this experiment was supplied; this establishes a distinct record focused on measured harness effects in a bounded coding-model comparison."}
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/cheap-but-unreliable-smaller-ai-models-are-tempting-057529/","markdown":"https://newruntime.com/posts/cheap-but-unreliable-smaller-ai-models-are-tempting-057529.md","json":"https://newruntime.com/posts/cheap-but-unreliable-smaller-ai-models-are-tempting-057529.json"}
related_patterns: ["harness-architecture-outlives-model-choice"]
---

# Cheap, but unreliable. Smaller AI models are tempting for coding tasks because they cost less, and teams rarely choose a model on raw intelligence alone.

## Retrieval answer

Thesis: A behavioral harness can make a cost-optimized coding model more dependable on a bounded task, but its value depends on the model and the task’s feedback surface.

Thesis: A behavioral harness can make a cost-optimized coding model more dependable on a bounded task, but its value depends on the model and the task’s feedback surface. In Naresh Jain’s experiment, GPT-6 Luna passed all 178 external acceptance tests in 3 of 10 unharnessed runs and 10 of 10 harnessed runs. GPT-6 Sol passed in all 10 runs either way.

Mechanism: Machine-readable API specifications and examples guided development, while deterministic checks and test feedback acted as sensors during coding. This turns part of the work of anticipating and detecting mistakes into executable feedback. Why now: Teams are weighing less costly coding models, and this experiment measures repeated-run reliability and cost rather than relying on a single demo. The harness raised Luna’s token use and runtime, yet its estimated average run cost was $0.273 versus $0.406 for GPT-6 Sol without a harness, a reported 33% difference.

First implication: A team evaluating smaller coding models should compare repeated task completion with and without task-specific feedback, not just model price or raw token use. Evidence boundary: This is one experiment on one unusually detailed service brief; it does not show that harnesses will improve every model, task, or production outcome. First watch condition: Whether similar repeated-run comparisons on less complete briefs and different software tasks preserve the reliability and cost advantage.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Models](https://newruntime.com/topics/models.md): Explore the models topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)