---
type: "raw_signal"
id: "cheap-but-unreliable-smaller-ai-models-are-tempting-22fb9f"
slug: "cheap-but-unreliable-smaller-ai-models-are-tempting-22fb9f"
title: "Cheap, but unreliable. Smaller AI models are tempting for coding tasks because they cost less, and teams rarely choose a model on raw intelligence alone."
description: "Naresh Jain reports a controlled comparison of coding agents building the same API service with and without a behavioral harness."
retrieval_nugget: "Naresh Jain reports a controlled comparison of coding agents building the same API service with and without a behavioral harness."
observed_at: "2026-10-20"
record_date: "2026-10-20"
date_kind: "observed_at"
why_it_matters: "Naresh Jain reports a controlled comparison of coding agents building the same API service with and without a behavioral harness. In 10 runs, GPT-6 Luna passed all 178 external acceptance tests in 3 runs without the harness and in all 10 runs with it. The harness supplied API specifications, examples, and deterministic checks that returned feedback during development. Harnessed Luna used more toke"
novelty: "notable"
verification_level: "source-linked"
signal_type: "research"
source_platform: "martinfowler.com"
topics: ["coding-agents"]
entities: ["Naresh Jain","GPT-6 Luna","GPT-6 Sol","Specmatic","Codex"]
related_patterns: ["harness-architecture-outlives-model-choice"]
novelty_label: "new_fact"
what_changed: "A coding-agent experiment found that a behavioral harness raised GPT-6 Luna success from 3/10 to 10/10 runs while costing 33% less per run than GPT-6 Sol."
prior_refs: ["memory:d022fda3-eff2-8ae9-a647-8d03d077547a","memory:77816356-a997-8f4a-a775-a470f0f88a64"]
source_url: "https://martinfowler.com/articles/are-harnesses-worth-the-cost.html"
source_urls: ["https://martinfowler.com/articles/are-harnesses-worth-the-cost.html"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "signal:cheap-but-unreliable-smaller-ai-models-are-tempting-22fb9f"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/signals/cheap-but-unreliable-smaller-ai-models-are-tempting-22fb9f/","markdown":"https://newruntime.com/signals/cheap-but-unreliable-smaller-ai-models-are-tempting-22fb9f.md","json":"https://newruntime.com/signals/cheap-but-unreliable-smaller-ai-models-are-tempting-22fb9f.json"}
---

# Cheap, but unreliable. Smaller AI models are tempting for coding tasks because they cost less, and teams rarely choose a model on raw intelligence alone.

## Retrieval answer

Naresh Jain reports a controlled comparison of coding agents building the same API service with and without a behavioral harness.

Naresh Jain reports a controlled comparison of coding agents building the same API service with and without a behavioral harness. In 10 runs, GPT-6 Luna passed all 178 external acceptance tests in 3 runs without the harness and in all 10 runs with it. The harness supplied API specifications, examples, and deterministic checks that returned feedback during development. Harnessed Luna used more tokens and time, but its average estimated cost was $0.273 per run, versus $0.406 for GPT-6 Sol without a harness—a reported 33% lower cost. GPT-6 Sol passed in all 10 runs in either condition; adding the harness made its runs substantially slower and more expensive without improving the observed pass rate. The experiment concerns one detailed service brief and does not establish how the method generalizes to other tasks.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Coding Agents](https://newruntime.com/topics/coding-agents.md): Explore the coding-agents topic hub.
- collection: [Raw Signals](https://newruntime.com/signals.json): Open every source-linked raw signal.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)