{"type":"raw_signal","id":"cheap-but-unreliable-smaller-ai-models-are-tempting-22fb9f","slug":"cheap-but-unreliable-smaller-ai-models-are-tempting-22fb9f","title":"Cheap, but unreliable. Smaller AI models are tempting for coding tasks because they cost less, and teams rarely choose a model on raw intelligence alone.","description":"Naresh Jain reports a controlled comparison of coding agents building the same API service with and without a behavioral harness.","retrieval_nugget":"Naresh Jain reports a controlled comparison of coding agents building the same API service with and without a behavioral harness.","observed_at":"2026-10-20","record_date":"2026-10-20","date_kind":"observed_at","why_it_matters":"Naresh Jain reports a controlled comparison of coding agents building the same API service with and without a behavioral harness. In 10 runs, GPT-6 Luna passed all 178 external acceptance tests in 3 runs without the harness and in all 10 runs with it. The harness supplied API specifications, examples, and deterministic checks that returned feedback during development. Harnessed Luna used more toke","novelty":"notable","verification_level":"source-linked","signal_type":"research","source_platform":"martinfowler.com","topics":["coding-agents"],"entities":["Naresh Jain","GPT-6 Luna","GPT-6 Sol","Specmatic","Codex"],"related_patterns":["harness-architecture-outlives-model-choice"],"novelty_label":"new_fact","what_changed":"A coding-agent experiment found that a behavioral harness raised GPT-6 Luna success from 3/10 to 10/10 runs while costing 33% less per run than GPT-6 Sol.","prior_refs":["memory:d022fda3-eff2-8ae9-a647-8d03d077547a","memory:77816356-a997-8f4a-a775-a470f0f88a64"],"source_url":"https://martinfowler.com/articles/are-harnesses-worth-the-cost.html","source_urls":["https://martinfowler.com/articles/are-harnesses-worth-the-cost.html"],"schema_version":"newruntime-agent-readable-v0.2","stable_id":"signal:cheap-but-unreliable-smaller-ai-models-are-tempting-22fb9f","status":"published","visuals":[],"editor":{"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]},"routes":{"html":"https://newruntime.com/signals/cheap-but-unreliable-smaller-ai-models-are-tempting-22fb9f/","markdown":"https://newruntime.com/signals/cheap-but-unreliable-smaller-ai-models-are-tempting-22fb9f.md","json":"https://newruntime.com/signals/cheap-but-unreliable-smaller-ai-models-are-tempting-22fb9f.json"},"next_reads":[{"type":"topic","path":"/topics/coding-agents/","url":"https://newruntime.com/topics/coding-agents/","title":"Coding Agents","reason":"Explore the coding-agents topic hub.","media_type":"text/html","markdown_url":"https://newruntime.com/topics/coding-agents.md","json_url":"https://newruntime.com/topics/coding-agents.json"},{"type":"collection","path":"/signals/","url":"https://newruntime.com/signals/","title":"Raw Signals","reason":"Open every source-linked raw signal.","media_type":"text/html","json_url":"https://newruntime.com/signals.json"},{"type":"collection","path":"/source-ledger.json","url":"https://newruntime.com/source-ledger.json","title":"Source Ledger","reason":"Inspect the public source evidence graph.","media_type":"application/json"}]}