{"type":"raw_signal","id":"benchmarks-pick-the-engine-3365ab","slug":"benchmarks-pick-the-engine-3365ab","title":"Benchmarks pick the engine.","description":"Hamel Husain’s 2024 essay “Your AI Product Needs Evals” argues that teams should evaluate AI products against user tasks, rather than rely on general model benchmarks or informal “vibe checks.” It describes three levels: scoped, fast unit-style tests; human and model evaluation informed by trace","retrieval_nugget":"Hamel Husain’s 2024 essay “Your AI Product Needs Evals” argues that teams should evaluate AI products against user tasks, rather than rely on general model benchmarks or informal “vibe checks.” It describes three levels: scoped, fast unit-style tests; human and model evaluation informed by trace","observed_at":"2026-10-10","record_date":"2026-10-10","date_kind":"observed_at","why_it_matters":"The essay grounds the framework in Lucy, Rechat’s real-estate AI assistant. As its surface area expanded, prompt improvements stopped producing reliable progress: fixing one failure mode could expose another, and the team lacked visibility across tasks. The proposed response is an iterative system centered on evaluation and curation, with different evaluation levels run at different cadences. http","novelty":"notable","verification_level":"source-linked","signal_type":"research","source_platform":"hamel.dev","topics":[],"entities":["Hamel Husain","Rechat","Lucy"],"related_patterns":[],"source_url":"https://hamel.dev/blog/posts/evals","source_urls":["https://hamel.dev/blog/posts/evals"],"schema_version":"newruntime-agent-readable-v0.2","stable_id":"signal:benchmarks-pick-the-engine-3365ab","status":"published","visuals":[],"editor":{"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]},"routes":{"html":"https://newruntime.com/signals/benchmarks-pick-the-engine-3365ab/","markdown":"https://newruntime.com/signals/benchmarks-pick-the-engine-3365ab.md","json":"https://newruntime.com/signals/benchmarks-pick-the-engine-3365ab.json"},"next_reads":[{"type":"collection","path":"/signals/","url":"https://newruntime.com/signals/","title":"Raw Signals","reason":"Open every source-linked raw signal.","media_type":"text/html","json_url":"https://newruntime.com/signals.json"},{"type":"collection","path":"/source-ledger.json","url":"https://newruntime.com/source-ledger.json","title":"Source Ledger","reason":"Inspect the public source evidence graph.","media_type":"application/json"}]}