---
type: "raw_signal"
id: "benchmarks-pick-the-engine-3365ab"
slug: "benchmarks-pick-the-engine-3365ab"
title: "Benchmarks pick the engine."
description: "Hamel Husain’s 2024 essay “Your AI Product Needs Evals” argues that teams should evaluate AI products against user tasks, rather than rely on general model benchmarks or informal “vibe checks.” It describes three levels: scoped, fast unit-style tests; human and model evaluation informed by trace"
retrieval_nugget: "Hamel Husain’s 2024 essay “Your AI Product Needs Evals” argues that teams should evaluate AI products against user tasks, rather than rely on general model benchmarks or informal “vibe checks.” It describes three levels: scoped, fast unit-style tests; human and model evaluation informed by trace"
observed_at: "2026-10-10"
record_date: "2026-10-10"
date_kind: "observed_at"
why_it_matters: "The essay grounds the framework in Lucy, Rechat’s real-estate AI assistant. As its surface area expanded, prompt improvements stopped producing reliable progress: fixing one failure mode could expose another, and the team lacked visibility across tasks. The proposed response is an iterative system centered on evaluation and curation, with different evaluation levels run at different cadences. http"
novelty: "notable"
verification_level: "source-linked"
signal_type: "research"
source_platform: "hamel.dev"
topics: []
entities: ["Hamel Husain","Rechat","Lucy"]
related_patterns: []
source_url: "https://hamel.dev/blog/posts/evals"
source_urls: ["https://hamel.dev/blog/posts/evals"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "signal:benchmarks-pick-the-engine-3365ab"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/signals/benchmarks-pick-the-engine-3365ab/","markdown":"https://newruntime.com/signals/benchmarks-pick-the-engine-3365ab.md","json":"https://newruntime.com/signals/benchmarks-pick-the-engine-3365ab.json"}
---

# Benchmarks pick the engine.

## Retrieval answer

Hamel Husain’s 2024 essay “Your AI Product Needs Evals” argues that teams should evaluate AI products against user tasks, rather than rely on general model benchmarks or informal “vibe checks.” It describes three levels: scoped, fast unit-style tests; human and model evaluation informed by trace

Hamel Husain’s 2024 essay “Your AI Product Needs Evals” argues that teams should evaluate AI products against user tasks, rather than rely on general model benchmarks or informal “vibe checks.” It describes three levels: scoped, fast unit-style tests; human and model evaluation informed by trace logging and inspection; and A/B testing for significant product changes.

The essay grounds the framework in Lucy, Rechat’s real-estate AI assistant. As its surface area expanded, prompt improvements stopped producing reliable progress: fixing one failure mode could expose another, and the team lacked visibility across tasks. The proposed response is an iterative system centered on evaluation and curation, with different evaluation levels run at different cadences. https://hamel.dev/blog/posts/evals

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- collection: [Raw Signals](https://newruntime.com/signals.json): Open every source-linked raw signal.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)