logging and inspection; and A/B testing for significant product changes.
The essay grounds the framework in Lucy, Rechat’s real-estate AI assistant. As its surface area expanded, prompt improvements stopped producing reliable progress: fixing one failure mode could expose another, and the team lacked visibility across tasks. The proposed response is an iterative system centered on evaluation and curation, with different evaluation levels run at different cadences. https://hamel.dev/blog/posts/evals