Hamel Husain updated his AI evals FAQ with five new questions, bringing it to 48 questions and answers. The additions cover whether a reference answer or rubric is needed before annotating data, running evals when traces contain sensitive data, reviewing very large traces, how much context to give an LLM judge, and what to do when a gold eval dataset becomes stale.
The new entries target operational problems that appear once evals run on real production traces. The post lists the questions only; the answers themselves are on the linked FAQ page and are not summarised here.