Google’s Agent and Model Evaluations release closes a common measurement gap: the test suite used before launch and the monitoring system used afterward can now run the same metrics.
The service ships with more than 20 metrics across quality, safety, grounding, tool use, trajectories, and reference-based tasks. Teams can also register deterministic Python checks or custom LLM judges. Every metric lives in a versioned organization-wide registry, so a production drift is less likely to be an artifact of changing the ruler.
Adaptive rubrics create case-specific pass/fail criteria from the task, developer instructions, and available tools. Google provides variants for task success, tool selection and arguments, trajectory quality, final responses, hallucination, grounding, and safety. Each failed run can be opened down to its trace and session log.
The experiment layer can generate cases, simulate multi-turn users, and intercept tool calls with a mocked environment. That makes it possible to force a slow backend, an error, or specific data without touching production. Server-side runs store artifacts in Cloud Storage for replay and audit.
After launch, online monitors sample real traces, apply the same metric definitions, and produce score-over-time dashboards and drift alerts. Issue clustering groups failures into an operator-defined taxonomy, turning a pile of low scores into a smaller set of repair queues.
The useful contract is continuous: every important metric must name the development cases it protects and the production traffic it samples. Offline green is a release condition. Online drift is a reason to create the next case.