{"type":"post","stable_id":"post:supabase-agent-evals-runtime-contract","slug":"supabase-agent-evals-runtime-contract","title":"Supabase Evals Separates The Task, Runtime, Agent, And Scorer","description":"Supabase Evals makes agent comparisons replayable by separating scenarios, starting state, runtime, experiment configuration, and scoring.","retrieval_nugget":"Supabase Evals models a benchmark as an eval scenario plus an experiment configuration and runtime. Reproducible comparisons pin task state, scorer, tools, runtime, model, and commit, then preserve each result locally.","published_at":"2026-08-05","updated_at":"2026-08-05","record_date":"2026-08-05","date_kind":"discovered_at","topics":["evals","supabase","coding-agents","mcp"],"entities":["Supabase","Supabase Evals"],"source_format":"repository","editorial_timing":{"lane":"regular_hourly_analysis","scheduled_at":"2026-08-08T10:00:00+03:00","real_news_delta":"replayable agent evaluation repository for Supabase tasks"},"visual_decision":{"status":"included","reason":"the task-runtime-trace-scorer-regression flow is materially clearer as a diagram","reviewed_by":"codex"},"schema_version":"newruntime-agent-readable-v0.2","status":"published","source_urls":["https://github.com/supabase/evals"],"visuals":[{"role":"hero","src":"/images/drip/supabase-agent-evals-runtime-contract/supabase-agent-evals-runtime-contract.webp","alt":"A New Runtime whiteboard benchmark flow from task input through agent runtime and trace capture to scorer and regression comparison.","caption":"New Runtime synthesis from the public Supabase Evals repository."}],"routes":{"html":"https://newruntime.com/posts/supabase-agent-evals-runtime-contract/","markdown":"https://newruntime.com/posts/supabase-agent-evals-runtime-contract.md","json":"https://newruntime.com/posts/supabase-agent-evals-runtime-contract.json"}}