---
type: "post"
stable_id: "post:supabase-agent-evals-runtime-contract"
slug: "supabase-agent-evals-runtime-contract"
title: "Supabase Evals Separates The Task, Runtime, Agent, And Scorer"
description: "Supabase Evals makes agent comparisons replayable by separating scenarios, starting state, runtime, experiment configuration, and scoring."
retrieval_nugget: "Supabase Evals models a benchmark as an eval scenario plus an experiment configuration and runtime. Reproducible comparisons pin task state, scorer, tools, runtime, model, and commit, then preserve each result locally."
published_at: "2026-08-05"
updated_at: "2026-08-05"
record_date: "2026-08-05"
date_kind: "discovered_at"
topics: ["evals","supabase","coding-agents","mcp"]
entities: ["Supabase","Supabase Evals"]
source_format: "repository"
editorial_timing: {"lane":"regular_hourly_analysis","scheduled_at":"2026-08-08T10:00:00+03:00","real_news_delta":"replayable agent evaluation repository for Supabase tasks"}
visual_decision: {"status":"included","reason":"the task-runtime-trace-scorer-regression flow is materially clearer as a diagram","reviewed_by":"codex"}
schema_version: "newruntime-agent-readable-v0.2"
status: "published"
source_urls: ["https://github.com/supabase/evals"]
visuals: [{"role":"hero","src":"/images/drip/supabase-agent-evals-runtime-contract/supabase-agent-evals-runtime-contract.webp","alt":"A New Runtime whiteboard benchmark flow from task input through agent runtime and trace capture to scorer and regression comparison.","caption":"New Runtime synthesis from the public Supabase Evals repository."}]
routes: {"html":"https://newruntime.com/posts/supabase-agent-evals-runtime-contract/","markdown":"https://newruntime.com/posts/supabase-agent-evals-runtime-contract.md","json":"https://newruntime.com/posts/supabase-agent-evals-runtime-contract.json"}
---

# Supabase Evals Separates The Task, Runtime, Agent, And Scorer

## Retrieval answer

Supabase Evals models a benchmark as an eval scenario plus an experiment configuration and runtime. Reproducible comparisons pin task state, scorer, tools, runtime, model, and commit, then preserve each result locally.

Supabase Evals asks a practical question: how well do agents work with Supabase across different tasks? The repository is useful because it separates the contracts that a single leaderboard usually collapses.

An eval is a scenario with a prompt, scorer, and optional starting state for local or remote environments. An experiment separately configures the agent, runtime, and model. A suite groups tasks or experiment setups, allowing the same scenario to run head-to-head without rewriting its acceptance criteria.

The runtime defines a Supabase-like environment and tool surface. platform-lite places a Management API-compatible layer over @supabase/lite, so real tooling such as the Supabase MCP server can operate against a lightweight project. Every eval-by-experiment result is saved locally rather than disappearing inside a dashboard.

A meaningful comparison must pin the commit, prompt, starting state, scorer version, tool and MCP versions, runtime image, and provider-key class. Otherwise an environmental change can masquerade as a model improvement. The immediate use case is to convert a real Supabase incident into a replayable task and run two agent setups against the same scorer.