Field note
Supabase Evals asks a practical question: how well do agents work with Supabase across different tasks? The repository is useful because it separates the contracts that a single leaderboard usually collapses.
An eval is a scenario with a prompt, scorer, and optional starting state for local or remote environments. An experiment separately configures the agent, runtime, and model. A suite groups tasks or experiment setups, allowing the same scenario to run head-to-head without rewriting its acceptance criteria.
The runtime defines a Supabase-like environment and tool surface. platform-lite places a Management API-compatible layer over @supabase/lite, so real tooling such as the Supabase MCP server can operate against a lightweight project. Every eval-by-experiment result is saved locally rather than disappearing inside a dashboard.
A meaningful comparison must pin the commit, prompt, starting state, scorer version, tool and MCP versions, runtime image, and provider-key class. Otherwise an environmental change can masquerade as a model improvement. The immediate use case is to convert a real Supabase incident into a replayable task and run two agent setups against the same scorer.