This shifts evaluation from choosing among public leaderboards alone toward testing configurations on work a developer actually does.
Observed facts: The author reports exporting about 1,270 sessions from Claude Code, Codex CLI, and Pi, then selecting 10 tasks for automated evaluation. In one comparison, Pi and native agents solved all tasks, while average cost per task was $0.070 for Pi and $0.138 for the native setup. The described process uses Harbor to package tasks and run agents in isolated environments.
Mechanism: A transcript is converted into a task specification that separates requirements, failure modes, and facts the agent must discover; a verifier tests the result, and repeated runs expose cost and speed differences.
Why now: The author’s guide and skill make this workflow more accessible to individual developers who want to compare their own agent setups.
First implication: Developers can use representative private tasks to decide whether a cheaper or faster configuration is adequate for their own work, rather than infer fit from a general benchmark.
Evidence boundary: These are the author’s results on a small set of personal tasks, not a general model ranking; the source does not establish that the same cost or success pattern will transfer to other repositories or workflows. It also notes that session exports may contain secrets and recommends scanning them before sharing.
First watch condition: Whether other developers can reproduce the workflow and obtain stable comparisons across repeated runs and independently selected tasks.