---
type: "post"
id: "ibragim-bad-published-a-guide-and-a-companion-e8d8ee"
slug: "ibragim-bad-published-a-guide-and-a-companion-e8d8ee"
title: "Ibragim Bad published a guide and a companion skill for turning coding-agent session histories into"
description: "**Thesis:** Personal coding-agent histories can become a practical evaluation set when past tasks are isolated, specified, and paired with a verifier."
retrieval_nugget: "**Thesis:** Personal coding-agent histories can become a practical evaluation set when past tasks are isolated, specified, and paired with a verifier."
published_at: "2026-10-16"
updated_at: "2026-10-16"
record_date: "2026-10-16"
date_kind: "published_at"
topics: ["coding-agents","evals","skills"]
source_urls: ["https://teletype.in/@ibragim_bad/fm5jSJvYb_5"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:ibragim-bad-published-a-guide-and-a-companion-e8d8ee"
status: "published"
visuals: []
editorial_provenance: {"schema_version":"newruntime-editorial-copy-v1","content_status":"source_grounded_final","final_copy_sha256":"sha256:49bf6965bbfa5288d96cdef18a60a558b2e21c9e1fe9ca6f8bd3c46821f64151","reviewed_at":"2026-10-16T20:11:23.635Z","source_evidence_count":1,"verified_claim_count":3,"site_analysis_schema_version":"newruntime-site-analysis-v1","site_object_kind":"field_note","observed_fact_count":3,"implication_count":1,"watch_condition_count":1,"related_record_count":0}
analysis: {"schema_version":"newruntime-site-analysis-v1","object_kind":"field_note","thesis":"Personal coding-agent histories can become a practical evaluation set when past tasks are isolated, specified, and paired with a verifier.","observed_facts":[{"text":"The author reports exporting about 1,270 sessions from Claude Code, Codex CLI, and Pi, then selecting 10 tasks for automated evaluation.","source_urls":["https://teletype.in/@ibragim_bad/fm5jSJvYb_5"]},{"text":"In one comparison, Pi and native agents solved all tasks, while average cost per task was $0.070 for Pi and $0.138 for the native setup.","source_urls":["https://teletype.in/@ibragim_bad/fm5jSJvYb_5"]},{"text":"The described process uses Harbor to package tasks and run agents in isolated environments.","source_urls":["https://teletype.in/@ibragim_bad/fm5jSJvYb_5"]}],"mechanism":"A transcript is converted into a task specification that separates requirements, failure modes, and facts the agent must discover; a verifier tests the result, and repeated runs expose cost and speed differences.","why_now":"The author’s guide and skill make this workflow more accessible to individual developers who want to compare their own agent setups.","implications":["Developers can use representative private tasks to decide whether a cheaper or faster configuration is adequate for their own work, rather than infer fit from a general benchmark."],"evidence_boundary":"These are the author’s results on a small set of personal tasks, not a general model ranking; the source does not establish that the same cost or success pattern will transfer to other repositories or workflows. It also notes that session exports may contain secrets and recommends scanning them before sharing.","watch_conditions":["Whether other developers can reproduce the workflow and obtain stable comparisons across repeated runs and independently selected tasks."],"related_records":[],"new_branch_reason":"No prior New Runtime record is supplied; this starts a distinct analytical record about converting personal coding-agent histories into verified evaluation sets."}
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/ibragim-bad-published-a-guide-and-a-companion-e8d8ee/","markdown":"https://newruntime.com/posts/ibragim-bad-published-a-guide-and-a-companion-e8d8ee.md","json":"https://newruntime.com/posts/ibragim-bad-published-a-guide-and-a-companion-e8d8ee.json"}
related_patterns: ["ai-engineering-evaluations-are-splitting-into-task-specific"]
related_shifts: ["owned-learning-loop"]
---

# Ibragim Bad published a guide and a companion skill for turning coding-agent session histories into

## Retrieval answer

**Thesis:** Personal coding-agent histories can become a practical evaluation set when past tasks are isolated, specified, and paired with a verifier.

**Thesis:** Personal coding-agent histories can become a practical evaluation set when past tasks are isolated, specified, and paired with a verifier. This shifts evaluation from choosing among public leaderboards alone toward testing configurations on work a developer actually does.

**Observed facts:** The author reports exporting about 1,270 sessions from Claude Code, Codex CLI, and Pi, then selecting 10 tasks for automated evaluation. In one comparison, Pi and native agents solved all tasks, while average cost per task was $0.070 for Pi and $0.138 for the native setup. The described process uses Harbor to package tasks and run agents in isolated environments.

**Mechanism:** A transcript is converted into a task specification that separates requirements, failure modes, and facts the agent must discover; a verifier tests the result, and repeated runs expose cost and speed differences.

**Why now:** The author’s guide and skill make this workflow more accessible to individual developers who want to compare their own agent setups.

**First implication:** Developers can use representative private tasks to decide whether a cheaper or faster configuration is adequate for their own work, rather than infer fit from a general benchmark.

**Evidence boundary:** These are the author’s results on a small set of personal tasks, not a general model ranking; the source does not establish that the same cost or success pattern will transfer to other repositories or workflows. It also notes that session exports may contain secrets and recommends scanning them before sharing.

**First watch condition:** Whether other developers can reproduce the workflow and obtain stable comparisons across repeated runs and independently selected tasks.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Coding Agents](https://newruntime.com/topics/coding-agents.md): Explore the coding-agents topic hub.
- topic hub: [Evals](https://newruntime.com/topics/evals.md): Explore the evals topic hub.
- topic hub: [Skills](https://newruntime.com/topics/skills.md): Explore the skills topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)