{"type":"post","id":"ibragim-bad-published-a-guide-and-a-companion-e8d8ee","slug":"ibragim-bad-published-a-guide-and-a-companion-e8d8ee","title":"Ibragim Bad published a guide and a companion skill for turning coding-agent session histories into","description":"**Thesis:** Personal coding-agent histories can become a practical evaluation set when past tasks are isolated, specified, and paired with a verifier.","retrieval_nugget":"**Thesis:** Personal coding-agent histories can become a practical evaluation set when past tasks are isolated, specified, and paired with a verifier.","published_at":"2026-10-16","updated_at":"2026-10-16","record_date":"2026-10-16","date_kind":"published_at","topics":["coding-agents","evals","skills"],"source_urls":["https://teletype.in/@ibragim_bad/fm5jSJvYb_5"],"schema_version":"newruntime-agent-readable-v0.2","stable_id":"post:ibragim-bad-published-a-guide-and-a-companion-e8d8ee","status":"published","visuals":[],"editorial_provenance":{"schema_version":"newruntime-editorial-copy-v1","content_status":"source_grounded_final","final_copy_sha256":"sha256:49bf6965bbfa5288d96cdef18a60a558b2e21c9e1fe9ca6f8bd3c46821f64151","reviewed_at":"2026-10-16T20:11:23.635Z","source_evidence_count":1,"verified_claim_count":3,"site_analysis_schema_version":"newruntime-site-analysis-v1","site_object_kind":"field_note","observed_fact_count":3,"implication_count":1,"watch_condition_count":1,"related_record_count":0},"analysis":{"schema_version":"newruntime-site-analysis-v1","object_kind":"field_note","thesis":"Personal coding-agent histories can become a practical evaluation set when past tasks are isolated, specified, and paired with a verifier.","observed_facts":[{"text":"The author reports exporting about 1,270 sessions from Claude Code, Codex CLI, and Pi, then selecting 10 tasks for automated evaluation.","source_urls":["https://teletype.in/@ibragim_bad/fm5jSJvYb_5"]},{"text":"In one comparison, Pi and native agents solved all tasks, while average cost per task was $0.070 for Pi and $0.138 for the native setup.","source_urls":["https://teletype.in/@ibragim_bad/fm5jSJvYb_5"]},{"text":"The described process uses Harbor to package tasks and run agents in isolated environments.","source_urls":["https://teletype.in/@ibragim_bad/fm5jSJvYb_5"]}],"mechanism":"A transcript is converted into a task specification that separates requirements, failure modes, and facts the agent must discover; a verifier tests the result, and repeated runs expose cost and speed differences.","why_now":"The author’s guide and skill make this workflow more accessible to individual developers who want to compare their own agent setups.","implications":["Developers can use representative private tasks to decide whether a cheaper or faster configuration is adequate for their own work, rather than infer fit from a general benchmark."],"evidence_boundary":"These are the author’s results on a small set of personal tasks, not a general model ranking; the source does not establish that the same cost or success pattern will transfer to other repositories or workflows. It also notes that session exports may contain secrets and recommends scanning them before sharing.","watch_conditions":["Whether other developers can reproduce the workflow and obtain stable comparisons across repeated runs and independently selected tasks."],"related_records":[],"new_branch_reason":"No prior New Runtime record is supplied; this starts a distinct analytical record about converting personal coding-agent histories into verified evaluation sets."},"editor":{"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]},"routes":{"html":"https://newruntime.com/posts/ibragim-bad-published-a-guide-and-a-companion-e8d8ee/","markdown":"https://newruntime.com/posts/ibragim-bad-published-a-guide-and-a-companion-e8d8ee.md","json":"https://newruntime.com/posts/ibragim-bad-published-a-guide-and-a-companion-e8d8ee.json"},"related_patterns":["ai-engineering-evaluations-are-splitting-into-task-specific"],"related_shifts":["owned-learning-loop"],"next_reads":[{"type":"topic","path":"/topics/coding-agents/","url":"https://newruntime.com/topics/coding-agents/","title":"Coding Agents","reason":"Explore the coding-agents topic hub.","media_type":"text/html","markdown_url":"https://newruntime.com/topics/coding-agents.md","json_url":"https://newruntime.com/topics/coding-agents.json"},{"type":"topic","path":"/topics/evals/","url":"https://newruntime.com/topics/evals/","title":"Evals","reason":"Explore the evals topic hub.","media_type":"text/html","markdown_url":"https://newruntime.com/topics/evals.md","json_url":"https://newruntime.com/topics/evals.json"},{"type":"topic","path":"/topics/skills/","url":"https://newruntime.com/topics/skills/","title":"Skills","reason":"Explore the skills topic hub.","media_type":"text/html","markdown_url":"https://newruntime.com/topics/skills.md","json_url":"https://newruntime.com/topics/skills.json"},{"type":"collection","path":"/posts/","url":"https://newruntime.com/posts/","title":"Field Notes","reason":"Open the latest editorial analysis.","media_type":"text/html","json_url":"https://newruntime.com/posts.json"},{"type":"collection","path":"/source-ledger.json","url":"https://newruntime.com/source-ledger.json","title":"Source Ledger","reason":"Inspect the public source evidence graph.","media_type":"application/json"}]}