--- type: "post" id: "aa-agentperf-methodology-replays-trajectories-from-agents-resolving-cf52ce" slug: "aa-agentperf-methodology-replays-trajectories-from-agents-resolving-cf52ce" title: "AA-AgentPerf methodology replays trajectories from agents resolving GitHub issues and inserts pauses that simulate tool latency." description: "AA-AgentPerf’s central thesis is that inference serving should be evaluated on agentic workloads rather than only on uniform prompt traffic." retrieval_nugget: "AA-AgentPerf’s central thesis is that inference serving should be evaluated on agentic workloads rather than only on uniform prompt traffic." published_at: "2026-10-18" updated_at: "2026-10-18" record_date: "2026-10-18" date_kind: "published_at" topics: ["github"] source_urls: ["https://artificialanalysis.ai/methodology/agentperf"] schema_version: "newruntime-agent-readable-v0.2" stable_id: "post:aa-agentperf-methodology-replays-trajectories-from-agents-resolving-cf52ce" status: "published" visuals: [] editorial_provenance: {"schema_version":"newruntime-editorial-copy-v1","content_status":"source_grounded_final","final_copy_sha256":"sha256:ec50bcdae8aa023334e89923c65050eb432b2b7d45836d4a211aeceb0b9b1a35","reviewed_at":"2026-10-18T08:03:43.538Z","source_evidence_count":1,"verified_claim_count":2,"site_analysis_schema_version":"newruntime-site-analysis-v1","site_object_kind":"field_note","observed_fact_count":2,"implication_count":1,"watch_condition_count":1,"related_record_count":0} analysis: {"schema_version":"newruntime-site-analysis-v1","object_kind":"field_note","thesis":"AA-AgentPerf’s central thesis is that inference serving should be evaluated on agentic workloads rather than only on uniform prompt traffic.","observed_facts":[{"text":"Artificial Analysis says the dataset contains coding-agent trajectories generated in the OpenCode harness using DeepSeek V3.2, GLM 4.7, and Kimi K2.5 to resolve issues in public repositories.","source_urls":["https://artificialanalysis.ai/methodology/agentperf"]},{"text":"The methodology inserts tool-call delays from under 0.1 seconds to 5 seconds, with a median of approximately one second.","source_urls":["https://artificialanalysis.ai/methodology/agentperf"]}],"mechanism":"The mechanism is trajectory replay with simulated tool latency: after each tool call, the benchmark inserts a delay sampled from a distribution partitioned by tool type.","why_now":"Why now: as inference providers serve more multi-step coding-agent traffic, benchmark rankings based on isolated requests may not capture the interaction between concurrency, context reuse, scheduler behavior, and time spent waiting on tools.","implications":["The practical implication is that teams choosing serving configurations should ask whether their evaluation workload resembles their agents’ actual trajectories, not just their token mix."],"evidence_boundary":"Evidence boundary: the cited page documents the benchmark’s design and metrics; it does not, by itself, establish that any system is faster, cheaper, or more suitable than another.","watch_conditions":["Watch whether published leaderboard entries disclose comparable configurations and whether results remain stable across model and software-stack updates."],"related_records":[],"new_branch_reason":"This begins a distinct analytical record because no prior New Runtime record was supplied for this specific methodology."} editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]} routes: {"html":"https://newruntime.com/posts/aa-agentperf-methodology-replays-trajectories-from-agents-resolving-cf52ce/","markdown":"https://newruntime.com/posts/aa-agentperf-methodology-replays-trajectories-from-agents-resolving-cf52ce.md","json":"https://newruntime.com/posts/aa-agentperf-methodology-replays-trajectories-from-agents-resolving-cf52ce.json"} related_patterns: ["agent-economics-moves-to-completed-work"] --- # AA-AgentPerf methodology replays trajectories from agents resolving GitHub issues and inserts pauses that simulate tool latency. ## Retrieval answer AA-AgentPerf’s central thesis is that inference serving should be evaluated on agentic workloads rather than only on uniform prompt traffic. AA-AgentPerf’s central thesis is that inference serving should be evaluated on agentic workloads rather than only on uniform prompt traffic. Artificial Analysis describes a dataset of coding-agent trajectories generated in the OpenCode harness, where DeepSeek V3.2, GLM 4.7, and Kimi K2.5 were prompted to resolve issues in public repositories. The reported input sequence lengths span about 5,000 to 131,000 tokens, with a mean near 27,000. The mechanism is trajectory replay with simulated tool latency: after each tool call, the benchmark inserts a delay sampled from a distribution partitioned by tool type. The stated range is under 0.1 seconds to 5 seconds, with a median of about one second. This changes the load shape from a stream of independent prompts to concurrent agents alternating between model work and tool waits. The methodology then compares total server throughput with per-request output speed across configurations. Why now: as inference providers serve more multi-step coding-agent traffic, benchmark rankings based on isolated requests may not capture the interaction between concurrency, context reuse, scheduler behavior, and time spent waiting on tools. The practical implication is that teams choosing serving configurations should ask whether their evaluation workload resembles their agents’ actual trajectories, not just their token mix. Evidence boundary: the cited page documents the benchmark’s design and metrics; it does not, by itself, establish that any system is faster, cheaper, or more suitable than another. Watch whether published leaderboard entries disclose comparable configurations and whether results remain stable across model and software-stack updates. This begins a distinct analytical record because no prior New Runtime record was supplied for this specific methodology. Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json ## Continue in the graph Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`. - topic hub: [Github](https://newruntime.com/topics/github.md): Explore the github topic hub. - collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis. - collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph. - whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)