{"schema_version":"newruntime-agent-readable-v0.2","type":"post","stable_id":"post:chatgpt-agent-loop-efficiency-stack","slug":"chatgpt-agent-loop-efficiency-stack","title":"ChatGPT Cuts Repeated Work Across The Agent Stack","description":"ByteByteGo's OpenAI engineering walkthrough connects persistent sessions, stable prompt prefixes, deferred tools, delta tokenization, cache-aware routing, and split inference.","retrieval_nugget":"ByteByteGo's OpenAI engineering walkthrough connects persistent sessions, stable prompt prefixes, deferred tools, delta tokenization, cache-aware routing, and split inference. ByteByteGo's walkthrough with OpenAI engineers is useful because it traces agent efficiency through three layers instead of attributing it to one faster model. The recurring objective is to avoid paying for the same work twice.","status":"published","published_at":"2026-08-03","updated_at":"2026-08-03","record_date":"2026-08-03","date_kind":"published_at","topics":["agent-harnesses","inference","performance","context-engineering"],"source_urls":["https://blog.bytebytego.com/p/how-chatgpt-optimizes-its-agent-loop"],"visuals":[{"id":"chatgpt-agent-loop-efficiency-stack","kind":"editorial-diagram","role":"hero","src":"https://newruntime.com/images/posts/chatgpt-agent-loop-efficiency-stack.webp","alt":"Hand-drawn three-layer agent stack where the harness preserves reusable context, the API processes only new work, and inference routes requests back to cached state before separating prompt processing from token generation.","caption":"The efficiency pattern is consistent across harness, API, and inference: preserve reusable state and avoid paying for the same work twice.","credit":"New Runtime synthesis from ByteByteGo's OpenAI engineering walkthrough","source_url":"https://blog.bytebytego.com/p/how-chatgpt-optimizes-its-agent-loop","generated_with":"gemini-3.1-flash-image","width":1600,"height":900,"legend":[{"label":"Harness","description":"Persistent connections, stable prefixes, deferred tools, and code execution keep repeated context small."},{"label":"API","description":"Only new conversation items are tokenized while safety checks overlap unavoidable inference time."},{"label":"Inference","description":"Cache-aware routing, KV management, speculative decoding, and split prefill/decode reduce repeated GPU work."}]}],"routes":{"html":"https://newruntime.com/posts/chatgpt-agent-loop-efficiency-stack/","markdown":"https://newruntime.com/posts/chatgpt-agent-loop-efficiency-stack.md","json":"https://newruntime.com/posts/chatgpt-agent-loop-efficiency-stack.json"},"source_format":"markdown","next_reads":[{"type":"topic","path":"/topics/context-engineering/","reason":"Explore the context engineering topic hub.","url":"https://newruntime.com/topics/context-engineering/","title":"Context engineering - New Runtime","media_type":"text/html"},{"type":"topic","path":"/topics/inference/","reason":"Explore the inference topic hub.","url":"https://newruntime.com/topics/inference/","title":"Inference - New Runtime","media_type":"text/html"},{"type":"related_material","path":"/posts/openai-gpt-5-6-efficiency-stack/","reason":"Shares agent harnesses and context engineering.","url":"https://newruntime.com/posts/openai-gpt-5-6-efficiency-stack/","title":"OpenAI Shows Efficiency Is a Full-Stack Agent Problem","media_type":"text/html"},{"type":"related_material","path":"/posts/anthropic-tool-search-programmatic-calls/","reason":"Shares agent harnesses and context engineering.","url":"https://newruntime.com/posts/anthropic-tool-search-programmatic-calls/","title":"Anthropic Moves Large Tool Libraries Out Of Context","media_type":"text/html"},{"type":"related_material","path":"/posts/genkit-agent-skills-progressive-disclosure/","reason":"Shares agent harnesses and context engineering.","url":"https://newruntime.com/posts/genkit-agent-skills-progressive-disclosure/","title":"Genkit Adds Progressive Disclosure For Agent Skills","media_type":"text/html"}]}