---
schema_version: "newruntime-agent-readable-v0.2"
type: "post"
stable_id: "post:chatgpt-agent-loop-efficiency-stack"
slug: "chatgpt-agent-loop-efficiency-stack"
title: "ChatGPT Cuts Repeated Work Across The Agent Stack"
description: "ByteByteGo's OpenAI engineering walkthrough connects persistent sessions, stable prompt prefixes, deferred tools, delta tokenization, cache-aware routing, and split inference."
retrieval_nugget: "ByteByteGo's OpenAI engineering walkthrough connects persistent sessions, stable prompt prefixes, deferred tools, delta tokenization, cache-aware routing, and split inference. ByteByteGo's walkthrough with OpenAI engineers is useful because it traces agent efficiency through three layers instead of attributing it to one faster model. The recurring objective is to avoid paying for the same work twice."
status: "published"
published_at: "2026-08-03"
updated_at: "2026-08-03"
record_date: "2026-08-03"
date_kind: "published_at"
topics: ["agent-harnesses","inference","performance","context-engineering"]
source_urls: ["https://blog.bytebytego.com/p/how-chatgpt-optimizes-its-agent-loop"]
visuals: [{"id":"chatgpt-agent-loop-efficiency-stack","kind":"editorial-diagram","role":"hero","src":"https://newruntime.com/images/posts/chatgpt-agent-loop-efficiency-stack.webp","alt":"Hand-drawn three-layer agent stack where the harness preserves reusable context, the API processes only new work, and inference routes requests back to cached state before separating prompt processing from token generation.","caption":"The efficiency pattern is consistent across harness, API, and inference: preserve reusable state and avoid paying for the same work twice.","credit":"New Runtime synthesis from ByteByteGo's OpenAI engineering walkthrough","source_url":"https://blog.bytebytego.com/p/how-chatgpt-optimizes-its-agent-loop","generated_with":"gemini-3.1-flash-image","width":1600,"height":900,"legend":[{"label":"Harness","description":"Persistent connections, stable prefixes, deferred tools, and code execution keep repeated context small."},{"label":"API","description":"Only new conversation items are tokenized while safety checks overlap unavoidable inference time."},{"label":"Inference","description":"Cache-aware routing, KV management, speculative decoding, and split prefill/decode reduce repeated GPU work."}]}]
routes: {"html":"https://newruntime.com/posts/chatgpt-agent-loop-efficiency-stack/","markdown":"https://newruntime.com/posts/chatgpt-agent-loop-efficiency-stack.md","json":"https://newruntime.com/posts/chatgpt-agent-loop-efficiency-stack.json"}
source_format: "markdown"
---

# ChatGPT Cuts Repeated Work Across The Agent Stack

## Retrieval answer

ByteByteGo's OpenAI engineering walkthrough connects persistent sessions, stable prompt prefixes, deferred tools, delta tokenization, cache-aware routing, and split inference. ByteByteGo's walkthrough with OpenAI engineers is useful because it traces agent efficiency through three layers instead of attributing it to one faster model. The recurring objective is to avoid paying for the same work twice.

ByteByteGo's walkthrough with OpenAI engineers is useful because it traces agent efficiency through three layers instead of attributing it to one faster model. The recurring objective is to avoid paying for the same work twice.

At the harness layer, a persistent WebSocket avoids connection setup on every turn. Stable prompt prefixes preserve cacheability. Deferred tool discovery keeps hundreds of unused schemas out of the prompt. Code Mode lets the model compose several tool calls in a small program, process intermediate results outside the model context, and return only the compact result.

At the API layer, the server can retain the tokenized conversation and process only new items on later turns. Safety classifiers run in parallel with inference so their latency overlaps time that would already pass before the first token, subject to the service's release policy.

The inference layer continues the same pattern. Cache-aware routing balances available capacity against the value of sending a conversation back to the machine that already holds its state. KV-cache management protects expensive context. Speculative decoding lets a smaller model propose tokens for parallel verification. Separating compute-heavy prefill from memory-heavy decode allows each phase to run on a fleet configured for its bottleneck.

These claims come from ByteByteGo's engineering account rather than an OpenAI product specification, so the individual mechanisms should be treated as an architecture map, not a public service guarantee.

For New Runtime, the actionable unit is the whole path. A cached prefix has little value if routing loses the KV state; a stateless client cannot benefit from server-side delta tokenization; deferred tools matter only if the runtime can retrieve the right definition on demand. Efficiency compounds when the layers preserve the same reusable boundaries.