ByteByteGo’s walkthrough with OpenAI engineers is useful because it traces agent efficiency through three layers instead of attributing it to one faster model. The recurring objective is to avoid paying for the same work twice.
At the harness layer, a persistent WebSocket avoids connection setup on every turn. Stable prompt prefixes preserve cacheability. Deferred tool discovery keeps hundreds of unused schemas out of the prompt. Code Mode lets the model compose several tool calls in a small program, process intermediate results outside the model context, and return only the compact result.
At the API layer, the server can retain the tokenized conversation and process only new items on later turns. Safety classifiers run in parallel with inference so their latency overlaps time that would already pass before the first token, subject to the service’s release policy.
The inference layer continues the same pattern. Cache-aware routing balances available capacity against the value of sending a conversation back to the machine that already holds its state. KV-cache management protects expensive context. Speculative decoding lets a smaller model propose tokens for parallel verification. Separating compute-heavy prefill from memory-heavy decode allows each phase to run on a fleet configured for its bottleneck.
These claims come from ByteByteGo’s engineering account rather than an OpenAI product specification, so the individual mechanisms should be treated as an architecture map, not a public service guarantee.
For New Runtime, the actionable unit is the whole path. A cached prefix has little value if routing loses the KV state; a stateless client cannot benefit from server-side delta tokenization; deferred tools matter only if the runtime can retrieve the right definition on demand. Efficiency compounds when the layers preserve the same reusable boundaries.