What happened
AgentForge has published its M0 baseline, dated 2026-10, as an open-source compiler and runtime project for AI inference silicon. The project is written in Rust, comprises seven crates, and has an Apache-2.0 licence. Its current working implementation is a trace-replay simulator; the project explicitly says that it has no hardware backend yet. M0 is the first of six planned milestones.
The central claim is that an inference system should compile two inputs together. The first is an Agent Program IR: an agent workflow expressed as LLM calls, prompt blocks marked stable or unstable, tool-call transitions with latency models, probable branches, and endings. The second is a TOML machine model describing devices, compute by numeric precision, memory capacities and bandwidths, memory tiers, and links. From those inputs, AgentForge intends to decide KV-cache residency, prefetches, prefill/decode placement, and kernel choices before the first token is served.
This is aimed at agentic workloads where a large prompt prefix persists across a short generation and a potentially long tool wait. The project cites a TraceLab reference workload from June 2026 in which a median coding-agent step carries 119,000 prefix tokens, appends 875 tokens, and generates 214. It reports prefix-cache hit rates of 95–96%. A miss after a tool wait can force expensive recomputation: the project estimates that a 119,000-token prefix for a 70B grouped-query model occupies about 38 GB in FP16 KV, while recomputation can take 6 to 30 seconds.
Its architecture spans agent, model, and tile IRs; a machine model; cost and simulation layers; compiler passes; planned backends; and a runtime. The baseline includes KV-liveness analysis, a residency planner, and prefill/decode placement. It describes planned output backends including PTX, CUDA Tile IR, LLVM AMDGPU, SPIR-V, MSL, tt-metal, and Cranelift, while stressing that the baseline has no MLIR dependency. The runtime is intended to execute the plan, manage KV blocks with plan-derived lifetimes, and transport data across tiers and devices.
The initial comparison is limited to synthetic traces. On 80 synthetic coding-agent sessions, with 1,719 steps and an eight-H200-SXM setup with a 945 GB KV budget and 2 TB host tier, a program-aware policy was compared with reactive LRU. In the reported seed, p90 time to first token fell from 33.2 to 23.0 seconds, p90 step latency from 36.4 to 27.8 seconds, foreground KV wait from 20.4 to 16.5 seconds, and pressure evictions from 188 to 114. The project also reports limits: mean TTFT barely moved, p99 metrics did not improve, and on a scarce, slow host tier the extra transfers made the planning policy lose to LRU.
What the archive adds
This is a new fact on a known runtime direction. Infinity addressed runtime scale by suspending agents at tool calls and fitting many waiting agents into memory. AgentForge addresses a different layer: using predicted returns from those waits to decide where an individual agent's KV state should reside.
Trend signal
The project supports Harness architecture outlives model choice. AgentForge treats the agent program, memory tiers, transfer links, cache policy, and placement logic as a compilation problem around the model. The evidence is preliminary, but the proposed boundary is clearly broader than model selection alone.
Why it matters
For teams building long-context coding or tool-using agents, cache policy may become part of application architecture rather than a serving-layer implementation detail. AgentForge makes a concrete version of that argument: tool identity and expected wait time can determine whether retaining, spilling, or prefetching a prefix is cheaper than recomputing it.
That does not make this a deployable serving stack. It does clarify a design question teams can ask of their own infrastructure: does the scheduler know the workflow's next likely transition, or does it only see independent model requests? If the latter, it may discard information that matters for tail latency and transfer contention. The project also shows why lower cache-miss cost is not synonymous with universally lower latency: memory admission, active-session KV demand, and link bandwidth remain separate constraints.
Limits and uncertainty
The reported benefits come from a simulator replaying synthetic coding-agent sessions, not from a hardware backend or a production deployment. The main comparison uses one stated configuration and a modeled policy with per-tool latency information; it does not establish that real agent traces can be modeled accurately enough to produce the same gains.
The source itself reports mixed outcomes across seeds and no meaningful improvement in mean TTFT in pressured regimes. It also says the approach loses when host storage is scarce and slow because additional transfers saturate a channel. There is no published evidence here for correctness, operational reliability, backend compatibility, or savings in a live multi-tenant serving environment.
Skill to build now
Instrument one agent workflow as a state machine this week. Record stable prompt-prefix size, generated tokens, tool called, tool-wait duration, cache reuse, and the tier from which context was restored. Then segment tail latency by the preceding tool transition rather than only by model endpoint. This creates the evidence needed to tell whether a program-aware cache policy could help, and whether bandwidth rather than cache eviction is the binding constraint.
Reader takeaway
AgentForge is an early attempt to make inference scheduling aware of an agent's workflow. Its simulator suggests that knowing which tool an agent used, and when it is likely to return, can reduce some KV-cache pressure and p90 latency. It is a research direction to evaluate, not yet a runtime to adopt.
Open question
Can real agent programs provide sufficiently reliable transition and tool-latency estimates for program-aware placement to outperform reactive policies outside a simulator?