---
type: "post"
id: "agentforge-research-preview-open-source-rust-compiler-and-40ab5b"
slug: "agentforge-research-preview-open-source-rust-compiler-and-40ab5b"
title: "AgentForge research preview: open-source Rust compiler and runtime that plans inference for whole agent programs"
description: "AgentForge is an Apache-2.0 Rust research project that compiles an agent program together with a model and a machine description."
retrieval_nugget: "AgentForge is an Apache-2.0 Rust research project that compiles an agent program together with a model and a machine description. Its premise is that long-context agents reveal enough about future work to plan KV-cache placement before requests arrive. The first results are simulation results, not production hardware measurements."
published_at: "2026-10-30"
updated_at: "2026-10-30"
record_date: "2026-10-30"
date_kind: "published_at"
topics: ["harness-engineering","agent-runtime"]
entities: ["AgentForge","TraceLab","Llama 3.1 70B","H200 SXM"]
related_patterns: ["harness-architecture-outlives-model-choice"]
novelty_label: "new_fact"
what_changed: "AgentForge is a distinct Rust compiler/runtime research preview that plans inference across whole agent programs, including KV residency and prefill/decode placement; priors describe Infinity’s suspendable agent runtime."
prior_refs: ["memory:74feca42-5f11-8e40-a0eb-6e0154d4e7ee","memory:5081d9eb-113f-8452-afd5-a75581f97b63"]
source_url: "https://warpy-ai.github.io/agentforge"
source_urls: ["https://warpy-ai.github.io/agentforge"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:agentforge-research-preview-open-source-rust-compiler-and-40ab5b"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/agentforge-research-preview-open-source-rust-compiler-and-40ab5b/","markdown":"https://newruntime.com/posts/agentforge-research-preview-open-source-rust-compiler-and-40ab5b.md","json":"https://newruntime.com/posts/agentforge-research-preview-open-source-rust-compiler-and-40ab5b.json"}
---

# AgentForge research preview: open-source Rust compiler and runtime that plans inference for whole agent programs

## Retrieval answer

AgentForge is an Apache-2.0 Rust research project that compiles an agent program together with a model and a machine description. Its premise is that long-context agents reveal enough about future work to plan KV-cache placement before requests arrive. The first results are simulation results, not production hardware measurements.

## Summary

AgentForge is an Apache-2.0 Rust research project that compiles an agent program together with a model and a machine description. Its premise is that long-context agents reveal enough about future work to plan KV-cache placement before requests arrive. The first results are simulation results, not production hardware measurements.

## What happened

AgentForge has published its M0 baseline, dated 2026-10, as an open-source compiler and runtime project for AI inference silicon. The project is written in Rust, comprises seven crates, and has an Apache-2.0 licence. Its current working implementation is a trace-replay simulator; the project explicitly says that it has no hardware backend yet. M0 is the first of six planned milestones.

The central claim is that an inference system should compile two inputs together. The first is an Agent Program IR: an agent workflow expressed as LLM calls, prompt blocks marked stable or unstable, tool-call transitions with latency models, probable branches, and endings. The second is a TOML machine model describing devices, compute by numeric precision, memory capacities and bandwidths, memory tiers, and links. From those inputs, AgentForge intends to decide KV-cache residency, prefetches, prefill/decode placement, and kernel choices before the first token is served.

This is aimed at agentic workloads where a large prompt prefix persists across a short generation and a potentially long tool wait. The project cites a TraceLab reference workload from June 2026 in which a median coding-agent step carries 119,000 prefix tokens, appends 875 tokens, and generates 214. It reports prefix-cache hit rates of 95–96%. A miss after a tool wait can force expensive recomputation: the project estimates that a 119,000-token prefix for a 70B grouped-query model occupies about 38 GB in FP16 KV, while recomputation can take 6 to 30 seconds.

Its architecture spans agent, model, and tile IRs; a machine model; cost and simulation layers; compiler passes; planned backends; and a runtime. The baseline includes KV-liveness analysis, a residency planner, and prefill/decode placement. It describes planned output backends including PTX, CUDA Tile IR, LLVM AMDGPU, SPIR-V, MSL, tt-metal, and Cranelift, while stressing that the baseline has no MLIR dependency. The runtime is intended to execute the plan, manage KV blocks with plan-derived lifetimes, and transport data across tiers and devices.

The initial comparison is limited to synthetic traces. On 80 synthetic coding-agent sessions, with 1,719 steps and an eight-H200-SXM setup with a 945 GB KV budget and 2 TB host tier, a program-aware policy was compared with reactive LRU. In the reported seed, p90 time to first token fell from 33.2 to 23.0 seconds, p90 step latency from 36.4 to 27.8 seconds, foreground KV wait from 20.4 to 16.5 seconds, and pressure evictions from 188 to 114. The project also reports limits: mean TTFT barely moved, p99 metrics did not improve, and on a scarce, slow host tier the extra transfers made the planning policy lose to LRU.

## What the archive adds

This is a new fact on a known runtime direction. [Infinity](https://newruntime.com/posts/aws-and-uc-berkeley-open-source-infinity-a-5436d7/) addressed runtime scale by suspending agents at tool calls and fitting many waiting agents into memory. AgentForge addresses a different layer: using predicted returns from those waits to decide where an individual agent's KV state should reside.

## Trend signal

The project supports [Harness architecture outlives model choice](https://newruntime.com/patterns/harness-architecture-outlives-model-choice/). AgentForge treats the agent program, memory tiers, transfer links, cache policy, and placement logic as a compilation problem around the model. The evidence is preliminary, but the proposed boundary is clearly broader than model selection alone.

## Why it matters

For teams building long-context coding or tool-using agents, cache policy may become part of application architecture rather than a serving-layer implementation detail. AgentForge makes a concrete version of that argument: tool identity and expected wait time can determine whether retaining, spilling, or prefetching a prefix is cheaper than recomputing it.

That does not make this a deployable serving stack. It does clarify a design question teams can ask of their own infrastructure: does the scheduler know the workflow's next likely transition, or does it only see independent model requests? If the latter, it may discard information that matters for tail latency and transfer contention. The project also shows why lower cache-miss cost is not synonymous with universally lower latency: memory admission, active-session KV demand, and link bandwidth remain separate constraints.

## Limits and uncertainty

The reported benefits come from a simulator replaying synthetic coding-agent sessions, not from a hardware backend or a production deployment. The main comparison uses one stated configuration and a modeled policy with per-tool latency information; it does not establish that real agent traces can be modeled accurately enough to produce the same gains.

The source itself reports mixed outcomes across seeds and no meaningful improvement in mean TTFT in pressured regimes. It also says the approach loses when host storage is scarce and slow because additional transfers saturate a channel. There is no published evidence here for correctness, operational reliability, backend compatibility, or savings in a live multi-tenant serving environment.

## Skill to build now

Instrument one agent workflow as a state machine this week. Record stable prompt-prefix size, generated tokens, tool called, tool-wait duration, cache reuse, and the tier from which context was restored. Then segment tail latency by the preceding tool transition rather than only by model endpoint. This creates the evidence needed to tell whether a program-aware cache policy could help, and whether bandwidth rather than cache eviction is the binding constraint.

## Reader takeaway

AgentForge is an early attempt to make inference scheduling aware of an agent's workflow. Its simulator suggests that knowing which tool an agent used, and when it is likely to return, can reduce some KV-cache pressure and p90 latency. It is a research direction to evaluate, not yet a runtime to adopt.

## Open question

Can real agent programs provide sufficiently reliable transition and tool-latency estimates for program-aware placement to outperform reactive policies outside a simulator?

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Harness Engineering](https://newruntime.com/topics/harness-engineering.md): Explore the harness-engineering topic hub.
- topic hub: [Agent Runtime](https://newruntime.com/topics/agent-runtime.md): Explore the agent-runtime topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)