What happened
In a research post published November 4, 2026, Together AI introduces Together Compiler, which it calls the first “truly continual optimization loop” for diverse inference workloads. The system takes traffic shape, model, and hardware as inputs, then uses agents to generate candidate optimizations for an inference engine. The motivation is variability: Together says it serves many trillions of tokens daily across thousands of customers, and that traffic differs in batch size, sequence length, expert imbalance for mixture-of-experts models, attention sparsity, session duration, and domain. The post also notes that hardware targets range from H100 and B200 to B300, NVL72, VR, and Groq.
TKCC’s central design choice is persistent engines. In the approach Together contrasts with, agents repeatedly launch and tear down engines such as vLLM, SGLang, or TensorRT-LLM to test each candidate. The company says those starts can take tens of minutes. TKCC instead starts an engine once and hot-reloads candidate forward passes without restarting. Multiple agents continually propose optimizations and communicate through a message board. In the post’s startup-time table, process and worker initialization, weight loading, KV-cache sizing, JIT warm-up, autotuning, and API readiness add up to 467 seconds for the status quo; Together reports 9 seconds for its setup. CUDA-graph capture is listed as 7 seconds versus 7.3 seconds, while the Together Compiler RPC call and setup is listed at 1.7 seconds.
The system also tries to balance experiment speed against realistic measurement. Fast engines load a few layers for rapid iteration and low-variance timings; slower engines load more layers to test whether candidate optimizations still work with real data distributions. This distinction matters for data-dependent operations such as MoE routing and sparse attention: the post argues that random inputs or isolated kernel tests can mislead when actual input distributions change which strategy performs best. A custom cost model narrows the search for parallelism configurations using hardware specifications, microbenchmarks, and measured forward passes.
Together reports results on three kinds of comparisons: hand-tuned megakernels, InferenceX engine configurations, and AI codebases in domains including weather modeling, partial differential equations, time series, and biology. For Llama 1B at batch size one, it reports 1.7× higher tokens per second than megakernel and CUDA-graph baselines, with tests on H100 and B200. On SemiAnalysis’s AgentX workloads, it says TKCC outperforms the selected open-source configurations by up to 2.6×. For low-latency batch-one tests, it reports 1,111 TPS on MiniMax M3.1-Flash and 1,205 TPS on DeepSeek v4.1-Flash, using its cost model to select parallelism rather than the InferenceX configuration. The post says the system is already serving trillions of tokens, but gives no separate deployment, access, or pricing details.
What the archive adds
This is a new fact on a known systems-research thread. AgentForge’s research preview also concerns planning inference across whole agent programs, but TKCC is a distinct system focused on continual, traffic-specific optimization. The Together GPU Clusters auto-repair note shares the company, not the subject.
Trend signal
TKCC supports harness architecture outlives model choice: performance depends on a durable system around model execution, including measurement, orchestration, and adaptation. This is a relevant infrastructure signal, though Together’s own benchmark claims do not establish that every production team needs a continuously optimizing compiler.
Why now
Together points to a widening and changing search space: model architectures, accelerators, customer workloads, and traffic shapes vary, while new model adoption can change prompt lengths quickly. It says prompt length increased 80% two weeks after recent model releases as users shifted toward agentic workloads. That churn gives a reason to revisit fixed, manually tuned configurations.
Why it matters
Inference teams often optimize for a fixed model-and-hardware pairing, then discover that real traffic changes the winning configuration. TKCC’s design makes optimization throughput part of the systems problem: if each experiment costs minutes of engine startup, agents spend less time testing alternatives; if tests use unrealistic inputs, fast results may not transfer to production. The persistent-engine and multi-fidelity approach offers one proposed answer to both constraints.
The results are most relevant to providers operating many models and heterogeneous accelerators, where small per-request improvements can compound across large token volumes. They are less directly actionable for application teams using hosted APIs, which cannot control serving internals. The broader takeaway for builders is to treat benchmark conditions, input distributions, concurrency, and latency targets as part of any performance claim—not to assume a headline TPS multiplier will transfer unchanged to another workload. Later coverage: Together AI launches Together Compiler, an agent-driven continual optimization loop for inference engines.
Limits and uncertainty
The post is Together’s account of its own system and experiments. The supplied material does not provide independent replication, full benchmark configurations, error bars, or a production cost comparison. “Up to” results describe selected cases, not a typical gain across all models or traffic. The excerpt does not establish whether TKCC is available to third parties, what it costs, or how much engineering effort deployment requires. It also does not quantify the overhead of the agents and measurement loop, or show that the reported speedups persist across every workload served by Together.
Reader takeaway
Together Compiler is a signal that inference optimization may move from occasional hand-tuning toward a continual loop that adapts to workload drift. The reported gains are striking but vendor-reported and workload-specific. Teams should read the methodology and configuration before using the numbers to forecast their own throughput or cost.
Open question
Can independent teams reproduce the reported gains on their own traffic and hardware without Together’s serving infrastructure?