What happened
In a post published November 4, 2026, Together AI introduced Together Compiler as the first component of its Polymorphic Inference initiative. The company frames the problem as a growing number of combinations among models, hardware, customers and traffic patterns. It says the top ten open-source LLMs on OpenRouter do not share one architecture, while its own service handles many trillions of tokens per day across thousands of customers. Its listed hardware spans H100, B200, B300, NVL72, VR and Groq systems.
TKCC takes a traffic shape, model and hardware target as inputs, then uses agents to generate and evaluate a sequence of inference-engine optimizations. Rather than asking agents to restart engines such as vLLM, SGLang or TRT-LLM for each experiment, Together uses persistent engines that start once and hot-reload candidate forward passes. Multiple agents continually generate proposals and coordinate through a message board. The company’s table compares 467 seconds from process initialization through API-server readiness in the status quo with 9 seconds for its compiler setup and CUDA-graph capture. That is a comparison of the phases shown in Together’s table, not a general benchmark of all inference deployments.
The system also tries to balance quick measurement with realistic evaluation. Together describes a tier of fast engines with few layers for rapid timing and slower engines with more layers to check candidates against real data distributions. This matters in its account because modern mixture-of-experts and sparse-attention architectures can behave differently depending on input data. The post also describes a cost model for selecting among tensor-, expert- and data-context-parallelism strategies. The supplied material does not state a purchase price, standalone access terms or a public release mechanism for TKCC.
Together reports that in less than 24 hours TKCC achieved up to 2.6× higher tokens per second across Kimi K3, GLM 5.3, MiniMax M3.1-Flash and DeepSeek v4.1-Flash. It reports 1,111 TPS on MiniMax M3.1-Flash and 1,205 TPS on DeepSeek v4.1-Flash on SemiAnalysis’ InferenceX low-latency benchmark, and 1.7× the performance of available open-source megakernels on its Llama 1B batch-size-one setting. The company says TKCC generalizes beyond language inference to weather modeling, PDEs, time series and biology, and is already serving trillions of tokens.
What the archive adds
AgentForge research preview: open-source Rust compiler and runtime that plans inference for whole agent programs covered a separate research-preview system for planning inference across agent programs. This is a new product event from Together AI, not the same compiler: TKCC optimizes inference engines continuously against workloads.
Trend signal
This supports Harness architecture outlives model choice: performance depends on the surrounding system that runs and measures models, not only on the model identity. The evidence here is narrower than the pattern’s full thesis, since TKCC concerns inference optimization rather than an agent harness as a whole.
Why now
Together says traffic changes with regional preferences, model releases and burstiness. It reports that prompt lengths can rise 80% two weeks after a new model release as users move from chat to agentic workloads, and that bursts can double prompt lengths and lower cache hit rates. Those shifts are the stated reason to keep retuning rather than rely on fixed optimizations.
Connections to other events
An earlier New Runtime Telegram post described inference serving for coding-agent workloads with contexts that accumulate during sessions. Together’s announcement addresses a related operational pressure from a different angle: optimizing the engine as traffic shape changes, rather than focusing on context management alone.
Why it matters
Inference performance is often treated as a one-time kernel or deployment choice. TKCC’s design treats it as a recurring operations loop: observe the workload, propose changes, measure them with realistic data, and keep the engine available while experiments proceed. If the reported speedups hold in production, that could help hosted-model providers improve throughput or latency without changing the model itself.
For teams building on an inference provider, the practical implication is to ask how performance claims map to their own traffic. Batch size, sequence length, expert imbalance and session duration can all affect which optimization wins. A provider’s aggregate throughput result does not establish the same gain for every customer, but continuous tuning makes workload-specific performance a more explicit part of the service boundary. Earlier coverage: Together Compiler: Continuous Compilation for AI inference.
Limits and uncertainty
The performance results and deployment scale in the post are Together’s own claims. The supplied material does not provide independent replication, complete benchmark configurations, error bars, or enough detail to compare every result with equivalent hardware and operating conditions. “Up to” 2.6× is a maximum across selected models, not a guaranteed gain. The post also does not establish what customers can configure, whether TKCC is separately available, how the system’s costs compare, or how often production optimizations change. Generalization to non-language domains is asserted without detailed results in the supplied excerpt.
Skill to build now
For a serving team, record a small workload profile before evaluating an inference optimization: model and hardware, prompt and output lengths, batch size, concurrency, and latency or throughput target. Compare candidate changes against representative traffic, not random inputs alone. That gives you a baseline to determine whether a provider’s or internal compiler’s reported gains transfer to your deployment.
Reader takeaway
Together’s central claim is architectural: persistent engines and parallel agents can make inference optimization continuous instead of episodic. The headline throughput numbers are worth testing, not assuming. For builders, the durable lesson is to benchmark against the shape of the traffic they actually serve.