---
type: "post"
id: "runanywhere-open-sources-trnrt-for-serving-glm-5-39e01e"
slug: "runanywhere-open-sources-trnrt-for-serving-glm-5-39e01e"
title: "RunAnywhere open-sources TrnRT for serving GLM-5.3-Flash on AWS Trainium2"
description: "RunAnywhere has open-sourced TrnRT, an inference engine for AWS Trainium2, and reports serving GLM-5.3-Flash on a single trn2.48xlarge."
retrieval_nugget: "RunAnywhere has open-sourced TrnRT, an inference engine for AWS Trainium2, and reports serving GLM-5.3-Flash on a single trn2.48xlarge. Its AgentX results offer a workload-specific view of large-model serving for coding agents, but are company-run measurements without a same-trace GPU comparison."
published_at: "2026-11-05"
updated_at: "2026-11-05"
record_date: "2026-11-05"
date_kind: "published_at"
topics: ["inference"]
entities: ["RunAnywhere","TrnRT","GLM-5.3-Flash","AWS Trainium2","Amazon Web Services","SemiAnalysis AgentX"]
related_patterns: ["harness-architecture-outlives-model-choice"]
novelty_label: "new_fact"
what_changed: "RunAnywhere adds a separate Trainium2 serving engine, TrnRT, for GLM-5.3-Flash, with AgentX benchmark results and NKI layer megakernels."
prior_refs: []
source_url: "https://runanywhere.ai/blog/glm-5-3-flash-trainium2-nki-megakernels"
source_urls: ["https://runanywhere.ai/blog/glm-5-3-flash-trainium2-nki-megakernels"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:runanywhere-open-sources-trnrt-for-serving-glm-5-39e01e"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/runanywhere-open-sources-trnrt-for-serving-glm-5-39e01e/","markdown":"https://newruntime.com/posts/runanywhere-open-sources-trnrt-for-serving-glm-5-39e01e.md","json":"https://newruntime.com/posts/runanywhere-open-sources-trnrt-for-serving-glm-5-39e01e.json"}
markdown_url: "https://newruntime.com/posts/runanywhere-open-sources-trnrt-for-serving-glm-5-39e01e.md"
---

# RunAnywhere open-sources TrnRT for serving GLM-5.3-Flash on AWS Trainium2

## Retrieval answer

RunAnywhere has open-sourced TrnRT, an inference engine for AWS Trainium2, and reports serving GLM-5.3-Flash on a single trn2.48xlarge. Its AgentX results offer a workload-specific view of large-model serving for coding agents, but are company-run measurements without a same-trace GPU comparison.

## Summary
RunAnywhere has open-sourced TrnRT, an inference engine for AWS Trainium2, and reports serving GLM-5.3-Flash on a single trn2.48xlarge. Its AgentX results offer a workload-specific view of large-model serving for coding agents, but are company-run measurements without a same-trace GPU comparison.

## What happened
RunAnywhere published a technical post on November 4, 2026, describing TrnRT, its inference engine for AWS Trainium2. The accompanying code is available under the Apache-2.0 license at [the TrnRT repository](https://github.com/RunanywhereAI/trnrt). The post focuses on serving GLM-5.3-Flash, a 320-billion-parameter mixture-of-experts model with a one-million-token context, on one AWS trn2.48xlarge instance. The company says the model activates about 18 billion parameters per token.

The reported workload test is SemiAnalysis’s AgentX benchmark, which RunAnywhere describes as replaying coding-agent sessions with long, mostly cached prompts, subagents, and users waiting for answers. The company says one box served up to 28.9 agent requests per minute with 32 agents and batch size 12, producing 3.57 million tokens per minute. At that operating point, the reported cost is $0.021 per request at AWS’s capacity-block price. At 16 agents, RunAnywhere reports 19.1 requests per minute and a median 1.24 seconds before a request with a cached prompt starts decoding; decoding runs at 35.9 tokens per second.

For a prompt of 1,039,912 tokens, one instance using half the box took 190.8 seconds to produce its first token. RunAnywhere compares this with 1,651.7 seconds published for Kiln v0.2.0 in the same half-box setup, calling its result 8.7 times faster. A follow-up question against a cached million-token context reached its first token in a median 1.84 seconds across three runs. These are the vendor’s measurements, not an independent benchmark result.

The implementation combines NKI layer megakernels with prefix caching. RunAnywhere says it fused each layer half into one NKI kernel, or a whole layer for sparse-attention layers. On its stated eight-request-per-instance decode test, the change reduced a decode step from 32.2 milliseconds in its October 17 build to 24.2 milliseconds on November 1, using the same 32-way tensor-parallel layout. The engine also maintains each agent’s context in device and host memory. The company says every kernel is guarded by a byte-for-byte or high-precision reference gate; one approximation remains in sparse-attention prompt processing and affects 1.2% of selections.

The hardware setup has 16 Trainium2 chips and 1.5 TB of memory. RunAnywhere runs two serving instances per box, each tensor-parallel across 32 logical cores on eight chips, with context parallelism for sparse-attention layers. It reports that at 32 agents requests queue for a decode slot, while up to 16 agents the median request starts in about a second. The post notes that roughly 95% of input tokens in throughput figures come from the prefix cache, and that the benchmarked build has not received a task-level accuracy evaluation.

## What the archive adds
This is a new serving implementation for GLM-5.3-Flash, not a restatement of the earlier model-access stories. The archive has related inference coverage, including [Liquid AI’s RIPL inference engine](https://newruntime.com/posts/liquid-ai-launched-ripl-a-sub-50ms-inference-10eb44/) and [DeepSeek’s DeepInfer framework](https://newruntime.com/posts/deepseek-open-sources-deepinfer-the-inference-framework-behind-893234/). Those stories provide context on inference-engine releases; they do not establish comparable performance for Trainium2 or this workload.

## Trend signal
This release weakly supports [harness architecture outlives model choice](https://newruntime.com/patterns/harness-architecture-outlives-model-choice/): RunAnywhere’s report attributes its serving result to kernel fusion, context caching, and workload-specific scheduling around a model. It does not show that this design transfers to other models or hardware, or that it outperforms other production stacks.

## Why it matters
Teams considering Trainium2 need more than peak-token numbers: long-context agent workloads repeatedly revisit context, and the balance between prompt processing, caching, and decoding can determine capacity and cost. RunAnywhere’s measurements make that workload shape explicit and offer a starting point for testing whether Trainium2’s memory capacity and TrnRT’s caching strategy suit a team’s own traffic.

The figures are not a general price or performance verdict. The reported $0.021 per request applies to one stated AgentX configuration and an AWS capacity-block price. Without running the same traces on GPUs, other serving engines, or a team’s own prompts, builders cannot infer a hardware winner or predict production economics. The repository makes independent evaluation possible, but teams should treat the blog as a benchmark claim to reproduce, not a procurement conclusion.

## Limits and uncertainty
The AgentX numbers are self-run by RunAnywhere, with one run per point, and the company says they are not official InferenceX results. It has not run GPU baselines on the same traces, so the post cannot establish Trainium2’s relative cost or performance against GPUs. The reported throughput is heavily shaped by cached prompt input; it should not be read as uncached throughput. The post also says it did not conduct a task-level accuracy evaluation on the benchmarked build. Its kernel accuracy gates and the stated sparse-attention approximation do not by themselves establish end-to-end coding-agent quality.

## Skill to build now
If Trainium2 is on your shortlist, reproduce one representative agent session before comparing instance prices. Separate cached prompt input, uncached input, and generated output; record time to first token and decode rate at your expected concurrency. This mirrors the source’s main caveat: throughput that depends on prefix caching may not predict performance for workloads with different context reuse.

## Reader takeaway
TrnRT gives builders an open implementation to evaluate for large-model serving on Trainium2, with useful measurements on a coding-agent-shaped workload. The results are specific to RunAnywhere’s setup and cache-heavy traffic. Re-run the workload that matters to your team, and do not treat the reported figures as a GPU comparison.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Inference](https://newruntime.com/topics/inference.md): Explore the inference topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)