{"slug": "show-hn-trnrt-an-open-source-llm-inference-engine-for-aws-trainium", "title": "Show HN: TrnRT – An open-source LLM inference engine for AWS Trainium", "summary": "RunanywhereAI released TrnRT, an open-source inference engine that serves an OpenAI-compatible API for large language models from a single AWS trn2.48xlarge instance, with GLM-5.3-Flash as its first supported model. TrnRT runs decode as NKI megakernels (about one fused kernel per layer half), supports a 1M-token context via sparse attention with top 2,048 selection plus linear attention, and keeps a radix-tree prefix cache over FP8 KV pages with a host-memory tier for evicted pages. Each box runs two engines, each tensor-parallel over 32 logical NeuronCores behind a session-affine router, and all graphs are compiled offline on any x86_64 host using Neuron SDK 2.32 (neuronx-cc 2.27) and Python 3.13, so the serving box compiles nothing by default.", "body_md": "[**Quickstart**](#quickstart) ·\n  [**Benchmarks**](https://github.com/RunanywhereAI/trnrt/blob/development/docs/RESULTS.md) ·\n  [**Docs**](https://github.com/RunanywhereAI/trnrt/blob/development/docs/README.md) ·\n  [**Kernels**](https://github.com/RunanywhereAI/trnrt/blob/development/docs/MEGAKERNELS.md) ·\n  [**Blog post**](https://runanywhere.ai/blog/glm-5-3-flash-trainium2-nki-megakernels) ·\n  **runanywhere.ai**\n\n**TrnRT** is an open inference engine for large language models on AWS Trainium. It serves an OpenAI-compatible API\nfrom one `trn2.48xlarge`, runs decode as NKI megakernels (about one kernel per layer half), and keeps a prefix cache\nacross device and host memory. **GLM-5.3-Flash is the first supported model; more open models are on the way.**\n\nHow we built it, with the AgentX results: [Serving GLM-5.3-Flash on one Trainium2 box with NKI layer megakernels](https://runanywhere.ai/blog/glm-5-3-flash-trainium2-nki-megakernels).\n\n| Feature | What it does | \n|---|---|\n| **NKI megakernels** | One fused kernel per layer half for decode: KDA attention, the fused DSA layer, mHC and the MoE FFN. | \n| **1M-token context** | Sparse attention (top 2,048) and linear attention, with fixed-size state. | \n| **Prefix cache** | A radix tree over FP8 KV pages, plus a host-memory tier for evicted pages and state checkpoints. | \n| **Two engines per box** | Each engine is tensor-parallel over 32 logical NeuronCores, behind a session-affine router. | \n| **OpenAI API** | Chat and completions, streaming, tool calls and reasoning output. | \n| **Offline compile** | Every graph is built on a CPU host; the serving box loads them and compiles nothing by default. | \n\nYou need Docker, a `trn2.48xlarge` to serve on, and any x86_64 host to compile on. New to Trainium? Start with\n[docs/AWS.md](https://github.com/RunanywhereAI/trnrt/blob/development/docs/AWS.md).\n\n**1. Build the image** (Neuron SDK 2.32 (neuronx-cc 2.27), Python 3.13)\n\n```\ngit clone https://github.com/RunanywhereAI/trnrt.git && cd trnrt\ndocker build -f docker/Dockerfile \\\n  --build-arg TRNRT_VERSION=$(python3 -c 'import trnrt; print(trnrt.__version__)') \\\n  --build-arg TRNRT_REVISION=$(git rev-parse HEAD) -t trnrt .\n```\n\n**2. Build the compile cache** (any x86_64 host, no NeuronCore needed; takes hours)\n\n```\nOUT=/data/trnrt REV=eb9eb208eb0d988989d07a6a12d0fdeb5f52574a\nmkdir -p $OUT/nki $OUT/work\ntrnrt() { docker run --rm -e HF_TOKEN -v $OUT:$OUT \\\n  -v $OUT/nki:/var/tmp/nki-intermediate-cache trnrt \"$@\"; }\ntrnrt compile shapes zai-org/GLM-5.3-Flash --revision $REV --out $OUT\ntrnrt compile decisions --model zai-org/GLM-5.3-Flash --revision $REV --out $OUT\ntrnrt compile capture --out $OUT -- --model $OUT/shapes\ntrnrt compile compile --out $OUT --work-root $OUT/work\ntrnrt compile strip --out $OUT\n```\n\n**3. Serve** (on the `trn2.48xlarge`, with `$OUT/cache` and `$OUT/nki` copied over)\n\n```\nhf download zai-org/GLM-5.3-Flash --revision eb9eb208eb0d988989d07a6a12d0fdeb5f52574a \\\n  --local-dir /models/GLM-5.3-Flash\ndocker run --rm --privileged --network host --ipc=host \\\n  -v /data/trnrt/cache:/cache \\\n  -v /data/trnrt/nki:/var/tmp/nki-intermediate-cache \\\n  -v /models/GLM-5.3-Flash:/model \\\n  -e NEURON_LIBTORCH_CACHE_ROOT=/cache \\\n  trnrt serve --model /model --port 8000\n```\n\n**4. Call it** with any OpenAI client\n\n``` python\nfrom openai import OpenAI\n\nclient = OpenAI(base_url=\"http://localhost:8000/v1\", api_key=\"unused\")\nreply = client.chat.completions.create(\n    model=\"glm\",\n    messages=[{\"role\": \"user\", \"content\": \"Write a haiku about Trainium.\"}],\n    extra_body={\"session_id\": \"my-session\"},  # keeps a conversation on one engine's prefix cache\n)\nprint(reply.choices[0].message.content)\n```\n\nThe full guide covers what each step needs, start-up checks, the batch 4, 8 and 12 configurations and every flag:\n[deploy](https://github.com/RunanywhereAI/trnrt/blob/development/docs/DEPLOY.md) · [API](https://github.com/RunanywhereAI/trnrt/blob/development/docs/API.md) · [configuration](https://github.com/RunanywhereAI/trnrt/blob/development/docs/CONFIGURATION.md).\n\n| Target | Status | \n|---|---|\n| **AWS Trainium2** ,`trn2.48xlarge` (16 chips, 2 engines × TP 32, LNC 2) | ✅ Supported | \n| **[GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash)** (320B MoE, 18B active, FP8, 1M-token context) | ✅ Supported | \n| More open models | 🔜 Coming soon. [Request a model](https://github.com/RunanywhereAI/trnrt/issues/new?template=30-model-request.yml) | \n\nGLM-5.3-Flash is where TrnRT starts. The engine, router, caches and compile pipeline are built to carry more model\nfamilies, and [docs/ARCHITECTURE.md](https://github.com/RunanywhereAI/trnrt/blob/development/docs/ARCHITECTURE.md#adding-a-model-family) describes what a new one needs.\n\n[SemiAnalysis AgentX](https://github.com/SemiAnalysisAI/agentx-harness) replays real coding-agent sessions. One\n`trn2.48xlarge` ran it at a 1M-token context with 32 concurrent agents, for one hour per point:\n\n| Decode batch | Requests / min | Tokens / min | Decode tok/s per user (p50) | First token, cached (p50) | $ / M tokens (spot) | \n|---|---|---|---|---|---|\n| 4 | 19.3 | 2.36M | 46.6 | 76.73 s | $0.061 | \n| 8 | 25.2 | 3.02M | 29.4 | 28.82 s | $0.047 | \n| **12** (default) | **28.9** | **3.57M** | 22.3 | 6.53 s | **$0.040** | \n\nTokens count input (cached input included) and output. Spot price: $8.60/h, us-east-2, 3 Nov 2026. On a capacity block\nat $35.76/h, the batch 12 cost is $0.167 per million tokens. Full tables, cold-prompt latencies, long prompts and\nmethodology are in [docs/RESULTS.md](https://github.com/RunanywhereAI/trnrt/blob/development/docs/RESULTS.md).\n\nThese numbers were measured with an earlier build of TrnRT. This repository passes these checks against that build\n([docs/KNOWN_ISSUES.md](https://github.com/RunanywhereAI/trnrt/blob/development/docs/KNOWN_ISSUES.md#verification-status)):\n\n- Graph equivalence: EQUAL on all 32 ranks, for all 157 graphs that build compiles in the default batch-12 configuration.\n- Outputs: byte-identical to that build's on 384 of 384 requests.\n- Serving: the two-engine serve in the [Quickstart](#quickstart) passes end to end.\n\nDecode is not limited by memory bandwidth here; small instructions, kernel boundaries and exposed collectives dominate it. So\neach decode megakernel follows three rules: keep the working set in SBUF, use both physical cores symmetrically, and move\nthe collectives into the kernel. [docs/MEGAKERNELS.md](https://github.com/RunanywhereAI/trnrt/blob/development/docs/MEGAKERNELS.md) walks through every kernel.\n\n```\ntrnrt/\n├── __main__.py   CLI: serve, engine, compile\n├── serve/        OpenAI API, session-affine router, output parsers\n├── engine/       scheduler, prefix caches, tensor parallelism\n├── models/       GLM-5.3-Flash: KDA, DSA, mHC, MoE, FP8 loading\n├── kernels/      NKI decode megakernels and prefill kernels\n└── compile/      offline compile: shapes, capture, neuronx-cc, strip\ndocker/           the serving and compile image\ntools/            graph-equivalence gate, device checks, doc checks\ntests/            CPU test suite (no NeuronCore needed)\ndocs/             guides, results, kernel notes\n```\n\nMore in [docs/ARCHITECTURE.md](https://github.com/RunanywhereAI/trnrt/blob/development/docs/ARCHITECTURE.md).\n\n```\nmake dev     # editable install with test and lint tools (Python 3.13)\nmake check   # lint, type-check, then the CPU test suite\n```\n\nSee [docs/DEVELOPMENT.md](https://github.com/RunanywhereAI/trnrt/blob/development/docs/DEVELOPMENT.md) for the test tiers and CI, [CONTRIBUTING.md](https://github.com/RunanywhereAI/trnrt/blob/development/CONTRIBUTING.md) to send a\nchange (pull requests target `development`), [SUPPORT.md](https://github.com/RunanywhereAI/trnrt/blob/development/SUPPORT.md) to get help, and [SECURITY.md](https://github.com/RunanywhereAI/trnrt/blob/development/SECURITY.md) to\nreport a vulnerability.\n\n| Resource | Link | \n|---|---|\n| Trainium2 and `trn2` instances | [aws.amazon.com/ec2/instance-types/trn2](https://aws.amazon.com/ec2/instance-types/trn2/) | \n| AWS Neuron SDK documentation | [awsdocs-neuron.readthedocs-hosted.com](https://awsdocs-neuron.readthedocs-hosted.com/) | \n| NKI (Neuron Kernel Interface) | [NKI documentation](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/general/nki/index.html) | \n| Neuron deep learning containers | [github.com/aws-neuron/deep-learning-containers](https://github.com/aws-neuron/deep-learning-containers) | \n| EC2 Capacity Blocks for ML | [aws.amazon.com/ec2/capacityblocks](https://aws.amazon.com/ec2/capacityblocks/) | \n\nThanks to the AWS Neuron team for NKI and its documentation, to Z.ai for releasing GLM-5.3-Flash, and to SemiAnalysis\nfor AgentX and its public traces. TrnRT includes code derived from other Apache-2.0 projects: see [NOTICE](https://github.com/RunanywhereAI/trnrt/blob/development/NOTICE)\nand [THIRD_PARTY_NOTICES.md](https://github.com/RunanywhereAI/trnrt/blob/development/THIRD_PARTY_NOTICES.md).\n\n<sub>Built by [RunAnywhere](https://runanywhere.ai).</sub>\n\n<sub>AWS, Amazon Web Services, Trainium, Inferentia and Neuron are trademarks of Amazon.com, Inc. or its affiliates. The\nAWS logo appears here as published in AWS's Architecture Icons. TrnRT is an independent open-source project, not\naffiliated with or endorsed by Amazon Web Services.</sub>", "url": "https://wpnews.pro/news/show-hn-trnrt-an-open-source-llm-inference-engine-for-aws-trainium", "canonical_source": "https://github.com/RunanywhereAI/trnrt", "published_at": "2026-11-17 16:01:04+00:00", "updated_at": "2026-11-17 21:18:32.692553+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "ai-tools"], "entities": ["RunanywhereAI", "TrnRT", "AWS Trainium2", "GLM-5.3-Flash", "Neuron SDK 2.32", "neuronx-cc 2.27", "trn2.48xlarge", "NKI"], "also_reported_by": [], "takeaway": "TrnRT serves GLM-5.3-Flash from one trn2.48xlarge with NKI layer megakernels, a 1M-token context, and an FP8 KV prefix cache, and its graphs are compiled offline on any x86_64 host so the serving box compiles nothing by default.", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/show-hn-trnrt-an-open-source-llm-inference-engine-for-aws-trainium", "markdown": "https://wpnews.pro/news/show-hn-trnrt-an-open-source-llm-inference-engine-for-aws-trainium.md", "text": "https://wpnews.pro/news/show-hn-trnrt-an-open-source-llm-inference-engine-for-aws-trainium.txt", "jsonld": "https://wpnews.pro/news/show-hn-trnrt-an-open-source-llm-inference-engine-for-aws-trainium.jsonld"}}