{"slug": "deploy-frontier-ai-models-on-aws-trainium-with-sglang-neuron", "title": "Deploy frontier AI models on AWS Trainium with SGLang-Neuron", "summary": "Amazon announced that SGLang-Neuron, an out-of-tree plugin connecting the SGLang serving framework to AWS Trainium chips, is publicly available through an AWS Deep Learning Container released by RadixArk in collaboration with Annapurna Labs. The plugin runs on native PyTorch via TorchNeuron, uses Neuron Kernel Interface (NKI) kernels compiled with torch.compile, and is selected by setting SGLANG_PLATFORM=neuron, while SGLang core continues to handle the API server, request lifecycle, RadixAttention and prefix-cache bookkeeping. SGLang-Neuron also supports prefill-decode disaggregation, transferring the prompt KV cache from a prefill server to a decode server over Elastic Fabric Adapter (EFA) using NIXL, and the walkthrough uses a Trainium3 Amazon EC2 instance with a pytorch-inference-sglang-neuronx container.", "body_md": "## [Artificial Intelligence](https://aws.amazon.com/blogs/machine-learning/)\n\n# Deploy frontier AI models on AWS Trainium with SGLang-Neuron\n\nToday, Amazon announced that [SGLang-Neuron](http://github.com/sgl-project/sglang-neuron) is publicly available and can be accessed through an AWS Deep Learning Container, released by RadixArk in collaboration with Annapurna Labs. With SGLang-Neuron, you can serve models and generate reinforcement learning (RL) rollouts with SGLang on [AWS Trainium](https://aws.amazon.com/ai/machine-learning/trainium/) using the same SGLang interface you use today. Trainium is a portfolio of AWS AI chips that delivers strong economics for AI training and inference at scale. Now, running SGLang on Trainium follows the SGLang workflow you already know: you launch models through SGLang’s server interface and invoke the endpoint using the same generation APIs. Underneath that interface, SGLang-Neuron runs on native PyTorch. With its out-of-tree plugin design, Trainium support and SGLang core develop independently and iterate faster.\n\n## How SGLang runs on Trainium\n\nSGLang-Neuron is an out-of-tree plugin that connects SGLang with AWS Trainium. Installing the plugin registers the neuron platform through SGLang’s plugin entry points, and setting `SGLANG_PLATFORM=neuron` selects it when the server starts. SGLang core continues to provide the API server, request lifecycle, RadixAttention, and prefix-cache bookkeeping. The plugin implements the Trainium-specific models, attention, mixture of experts (MoE), and the graph runner.\n\n[TorchNeuron](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/torch/index.html) registers Trainium as a native PyTorch device. SGLang-Neuron uses standard PyTorch operations for the model execution path and incorporates kernels written with the [Neuron Kernel Interface](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/nki/index.html) (NKI) for performance. It compiles these operations and kernels into executable programs using `torch.compile`, which the graph runner prepares and executes during serving.\n\nSGLang-Neuron also supports [PD disaggregation](https://docs.sglang.io/docs/advanced_features/pd_disaggregation). In this configuration, a prefill server computes the prompt KV cache and transfers it to a decode server using [NIXL](https://github.com/ai-dynamo/nixl) over [Elastic Fabric Adapter (EFA)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-start-nixl.html). The decode server then generates the remaining tokens. Each server still uses SGLang’s scheduler and request lifecycle, while the plugin manages the Trainium execution and KV-cache transfer path.\n\n## Serving models on AWS Trainium using SGLang-Neuron\n\nTo follow this walkthrough, you use a Trainium3 Amazon Elastic Compute Cloud (Amazon EC2) instance with a `pytorch-inference-sglang-neuronx` container. Here are the steps:\n\n1. Start the container.\n2. Launch the SGLang model server inside the container.\n3. Send a request to the model server.\n\nFor additional information, please see the SGLang-Neuron getting started guide for setup instructions, and explore the repository documentation for supported models, features, and validated configurations.\n\n## Conclusion\n\n[SGLang-Neuron](http://github.com/sgl-project/sglang-neuron) brings SGLang’s familiar serving workflow to [AWS Trainium](https://aws.amazon.com/ai/machine-learning/trainium/) through native PyTorch. You can get started by launching the [SGLang-Neuron container](https://gallery.ecr.aws/neuron/pytorch-inference-sglang-neuron) and following the [quickstart](https://github.com/sgl-project-dev/sglang-trainium/blob/main/docs/getting-started/quickstart.md) to serve a supported model. Try it with your workloads, contribute through the project repository, and share feedback to help shape future model and feature support.", "url": "https://wpnews.pro/news/deploy-frontier-ai-models-on-aws-trainium-with-sglang-neuron", "canonical_source": "https://aws.amazon.com/blogs/machine-learning/deploy-frontier-ai-models-on-aws-trainium-with-sglang-neuron/", "published_at": "2026-11-12 22:03:34+00:00", "updated_at": "2026-11-12 22:23:48.750591+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-chips", "large-language-models", "mlops", "developer-tools"], "entities": ["Amazon", "SGLang-Neuron", "AWS Trainium", "RadixArk", "Annapurna Labs", "TorchNeuron", "Neuron Kernel Interface", "NIXL"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/deploy-frontier-ai-models-on-aws-trainium-with-sglang-neuron", "markdown": "https://wpnews.pro/news/deploy-frontier-ai-models-on-aws-trainium-with-sglang-neuron.md", "text": "https://wpnews.pro/news/deploy-frontier-ai-models-on-aws-trainium-with-sglang-neuron.txt", "jsonld": "https://wpnews.pro/news/deploy-frontier-ai-models-on-aws-trainium-with-sglang-neuron.jsonld"}}