Jetson AGX Thor hits 6.4x faster MLPerf Edge Agentic speeds with TensorRT Edge-LLM

NVIDIA's Jetson AGX Thor achieved a 6.4x speed increase on the MLPerf Edge Agentic benchmark using the TensorRT Edge-LLM inference engine, according to the report. The gain comes from TensorRT Edge-LLM's management of the KV cache and model weights on the Thor architecture, which reduces token-generation latency in multi-step agentic loops. NVIDIA says the speedup makes local agentic loops feasible for production robotics and autonomous systems, with deployment requiring the latest JetPack version, model conversion to TensorRT engine format, and the TensorRT Edge-LLM runtime.

Jetson AGX Thor hits 6.4x faster MLPerf Edge Agentic speeds with TensorRT Edge-LLM The Jetson AGX Thor just clocked a 6.4x speed increase on the MLPerf Edge Agentic benchmark thanks to TensorRT Edge-LLM. This is a significant jump for edge-based AI agents that need to handle multi-step reasoning and tool selection locally rather than relying on a cloud round-trip. The real-world win here is the ability to process long-context conversations and execute sequences of tasks without the latency spikes that usually kill agentic workflows on edge hardware. Why agentic workflows break on the edge Standard chatbots are simple request-response loops, but agents are different. They have to reason, pick a tool, observe the output, and then decide the next move. This creates a compounding latency problem. If the model generates tokens slowly, the entire agent loop crawls, making the device feel unresponsive. To make this viable on hardware like the AGX Thor, you need more than just raw TFLOPS; you need an inference engine that can handle the specific memory and compute patterns of agentic loops. That is where TensorRT Edge-LLM comes in. It is designed to optimize the token generation speed specifically for these iterative processes, which is why we are seeing such a massive delta in the MLPerf results. Breaking down the performance gain The 6.4x improvement isn't just a random number—it reflects how TensorRT Edge-LLM manages the KV cache and model weights on the Thor architecture. In agentic benchmarks, the model is often dealing with expanding conversation histories. If the engine can't keep up with the token throughput, the "reasoning" phase of the agent becomes a bottleneck. By optimizing for the Jetson AGX Thor, NVIDIA has essentially reduced the time it takes for an agent to move from "thinking" to "acting." This means a robot or a vehicle can evaluate a sensor input, decide on a tool, and execute a command in a fraction of the time it previously took. Getting started with TensorRT Edge-LLM If you are deploying on AGX Thor, the path to these speeds involves integrating the TensorRT Edge-LLM library into your pipeline. You aren't just running a raw model; you are using a specialized engine that optimizes the graph for the specific hardware. 1. Ensure your environment is flashed with the latest JetPack version compatible with AGX Thor. 2. Convert your model to the TensorRT engine format using the provided calibration tools to ensure the quantization doesn't tank your reasoning accuracy. 3. Deploy the model using the TensorRT Edge-LLM runtime to leverage the optimized kernels. The focus here is on throughput. For anyone building autonomous systems, the gap between a 1x and a 6.4x speedup is the difference between a system that feels like it's lagging and one that reacts in real-time. It makes local agentic loops actually feasible for production robotics. Next Stop wasting tokens on shared selective persistent memory for agentic LLMs → https://promptcube3.com/en/threads/9481/ All Replies （3） I want to try this tonight for my local bot. I'm still struggling with Llama-3 latency on the 64GB Orin... I'm dying to test this. Does the speedup hold for 7B models or is it mostly 1B stuff? Finally some breathing room. My current Orin setup chokes on agentic loops, but I wonder if this works with vLLM?