{"slug": "run-ai-models-for-edge-application-development-with-just-a-few-commands", "title": "Run AI Models for Edge Application Development with Just a Few Commands", "summary": "NVIDIA released TensorRT Edge-LLM 0.11.1, a lightweight C++ inference runtime that lets embedded developers download a model and build TensorRT engines for edge deployment with a few trtedge CLI commands. The tutorial runs NVIDIA Nemotron-3.5 Lightning, a 30B-A3B NVFP4 text-only model, on the 64GB DGX Spark, requiring CUDA 13 and TensorRT 10.16.1.11 or newer within the 10.x series plus Python 3.10 to 3.12, with an approximately 21GB first-time model download cached in ~/.cache/tensorrt-edgellm. The same runtime targets deployment on NVIDIA Jetson Thor, DRIVE AGX Thor and IGX Thor, and the tutorial enables DSpark speculative decoding to accelerate generation.", "body_md": "AI is expanding beyond the data center as developers build a new generation of [agents](https://www.nvidia.com/en-us/glossary/ai-agents/) to act at the edge: monitoring driver attention and cabin conditions, helping robots understand their surroundings and plan tasks, and coordinating fleets of machines on production lines. These [physical AI applications](https://developer.nvidia.com/blog/?p=113671) rely on models that perceive, reason, and act within tight memory, power, and latency budgets. But getting a model ready to run can involve substantial setup.\n\n[NVIDIA DGX Spark, now available in a 64GB configuration](https://blogs.nvidia.com/blog/local-ai-dgx-spark-64gb-sync/), gives embedded developers a platform to prototype, fine-tune, and test models and agents. [NVIDIA TensorRT Edge-LLM](https://github.com/NVIDIA/TensorRT-Edge-LLM) provides a lightweight C++ inference runtime designed for embedded workloads, bringing the same inference framework to development on Spark and deployment on platforms such as [NVIDIA Jetson Thor](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-thor/), [NVIDIA DRIVE AGX Thor](https://developer.nvidia.com/drive/agx), and [NVIDIA IGX Thor](https://www.nvidia.com/en-us/edge-computing/products/igx/).\n\nYou can get started in just a few commands with the trtedge command-line interface (CLI). It downloads your selected model and builds [TensorRT](https://github.com/nvidia/tensorrt) engines optimized for inference on Spark.\n\nIn this tutorial, you’ll run [NVIDIA Nemotron-3.5 Lightning](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4), a text-only model, on Spark using TensorRT Edge-LLM. You’ll test prompts in your terminal and serve the model through an API for application prototyping. You’ll also enable DSpark speculative decoding to accelerate generation.\n\n## Prerequisites: Set up your development environment\n\nTo get started, you’ll need a DGX Spark with a supported NVIDIA software stack, NVIDIA CUDA 13, TensorRT 10.16.1.11 or newer within the 10.x series, and Python 3.10 to 3.12. The examples below were tested with TensorRT Edge-LLM 0.11.1 and TensorRT 10.16.1.11.\n\nOpen a terminal on your DGX Spark, create and activate a virtual environment, and install TensorRT Edge-LLM:\n\n```\npython3 -m venv --system-site-packages .venv-edgellm\nsource .venv-edgellm/bin/activate\npython -m pip install --upgrade pip\npython -m pip install \"tensorrt-edgellm[server]==0.11.1\"\n```\n\nNote: The virtual environment keeps newly installed packages separate while allowing access to all system Python packages, including TensorRT.\n\nCheck the TensorRT version and confirm that the TensorRT Edge-LLM runtime loads:\n\n``` python\npython -c \"import tensorrt as trt; v = tuple(map(int, trt.__version__.split('.'))); assert (10, 16, 1, 11) <= v < (11,), f'TensorRT >=10.16.1.11 and <11 required; found {trt.__version__}'; from tensorrt_edgellm import runtime; runtime.load()\"\n```\n\nNote: If compatible system TensorRT bindings are unavailable, follow the [NVIDIA TensorRT pip installation instructions](https://docs.nvidia.com/deeplearning/tensorrt/10.x.x/installing-tensorrt/install-pip.html) to install a compatible TensorRT 10.x version in the activated environment, then rerun the verification command.\n\nWith the virtual environment still active, start a chat with Nemotron-3.5 Lightning:\n\n```\ntrtedge chat nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4\n```\n\nOn your first launch, wait for the model download (approximately 21GB) and engine build to finish. Files are stored in `~/.cache/tensorrt-edgellm` by default, so allow space for both the model and engines. Later launches reuse matching cached engines.\n\nOnce the chat prompt appears, enter:\n\n```\nConvert 25°C to Fahrenheit. Answer in one sentence.\n```\n\nExample response:\n\n```\n25°C is equal to 77°F.\n```\n\nContinue the conversation with follow-up questions, or type `/exit` to end the session.\n\n## Serve your model\n\nServe your model through a local OpenAI-compatible API on Spark for application development and testing.\n\nWith your virtual environment active, start the server:\n\n```\ntrtedge serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \\\n  --host 127.0.0.1 \\\n  --port 8000\n```\n\nThe server builds or reuses an engine for this configuration. Once startup completes, open a second terminal on your DGX Spark and check that it is ready:\n\n```\ncurl -fsS http://127.0.0.1:8000/health/ready\n```\n\nA response containing `\"status\":\"ready\"` confirms that the server is ready. Send your first prompt to the chat `completions` endpoint:\n\n```\ncurl -fsS -N http://127.0.0.1:8000/v1/chat/completions \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n  \"model\": \"nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4\",\n  \"messages\": [\n    {\n      \"role\": \"user\",\n      \"content\": \"Convert 25°C to Fahrenheit. Answer in one sentence.\"\n    }\n  ],\n  \"stream\": true,\n  \"temperature\": 0,\n  \"max_tokens\": 128\n}'\n```\n\nThe response streams as JSON events. Keep the server running for more requests, or press Ctrl+C in the server terminal to stop it.\n\n## Optional: Enable speculative decoding\n\nYou can speed up generation with DSpark [speculative decoding](https://developer.nvidia.com/blog/an-introduction-to-speculative-decoding-for-reducing-latency-in-ai-inference/), which uses a draft model to propose several tokens for the main model to verify in a single pass. To enable it with three draft tokens, press Ctrl+C in the server terminal, then restart the server with DSpark enabled:\n\n```\ntrtedge serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \\\n  --host 127.0.0.1 \\\n  --port 8000 \\\n  --speculative-config '{\n    \"method\": \"dspark\",   \n    \"model\": \"nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark\",\n    \"num_speculative_tokens\": 3\n  }'\n```\n\nOnce the server is ready, send another prompt to the chat `completions` endpoint to try DSpark-enabled generation.\n\nYou can experiment with `num_speculative_tokens` and other settings to tune generation speed for your workload. See the [speculative decoding documentation to learn more](https://nvidia.github.io/TensorRT-Edge-LLM/latest/user_guide/examples/speculative-decoding.html).\n\n## Performance on DGX Spark 64GB\n\nTensorRT Edge-LLM delivers low response latency and fast token generation with NVIDIA Nemotron-3.5 Lightning on DGX Spark 64GB.\n\nThe table below reports time to first token (TTFT) and generation speed with and without DSpark speculative decoding.\n\n| **Decoding** | **Workload** | **Median TTFT (ms)** | **Mean generation speed (tokens/s)** | \n| DSpark OFF | SpeedBench 1K | 201.0 | 73.3 | \n| DSpark OFF | SpeedBench 2K | 329.5 | 72.8 | \n| DSpark OFF | HumanEval prompts | 98.1 | 73.4 | \n| DSpark ON | SpeedBench 1K | 203.0 | 108.5 | \n| DSpark ON | SpeedBench 2K | 331.6 | 105.3 | \n| DSpark ON | HumanEval prompts | 98.7 | 126.9 | \n\n*Table 1. Nemotron-3.5 Lightning performance on DGX Spark 64GB with and without DSpark speculative decoding.*\n\nGeneration speed is the average token generation rate per request, excluding time to first token. DSpark uses three draft tokens. Peak total system memory use during inference with DSpark was 27.9GB, including the OS.\n\nMeasured on DGX Spark 64GB running DGX OS 7.6.0 (Ubuntu 24.04.4 LTS), using NVFP4 with thinking enabled, batch size one, one concurrent streaming request, and a 128-token output limit. Average input lengths were 1,153 tokens for SpeedBench 1K, 2,247 for SpeedBench 2K, and 139 for HumanEval.\n\n## Next steps\n\nTry another model from the [supported models list](https://nvidia.github.io/TensorRT-Edge-LLM/latest/user_guide/getting_started/supported-models.html) and continue developing your application on Spark. Options include vision-language models that interpret images and video, as well as speech recognition and text-to-speech models.\n\nExplore the [TensorRT Edge-LLM documentation](https://nvidia.github.io/TensorRT-Edge-LLM/latest/index.html) for model-specific workflows and requirements on your target embedded platform.\n\nAnd visit the [Nemotron developer page](https://developer.nvidia.com/nemotron) for resources to get started. Explore open Nemotron models and datasets on [Hugging Face](https://huggingface.co/collections/nvidia/nvidia-nemotron-v3) and [Blueprints](https://build.nvidia.com/blueprints) on [build.nvidia.com](http://build.nvidia.com/).", "url": "https://wpnews.pro/news/run-ai-models-for-edge-application-development-with-just-a-few-commands", "canonical_source": "https://developer.nvidia.com/blog/run-ai-models-for-edge-application-development-with-just-a-few-commands/", "published_at": "2026-10-30 20:33:18+00:00", "updated_at": "2026-10-30 20:49:53.375011+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-tools", "developer-tools", "ai-agents"], "entities": ["NVIDIA", "TensorRT Edge-LLM", "NVIDIA Nemotron-3.5 Lightning", "NVIDIA DGX Spark", "NVIDIA Jetson Thor", "NVIDIA DRIVE AGX Thor", "NVIDIA IGX Thor", "TensorRT"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/run-ai-models-for-edge-application-development-with-just-a-few-commands", "markdown": "https://wpnews.pro/news/run-ai-models-for-edge-application-development-with-just-a-few-commands.md", "text": "https://wpnews.pro/news/run-ai-models-for-edge-application-development-with-just-a-few-commands.txt", "jsonld": "https://wpnews.pro/news/run-ai-models-for-edge-application-development-with-just-a-few-commands.jsonld"}}