Inside NVIDIA Nemotron 3.5 Lightning: Building a (Local) Mixture-of-Experts Explorer A developer built nvidia-nemotron-lightning-35-test, an open local explorer and testing framework for running NVIDIA's Nemotron 3.5 Lightning Mixture-of-Experts model through Ollama. The project provides a sandbox for probing the model's sparse routing, speculative decoding with draft models (DSpark, MTP, DFlash), and serving via vLLM, llama.cpp, and LoRA, while separating measured backend telemetry from illustrative visual simulations. The 30B-parameter model activates only 3B parameters per token, trading dense compute for sparse expert routing. A technical exploration ofthe Mixture-of-Experts MoE design of NVIDIA Nemotron 3.5 Lightning. Introduction Modern AI systems face a fundamental tension: users demand the deep reasoning and multi-step planning of large frontier models, yet running dense 30B+ parameter architectures for high-frequency subtasks like code routing, tool parsing, and syntax validation imposes massive computational and financial overhead. To solve this, NVIDIA introduced Nemotron 3.5 Lightning , an open-weight 30-billion parameter Mixture-of-Experts MoE model that activates only 3 billion parameters per token . This article explores nvidia-nemotron-lightning-35-test , a production-grade local explorer and testing framework designed to interact with, monitor, and visualize Nemotron 3.5 Lightning locally via Ollama. Rather than just wrapping API endpoints, this project provides a clear architectural blueprint for building transparent, low-latency AI applications. Implementation The nvidia-nemotron-lightning-35-test project provides a robust framework and sandbox to explore, serve, and evaluate NVIDIA Nemotron 3.5 Lightning. The repository architecture is structured around core demonstration modules, documentation sets, and utility scripts designed to test MoE behavior, speculative decoding, and application-level routing. At the heart of the system is a Mixture-of-Experts MoE transformer backbone. Unlike dense models where all parameters are activated for every token, Nemotron 3.5 Lightning routes tokens through sparse expert layers, ensuring high representational capacity 30B total combined with rapid execution speed 3B active parameters . Architectural Philosophy: The MoE Paradigm & Efficiency Unlike dense models that utilize their entire parameter set for every token generated, Nemotron 3.5 Lightning routes tokens dynamically through sparse expert layers. This decoupled design provides the representational capacity of a 30B model with the execution speed of a 3B model. +-------------------------------------------------------------------+ | Client / User Input | +-------------------------------------------------------------------+ │ ▼ +-------------------------------------------------------------------+ | NeMo Switchyard / Router Layer | +-------------------------------------------------------------------+ │ ▼ +-------------------------------------------------------------------+ | NVIDIA Nemotron 3.5 Lightning MoE | | - 30B Total Parameters / 3B Active Parameters | | - Multi-Token Prediction MTP & Speculative Decoding Support | +-------------------------------------------------------------------+ │ │ ▼ ▼ +-------------------------+ +-------------------------+ | Draft Models | | Serving Frameworks | | DSpark / MTP / DFlash | | vLLM / llama.cpp / LoRA | +-------------------------+ +-------------------------+ To maintain complete intellectual honesty, the project’s application architecture enforces a strict boundary between hard telemetry data directly measured from backend inference and illustrative modeling educational visual simulations : | Data Component | Origin | Status | | ------------------------------------- | ------------------------------------------------------------ | -------------------------------- | | Connection State & Model Metadata | Ollama /api/version , /api/tags , /api/ps blog-post-V2.md | Measured blog-post-V2.md | | Generation Speed & Token Counts | Ollama final streaming chunk eval count , eval duration blog-post-V2.md | Measured blog-post-V2.md | | Time to First Token TTFT | Client-side performance timer blog-post-V2.md | Measured blog-post-V2.md | | Expert Activation Heatmap | moe demo/router sim.py Deterministic BLAKE2b simulation blog-post-V2.md+ 1 | Illustrative blog-post-V2.md | ⚠️ Design Note on Ollama: Ollama does not currently expose internal per-token router telemetry e.g., revealing exactly which expert ID fired for a specific token . The application explicitly flags its expert heatmaps as simulated, preserving user trust and technical accuracy. Core Project Structure The codebase is organized around a clean separation of concerns, isolating the GUI layer from core business logic and state computation: nvidia-nemotron-lightning-35-test/ ├── Docs/ System documentation and architectural references ├── docs tools/ Automated document parsers and build scripts ├── moe demo/ Core application packages │ ├── init .py │ ├── categories.py Task heuristic classifiers and expert affinities │ ├── config.py Environment-based configuration loader │ ├── engine.py UI-agnostic request builder and stream handler │ ├── metrics.py Nanosecond-to-throughput converters │ ├── ollama client.py Robust REST wrapper for Ollama APIs │ ├── router sim.py Deterministic top-6-of-128 routing simulator │ └── ui components.py Streamlit visual cards and heatmaps └── app.py Central Streamlit GUI entrypoint Implementation Deep Dive Environment Configuration moe demo/config.py Configuration is managed via environment variables with fallback support for local .env files. Using a frozen dataclass ensures settings remain immutable throughout the application lifecycle: """Runtime configuration. Values come from the environment / .env see .env.example .""" from future import annotations import os from dataclasses import dataclass from pathlib import Path ROOT = Path file .resolve .parent.parent def load env file path: Path | None = None - None: """Load KEY=VALUE lines from .env into os.environ without overriding real env vars. A tiny parser is used on purpose so the app has no extra dependency and also works when started with streamlit run app.py instead of scripts/start.sh . """ path = path or ROOT / ".env" if not path.is file : return for raw in path.read text encoding="utf-8" .splitlines : line = raw.strip if not line or line.startswith " " or "=" not in line: continue key, , value = line.partition "=" value = value.strip .strip '"' .strip "'" os.environ.setdefault key.strip , value @dataclass frozen=True class Settings: base url: str model: str keep alive: str request timeout: float def get settings - Settings: """Build settings from the environment defaults only for non-sensitive values .""" load env file return Settings base url=os.environ.get "OLLAMA BASE URL", "http://localhost:11434" .rstrip "/" , model=os.environ.get "OLLAMA MODEL", "nemotron-3.5-lightning:30b-a3b-mlx" , keep alive=os.environ.get "OLLAMA KEEP ALIVE", "15m" , request timeout=float os.environ.get "OLLAMA TIMEOUT", "600" , Note: The default OLLAMA TIMEOUT is set to 600 seconds to gracefully handle cold-starting a massive ∼19 GB∼19 GB model artifact without throwing premature timeouts. Resilient Ollama Client moe demo/ollama client.py The client wraps Ollama's REST API /api/version , /api/tags , /api/ps , /api/generate , and /api/chat , translating raw HTTP errors into a clean exception hierarchy: class OllamaError Exception : ... class OllamaConnectionError OllamaError : ... class OllamaModelNotFound OllamaError : ... class OllamaTimeout OllamaError : ... During streaming chat completions, NDJSON response streams are safely parsed into typed StreamEvent objects, separating thinking/reasoning blocks thinking from final generation tokens content and server statistics done : for line in resp.iter lines : if not line: continue data = json.loads line if "error" in data: raise OllamaError str data "error" msg = data.get "message" or {} if msg.get "thinking" : yield StreamEvent "thinking", msg "thinking" if msg.get "content" : yield StreamEvent "content", msg "content" if data.get "done" : yield StreamEvent "done", stats=data UI-Agnostic Generation Engine moe demo/engine.py The engine converts user parameters into validated option payloads and manages conversation window constraints: php """UI-independent generation logic: parameters - Ollama request - live metrics. Keeping this separate from Streamlit makes it unit-testable against a stub Ollama server. """ from future import annotations import time from dataclasses import dataclass from typing import Iterator from .metrics import RunMetrics, from stats from .ollama client import OllamaClient THINK MODES = "Auto per category ", "Off", "On", "Medium" def build options temperature: float, top p: float, num ctx: int, num predict: int - dict: """Map UI values to Ollama options . These are re-read on every request, so changing a slider affects the very next message.""" if not 0.0 <= temperature <= 2.0: raise ValueError "temperature must be within 0..2" if not 0.0 < top p <= 1.0: raise ValueError "top p must be within 0, 1 " if num ctx < 512: raise ValueError "num ctx must be at least 512" return {"temperature": temperature, "top p": top p, "num ctx": int num ctx , "num predict": int num predict } def resolve think mode: str, category think: str - bool | str: """Translate the UI reasoning selector into the value of Ollama's think field. The model accepts False, True or "medium" see ollama show .""" chosen = category think if mode.startswith "Auto" else mode.lower if chosen == "off": return False if chosen == "medium": return "medium" if chosen == "on": return True raise ValueError f"unknown reasoning mode: {mode r}" def build messages system prompt: str, history: list dict , prompt: str, use history: bool, max turns: int = 6 - list dict : """System prompt + optionally the last max turns turns of the conversation + new prompt.""" messages: list dict = if system prompt: messages.append {"role": "system", "content": system prompt} if use history: for m in history -max turns 2: : if m "role" in "user", "assistant" and m.get "content" : messages.append {"role": m "role" , "content": m "content" } messages.append {"role": "user", "content": prompt} return messages @dataclass class LiveUpdate: """Snapshot yielded while streaming.""" content: str thinking: str chunks: int streamed chunks ~ tokens live approximation elapsed: float ttft: float live tps: float approximate until the final exact metrics arrive final: RunMetrics | None = None def stream answer client: OllamaClient, model: str, messages: list dict , options: dict, think: bool | str, keep alive: str - Iterator LiveUpdate : """Stream a reply and yield LiveUpdate objects; the last one carries exact final metrics.""" start = time.perf counter first: float | None = None content, thinking, chunks = "", "", 0 for ev in client.chat stream model, messages, options=options, think=think, keep alive=keep alive : now = time.perf counter if ev.kind in "content", "thinking" : if first is None: first = now - start chunks += 1 if ev.kind == "content": content += ev.text else: thinking += ev.text gen time = max now - start - first or 0 , 1e-6 yield LiveUpdate content, thinking, chunks, now - start, first or 0.0, chunks / gen time if chunks 1 else 0.0 elif ev.kind == "done": wall = now - start final = from stats ev.stats, ttft=first or wall, wall=wall yield LiveUpdate content, thinking, chunks, wall, first or wall, final.tokens per second, final def reasoning share thinking: str, content: str - float: """Fraction of streamed characters that belong to the reasoning trace 0..1 .""" total = len thinking + len content return len thinking / total if total else 0.0 To prevent unbounded context bloat during long sessions, build messages automatically caps history to the last 6 turns while preserving system instructions. Exact Metric Conversion moe demo/metrics.py Ollama returns internal run durations natively in nanoseconds. The metrics module transforms these numbers into actionable engineering metrics, calculating decode speed exclusively over evaluation duration eval duration to avoid diluting performance stats with prefill phase latencies: """Inference metrics derived from Ollama's final streaming chunk plus client-side timing. Ollama reports durations in nanoseconds. All values here are measured , not estimated. """ from future import annotations from dataclasses import asdict, dataclass from typing import Any NS = 1 000 000 000.0 @dataclass class RunMetrics: prompt tokens: int = 0 generated tokens: int = 0 includes reasoning "thinking" tokens load seconds: float = 0.0 time spent loading the model into memory prompt eval seconds: float = 0.0 generation seconds: float = 0.0 total seconds: float = 0.0 server-side total ttft seconds: float = 0.0 client-side time to first streamed token wall seconds: float = 0.0 client-side end-to-end @property def tokens per second self - float: """Generation speed decode phase .""" return self.generated tokens / self.generation seconds if self.generation seconds 0 else 0.0 @property def prompt tokens per second self - float: return self.prompt tokens / self.prompt eval seconds if self.prompt eval seconds 0 else 0.0 @property def total tokens self - int: return self.prompt tokens + self.generated tokens def to dict self - dict str, Any : d = asdict self d "tokens per second" = round self.tokens per second, 2 d "prompt tokens per second" = round self.prompt tokens per second, 2 d "total tokens" = self.total tokens return d def from stats stats: dict str, Any , ttft: float = 0.0, wall: float = 0.0 - RunMetrics: """Build RunMetrics from the done chunk of /api/chat.""" return RunMetrics prompt tokens=int stats.get "prompt eval count" or 0 , generated tokens=int stats.get "eval count" or 0 , load seconds= stats.get "load duration" or 0 / NS, prompt eval seconds= stats.get "prompt eval duration" or 0 / NS, generation seconds= stats.get "eval duration" or 0 / NS, total seconds= stats.get "total duration" or 0 / NS, ttft seconds=ttft, wall seconds=wall, def active fractions total params b: float = 30.0, active params b: float = 3.0, n experts: int = 128, top k: int = 6 - dict str, float : """Sparsity facts from the published model card: share of weights / experts used per token.""" return { "active param fraction": active params b / total params b, "active expert fraction": top k / n experts, } Deterministic Expert Routing Simulation moe demo/router sim.py To visualize how sparsity behaves across different task types e.g., Code vs. Creative Writing vs. Factual Q&A , the simulator models top-6 selections out of 128 total experts grouped into 8 semantic categories . Using blake2b hashing on individual token-expert pairs ensures the simulation is completely deterministic —the same prompt always produces identical heatmap visualizations—while maintaining realistic variance: """Deterministic simulation of top-k expert routing, for visualisation only. Why a simulation? Ollama gives no access to the router's per-token decisions. To still let users see the idea of sparse routing we mimic it: 128 routed experts as in the published Nemotron 3.5 Lightning card , 6 chosen per token. The experts are arranged in 8 groups of 16 see categories.EXPERT GROUPS . Each token gets a score for every expert = category affinity of the expert's group + a deterministic pseudo-random "token-specific" component hash of token and expert id . The top-k scores win. That mimics a learned router: a soft preference for the right specialists, but with token-level variety, so several groups are used per category. The result is reproducible same prompt - same picture and is NOT telemetry from the model. """ from future import annotations import hashlib import re from dataclasses import dataclass from .categories import CATEGORIES, EXPERT GROUPS N EXPERTS = 128 TOP K = 6 GROUP SIZE = N EXPERTS // len EXPERT GROUPS 16 AFFINITY STRENGTH = 0.5 how strongly category affinity biases the router vs. token noise MAX TOKENS = 200 cap simulated tokens so very long prompts stay cheap @dataclass class RoutingResult: category: str n tokens: int expert counts: list int length 128: how many tokens selected each expert group share: dict str, float group - share of all expert selections sums to 1 top experts: list tuple int, int expert id, count , most used first def group of expert id: int - int: return expert id // GROUP SIZE def noise token: str, expert id: int - float: """Deterministic pseudo-random value in 0, 1 for a token, expert pair.""" digest = hashlib.blake2b f"{token}|{expert id}".encode , digest size=4 .digest return int.from bytes digest, "big" / 2 32 def tokenize text: str - list str : """Cheap word/punctuation tokenizer; only used to size the simulation.""" return re.findall r"\w+| ^\w\s ", text.lower :MAX TOKENS def route token token: str, category key: str - list int : """Return the TOP K expert ids chosen for one token under the given category.""" affinity = CATEGORIES category key .expert affinity scores = for e in range N EXPERTS : group name = EXPERT GROUPS group of e scores.append AFFINITY STRENGTH affinity.get group name, 0.0 + noise token, e , e scores.sort reverse=True return e for , e in scores :TOP K def simulate text: str, category key: str - RoutingResult: """Simulate routing of every token of text and aggregate the selections.""" if category key not in CATEGORIES: raise ValueError f"unknown category: {category key}" tokens = tokenize text or "