{"slug": "the-htop-for-llm-inference", "title": "The Htop for LLM Inference", "summary": "LLM Inspector, a new open-source CLI tool, provides detailed GPU memory breakdowns for live LLM inference processes, showing weights, KV cache, and workspace usage. It also projects optimization savings from quantization strategies without modifying the model. The tool aims to bridge system observability and model internals, offering deeper insights than traditional GPU monitors.", "body_md": "LLM Inspector inspects live inference processes and shows exactly how GPU memory is being used, what model is running, how the runtime is configured, and where every reported value comes from.\n\nUnlike traditional monitoring tools, it doesn't stop at inspection. It also analyzes the running workload and projects optimization opportunities—starting with quantization—to help you understand how different strategies would impact GPU memory **before** making any changes.\n\nExisting tools tell you that your GPU is using 18 GB.\n\nLLM Inspector tells you **why**:\n\n```\nWeights       7 GB\nKV Cache      8 GB\nWorkspace     1 GB\nOther         2 GB\n```\n\nThen it tells you what would happen if you optimized:\n\n```\nFP8 would save ~3 GB of weights.\nAWQ would save ~5 GB of weights.\nWeight quantization won't fix an 8 GB KV Cache bottleneck.\n```\n\nThat is the difference between a GPU monitor and an inference advisor.\n\nMost tools stop here:\n\n```\nGPU\n└── Process A\n    └── 17.3 GB\n```\n\nLLM Inspector goes one level deeper:\n\n```\nGPU\n└── Process A\n    ├── Weights\n    ├── KV Cache\n    ├── Workspace\n    ├── Activations\n    └── Optimization Analysis (Projected)\n```\n\nThat bridge—**system observability + model internals**, with an `htop`\n\n-style CLI—is the innovation. Deep metrics come from optional embedded `attach()`\n\ninside the inference process (see the install guide).\n\n| Section | Kind |\n|---|---|\n| Process, Hardware, Model, Memory, Runtime | Measured from live sources (or `Unavailable` with a reason) |\n| Optimization Analysis | Projected from measured inputs — never mutates the model |\n\n```\nllminspect inspect <pid> --verbose   # show provenance for every field\n```\n\nWorks on any NVIDIA GPU machine — laptop, workstation, cloud VM, bare-metal server, or DGX. Docker is optional.\n\n```\npip install llm-inspector\n```\n\nFor embedded deep metrics (Weights / KV / Activations) in the same Python env as the model:\n\n```\npip install \"llm-inspector[torch]\"\n```\n\nFrom source (contributors):\n\n```\ngit clone https://github.com/helasaoudi/llm-inspector\ncd llm-inspector\npython -m venv .venv && source .venv/bin/activate\npip install -e \".[torch]\"\nllminspect ps\nllminspect inspect <pid>\nllminspect inspect <pid> --verbose\n```\n\nWith Ollama on the host (no Docker):\n\n```\nollama run llama3\nllminspect inspect $(pgrep -f \"ollama serve\") --verbose\n```\n\nOllama · vLLM · HuggingFace Transformers · FastAPI · custom PyTorch\n\nmacOS · Linux · any NVIDIA GPU server (including DGX)\n\n| Guide | When to read it |\n|---|---|\n|\nPyPI install, test on an inference service, Docker, `attach()` , troubleshooting |\n|\n\nMIT", "url": "https://wpnews.pro/news/the-htop-for-llm-inference", "canonical_source": "https://github.com/helasaoudi/llm-inspector", "published_at": "2026-07-18 10:42:26+00:00", "updated_at": "2026-07-18 10:51:10.683262+00:00", "lang": "en", "topics": ["developer-tools", "ai-infrastructure", "large-language-models", "ai-tools"], "entities": ["LLM Inspector", "NVIDIA", "Ollama", "vLLM", "HuggingFace Transformers", "FastAPI", "PyTorch", "DGX"], "alternates": {"html": "https://wpnews.pro/news/the-htop-for-llm-inference", "markdown": "https://wpnews.pro/news/the-htop-for-llm-inference.md", "text": "https://wpnews.pro/news/the-htop-for-llm-inference.txt", "jsonld": "https://wpnews.pro/news/the-htop-for-llm-inference.jsonld"}}