The Htop for LLM Inference

LLM Inspector, a new open-source CLI tool, provides detailed GPU memory breakdowns for live LLM inference processes, showing weights, KV cache, and workspace usage. It also projects optimization savings from quantization strategies without modifying the model. The tool aims to bridge system observability and model internals, offering deeper insights than traditional GPU monitors.

LLM Inspector inspects live inference processes and shows exactly how GPU memory is being used, what model is running, how the runtime is configured, and where every reported value comes from. Unlike traditional monitoring tools, it doesn't stop at inspection. It also analyzes the running workload and projects optimization opportunities—starting with quantization—to help you understand how different strategies would impact GPU memory before making any changes. Existing tools tell you that your GPU is using 18 GB. LLM Inspector tells you why : Weights 7 GB KV Cache 8 GB Workspace 1 GB Other 2 GB Then it tells you what would happen if you optimized: FP8 would save ~3 GB of weights. AWQ would save ~5 GB of weights. Weight quantization won't fix an 8 GB KV Cache bottleneck. That is the difference between a GPU monitor and an inference advisor. Most tools stop here: GPU └── Process A └── 17.3 GB LLM Inspector goes one level deeper: GPU └── Process A ├── Weights ├── KV Cache ├── Workspace ├── Activations └── Optimization Analysis Projected That bridge— system observability + model internals , with an htop -style CLI—is the innovation. Deep metrics come from optional embedded attach inside the inference process see the install guide . | Section | Kind | |---|---| | Process, Hardware, Model, Memory, Runtime | Measured from live sources or Unavailable with a reason | | Optimization Analysis | Projected from measured inputs — never mutates the model | llminspect inspect <pid --verbose show provenance for every field Works on any NVIDIA GPU machine — laptop, workstation, cloud VM, bare-metal server, or DGX. Docker is optional. pip install llm-inspector For embedded deep metrics Weights / KV / Activations in the same Python env as the model: pip install "llm-inspector torch " From source contributors : git clone https://github.com/helasaoudi/llm-inspector cd llm-inspector python -m venv .venv && source .venv/bin/activate pip install -e ". torch " llminspect ps llminspect inspect <pid llminspect inspect <pid --verbose With Ollama on the host no Docker : ollama run llama3 llminspect inspect $ pgrep -f "ollama serve" --verbose Ollama · vLLM · HuggingFace Transformers · FastAPI · custom PyTorch macOS · Linux · any NVIDIA GPU server including DGX | Guide | When to read it | |---|---| | PyPI install, test on an inference service, Docker, attach , troubleshooting | | MIT