Nvitop: A better top for Nvidia GPUs

Nvitop is an open-source interactive NVIDIA GPU process viewer and monitoring tool that queries device status directly through NVML Python bindings rather than parsing nvidia-smi output. The package ships a CUDA device selection tool called nvisel for deep learning researchers and provides APIs for developers to build their own monitoring tools, running on both Linux and Windows with Python 3.8+ and dependencies including nvidia-ml-py, psutil, and curses. Nvitop adds monitor mode with bar charts, history graphs, process sorting, filtering, tree-view, environment variable screens, and keystroke signal sending over nvidia-smi.

An interactive NVIDIA-GPU process viewer and beyond, the one-stop solution for GPU process management. The full API references host at https://nvitop.readthedocs.io https://nvitop.readthedocs.io . Monitor mode of nvitop . TERM: GNOME Terminal / OS: Ubuntu 16.04 LTS over SSH / Locale: en US.UTF-8 A Grafana dashboard built on top of nvitop-exporter . nvitop is an interactive NVIDIA device and process monitoring tool. It has a colorful and informative interface that continuously updates the status of the devices and processes. As a resource monitor, it includes many features and options, such as tree-view, environment variable viewing, process filtering, process metrics monitoring, etc. Beyond that, the package also ships a CUDA device selection tool nvisel cuda-visible-devices-selection-tool for deep learning researchers. It also provides handy APIs that allow developers to write their own monitoring tools. Please refer to section More than a Monitor more-than-a-monitor and the full API references at https://nvitop.readthedocs.io https://nvitop.readthedocs.io for more information. Process filtering and a more colorful interface. - Informative and fancy output : show more information than nvidia-smi with colorized fancy box drawing. - Monitor mode : can run as a resource monitor, rather than print the results only once. - bar charts and history graphs - process sorting - process filtering - send signals to processes with a keystroke - tree-view screen for GPU processes and their parent processes - environment variable screen - help screen - mouse support - Interactive : responsive for user input from keyboard and/or mouse in monitor mode. vs. gpustat https://github.com/wookayin/gpustat & py3nvml https://github.com/fbcotter/py3nvml - Efficient : - query device status using NVML Python bindings https://pypi.org/project/nvidia-ml-py directly, instead of parsing the output of nvidia-smi . vs. nvidia-htop https://github.com/peci1/nvidia-htop - support sparse query and cache results with TTLCache from cachetools https://github.com/tkem/cachetools . vs. gpustat https://github.com/wookayin/gpustat - display information using the curses library rather than print with ANSI escape codes. vs. py3nvml https://github.com/fbcotter/py3nvml - asynchronously gather information using multi-threading and correspond to user input much faster. vs. nvtop https://github.com/Syllo/nvtop - query device status using - Portable : work on both Linux and Windows. - get host process information using the cross-platform library psutil https://github.com/giampaolo/psutil instead of calling ps -p <pid in a subprocess. vs. nvidia-htop https://github.com/peci1/nvidia-htop & py3nvml https://github.com/fbcotter/py3nvml - written in pure Python, easy to install with pip . vs. nvtop https://github.com/Syllo/nvtop - get host process information using the cross-platform library - Integrable : easy to integrate into other applications, more than monitoring. vs. nvidia-htop https://github.com/peci1/nvidia-htop & nvtop https://github.com/Syllo/nvtop nvitop supports Windows SHELL: PowerShell / TERM: Windows Terminal / OS: Windows 10 / Locale: en-US - Python 3.8+ - NVIDIA Management Library NVML - nvidia-ml-py - psutil - curses<sup curses </sup with libncursesw NOTE: The NVIDIA Management Library NVML https://developer.nvidia.com/nvidia-management-library-nvml is a C-based programmatic interface for monitoring and managing various states. The runtime version of the NVML library ships with the NVIDIA display driver available at Download Drivers | NVIDIA https://www.nvidia.com/Download/index.aspx , or can be downloaded as part of the NVIDIA CUDA Toolkit available at CUDA Toolkit | NVIDIA Developer https://developer.nvidia.com/cuda-downloads . The lists of OS platforms and NVIDIA-GPUs supported by the NVML library can be found in the NVML API Reference https://docs.nvidia.com/deploy/nvml-api/nvml-api-reference.html . This repository contains a Bash script to install/upgrade the NVIDIA drivers for Ubuntu Linux. For example: git clone --depth=1 https://github.com/XuehaiPan/nvitop.git && cd nvitop Change to tty3 console required for desktop users with GUI tty2 Optional for SSH users sudo chvt 3 or use keyboard shortcut: Ctrl-LeftAlt-F3 bash install-nvidia-driver.sh --package=nvidia-driver-595 install the R595 driver from ppa:graphics-drivers bash install-nvidia-driver.sh --latest install the latest driver from ppa:graphics-drivers bash install-nvidia-driver.sh --latest --open install the latest open-kernel-module driver NVIDIA driver installer for Ubuntu Linux. Run bash install-nvidia-driver.sh --help for more information. The curses library is a built-in module of Python on Unix-like systems, and it is supported by a third-party package called windows-curses on Windows using PDCurses. Inconsistent behavior of nvitop may occur on different terminal emulators on Windows, such as missing mouse support. It is highly recommended to install nvitop in an isolated virtual environment. Simple installation and run via uvx https://docs.astral.sh/uv/guides/tools a.k.a. uv tool run or pipx https://pypa.github.io/pipx : uvx nvitop or pipx run nvitop You can also set this command as an alias in your shell startup file, e.g.: For Bash echo 'alias nvitop="uvx nvitop"' ~/.bashrc For Zsh echo 'alias nvitop="uvx nvitop"' ~/.zshrc For Fish mkdir -p ~/.config/fish echo 'alias nvitop="uvx nvitop"' ~/.config/fish/config.fish For PowerShell New-Item -Path Split-Path -Parent -Path $PROFILE.CurrentUserAllHosts -ItemType Directory -Force 'Function nvitop { uvx nvitop @Args }' $PROFILE.CurrentUserAllHosts or For Bash echo 'alias nvitop="pipx run nvitop"' ~/.bashrc For Zsh echo 'alias nvitop="pipx run nvitop"' ~/.zshrc For Fish mkdir -p ~/.config/fish echo 'alias nvitop="pipx run nvitop"' ~/.config/fish/config.fish For PowerShell New-Item -Path Split-Path -Parent -Path $PROFILE.CurrentUserAllHosts -ItemType Directory -Force 'Function nvitop { pipx run nvitop @Args }' $PROFILE.CurrentUserAllHosts pip3 install --upgrade nvitop conda install -c conda-forge nvitop Install the latest version from GitHub : https://camo.githubusercontent.com/673a0879f7c175fb049fd033a13194b80884f597d0387e8d08188d76bf21d2f7/68747470733a2f2f696d672e736869656c64732e696f2f6769746875622f636f6d6d6974732d73696e63652f58756568616950616e2f6e7669746f702f76312e372e31 pip3 install --upgrade pip setuptools pip3 install git+https://github.com/XuehaiPan/nvitop.git Or, clone this repo and install manually: git clone --depth=1 https://github.com/XuehaiPan/nvitop.git && cd nvitop pip3 install . NOTE: If you encounter the "nvitop: command not found" error after installation, please check whether you have added the Python console script path e.g., "${HOME}/.local/bin" to your PATH environment variable. Alternatively, you can use python3 -m nvitop . Query the device and process status. The output is similar to nvidia-smi , but has been enriched and colorized. bash Query the status of all devices $ nvitop -1 or use python3 -m nvitop -1 Specify query devices by integer indices $ nvitop -1 -o 0 1 only show <GPU 0 and <GPU 1 Only show devices in CUDA VISIBLE DEVICES by integer indices or UUID strings $ nvitop -1 -ov Only show GPU processes with the compute context type: 'C' or 'C+G' $ nvitop -1 -c When the -1 switch is on, the result will be displayed ONLY ONCE same as the default behavior of nvidia-smi . This is much faster and has lower resource usage. See Command Line Options command-line-options-and-environment-variables for more command options. There is also a CLI tool called nvisel that ships with the nvitop PyPI package. See CUDA Visible Devices Selection Tool cuda-visible-devices-selection-tool for more information. Run as a resource monitor: Monitor mode when the display mode is omitted, NVITOP MONITOR MODE will be used $ nvitop or use python3 -m nvitop Automatically configure the display mode according to the terminal size $ nvitop -m auto shortcut: a key Arbitrarily display as full mode $ nvitop -m full shortcut: f key Arbitrarily display as compact mode $ nvitop -m compact shortcut: c key Specify query devices by integer indices $ nvitop -o 0 1 only show <GPU 0 and <GPU 1 Only show devices in CUDA VISIBLE DEVICES by integer indices or UUID strings $ nvitop -ov Only show GPU processes with the compute context type: 'C' or 'C+G' $ nvitop -c Use ASCII characters only $ nvitop -U useful for terminals without Unicode support For light terminals $ nvitop --light For spectrum-like bar charts requires the terminal supports 256-color $ nvitop --colorful You can configure the default monitor mode with the NVITOP MONITOR MODE environment variable default auto if not set . See Command Line Options and Environment Variables command-line-options-and-environment-variables for more command options. In monitor mode, you can use Ctrl-c / T / K keys to interrupt / terminate / kill a process. And it's recommended to terminate or kill a process in the tree-view screen shortcut: t . For normal users, nvitop will shallow other users' processes in low-intensity colors . For system administrators , you can use sudo nvitop to terminate other users' processes. To run nvitop as a viewer only and disable all process-mutating shortcuts, pass --readonly or set NVITOP MONITOR MODE="readonly" . The signal keys above become no-ops, the on-screen "Press ^C INT /T TERM /K KILL to send signals" hint is hidden, and the corresponding rows in the help screen are dimmed. Use this when sharing a session over SSH, demoing on a multi-tenant box, or wrapping nvitop in a non-admin alias. Also, to enter the process metrics screen, select a process and then press the Enter / Return key . nvitop dynamically displays the process metrics with live graphs. Watch metrics for a specific process shortcut: Enter / Return . Press h for help or q to return to the terminal. See Keybindings for Monitor Mode keybindings-for-monitor-mode for more shortcuts. nvitop comes with a help screen shortcut: h . Build and run the Docker image with nvidia-container-toolkit https://github.com/NVIDIA/nvidia-container-toolkit : docker run -it --rm --runtime=nvidia --gpus=all --pid=host ghcr.io/xuehaipan/nvitop:latest NOTE: Don't forget to add the --pid=host option when running the container. If you only need to set up the Grafana dashboard, you can start a dashboard at http://localhost:3000 http://localhost:3000 with the following command: docker compose --project-directory=nvitop-exporter/grafana up --build --detach See nvitop-exporter https://github.com/XuehaiPan/nvitop/blob/main/nvitop-exporter/README.md for more details. Run nvitop directly on the SSH session instead of a login shell: ssh user@host -t nvitop installed by sudo pip3 install ... ssh user@host -t '~/.local/bin/nvitop' installed by pip3 install --user ... NOTE: Users need to add the -t option to allocate a pseudo-terminal over the SSH session for monitor mode. Type nvitop --help for more command options: usage: nvitop --help --version --once | --monitor {auto,full,compact} --interval SEC --no-unicode --readonly --colorful --force-color --light --gpu-util-thresh th1 th2 --mem-util-thresh th1 th2 --only INDEX INDEX ... --only-visible --compute --only-compute --graphics --only-graphics --user USERNAME ... --pid PID PID ... An interactive NVIDIA-GPU process viewer. options: --help, -h Show this help message and exit. --version, -V Show nvitop's version number and exit. --once, -1 Report query data only once. --monitor, -m {auto,full,compact} Run as a resource monitor. Continuously report query data and handle user inputs. If the argument is omitted, the value from NVITOP MONITOR MODE will be used. default fallback mode: auto --interval SEC Process status update interval in seconds. default: 2 --no-unicode, --ascii, -U Use ASCII characters only, which is useful for terminals without Unicode support. --readonly Disable all system and process changing features e.g., terminating processes . Set variable NVITOP MONITOR MODE="readonly" for convenience. coloring: --colorful Use gradient colors to get spectrum-like bar charts. Set variable NVITOP MONITOR MODE="colorful" for convenience. This option is only available when the terminal supports 256 colors. You may need to set environment variable TERM="xterm-256color" . Note that the terminal multiplexer, such as tmux , may override the TERM variable. --force-color Force colorize even when stdout is not a TTY terminal. --light Tweak visual results for light theme terminals in monitor mode. Set variable NVITOP MONITOR MODE="light" on light terminals for convenience. --gpu-util-thresh th1 th2 Thresholds of GPU utilization to determine the load intensity. Coloring rules: light < th1 % <= moderate < th2 % <= heavy. 1 <= th1 < th2 <= 99, defaults: 10 75 --mem-util-thresh th1 th2 Thresholds of GPU memory percent to determine the load intensity. Coloring rules: light < th1 % <= moderate < th2 % <= heavy. 1 <= th1 < th2 <= 99, defaults: 10 80 device filtering: --only, -o INDEX INDEX ... Only show the specified devices, suppress option --only-visible . --only-visible, -ov Only show devices in the CUDA VISIBLE DEVICES environment variable. process filtering: --compute, -c Only show GPU processes with the compute context. type: 'C' or 'C+G' --only-compute, -C Only show GPU processes exactly with the compute context. type: 'C' only --graphics, -g Only show GPU processes with the graphics context. type: 'G' or 'C+G' --only-graphics, -G Only show GPU processes exactly with the graphics context. type: 'G' only --user, -u USERNAME ... Only show processes of the given users or $USER for no argument . --pid, -p PID PID ... Only show processes of the given PIDs. nvitop can accept the following environment variables for monitor mode: | Name | Description | Valid Values | Default Value | |---|---|---|---| | NVITOP MONITOR MODE | The default display mode a comma-separated string | auto / full / compact plain / colorful dark / light readonly disables process-mutating shortcuts | auto,plain,dark | | NVITOP GPU UTILIZATION THRESHOLDS | Thresholds of GPU utilization | 10,75 , 1,99 , ... | 10,75 | | NVITOP MEMORY UTILIZATION THRESHOLDS | Thresholds of GPU memory percent | 10,80 , 1,99 , ... | 10,80 | | LOGLEVEL | Log level for log messages | DEBUG , INFO , WARNING , ... | WARNING | For example: Replace the following export statements if you are not using Bash / Zsh export NVITOP MONITOR MODE="full,light" Full monitor mode with light terminal tweaks nvitop For convenience, you can add these environment variables to your shell startup file, e.g.: For Bash echo 'export NVITOP MONITOR MODE="full"' ~/.bashrc For Zsh echo 'export NVITOP MONITOR MODE="full"' ~/.zshrc For Fish echo 'set -gx NVITOP MONITOR MODE "full"' ~/.config/fish/config.fish For PowerShell '$Env:NVITOP MONITOR MODE = "full"' $PROFILE.CurrentUserAllHosts | Key | Binding | |---|---| | q | Quit and return to the terminal. | | h / ? | Go to the help screen. | | a / f / c | Change the display mode to auto / full / compact . | | r / <C-r / <F5 | Force refresh the window. | | <Up / <Down <A-k / <A-j <Tab / <S-Tab <Wheel | Select and highlight a process. | | <Left / <Right <A-h / <A-l <S-Wheel | Scroll the host information of processes. | | <Home | Select the first process. | | <End | Select the last process. | | <C-a ^ | Scroll left to the beginning of the process entry i.e. beginning of line . | | <C-e $ | Scroll right to the end of the process entry i.e. end of line . | | <PageUp / <PageDown <A-K / <A-J / | scroll entire screen for large amounts of processes . | | <Space | Tag/untag current process. | | <Esc | Clear process selection. | | <C-c I | Send signal.SIGINT to the selected process interrupt . disabled under --readonly | | T | Send signal.SIGTERM to the selected process terminate . disabled under --readonly | | K | Send signal.SIGKILL to the selected process kill . disabled under --readonly | | e | Show process environment. | | t | Toggle tree-view screen. | | <Enter | Show process metrics. | | , / . | Select the sort column. | | / | Reverse the sort order. | | on oN | Sort processes in the natural order, i.e., in ascending descending order of GPU . | | ou oU | Sort processes by USER in ascending descending order. | | op oP | Sort processes by PID in descending ascending order. | | og oG | Sort processes by GPU-MEM in descending ascending order. | | os oS | Sort processes by %SM in descending ascending order. | | oc oC | Sort processes by %CPU in descending ascending order. | | om oM | Sort processes by %MEM in descending ascending order. | | ot oT | Sort processes by TIME in descending ascending order. | HINT: It's recommended to terminate or kill a process in the tree-view screen shortcut: t . Automatically select CUDA VISIBLE DEVICES from the given criteria. Example usage of the CLI tool: bash All devices but sorted $ nvisel or use python3 -m nvitop.select 6,5,4,3,2,1,0,7,8 A simple example to select 4 devices $ nvisel -n 4 or use python3 -m nvitop.select -n 4 6,5,4,3 Select available devices that satisfy the given constraints $ nvisel --min-count 2 --max-count 3 --min-free-memory 5GiB --max-gpu-utilization 60 6,5,4 Set CUDA VISIBLE DEVICES environment variable using nvisel $ export CUDA DEVICE ORDER="PCI BUS ID" CUDA VISIBLE DEVICES="$ nvisel -c 1 -f 10GiB " CUDA VISIBLE DEVICES="6,5,4,3,2,1,0" Use UUID strings in CUDA VISIBLE DEVICES environment variable $ export CUDA VISIBLE DEVICES="$ nvisel -O uuid -c 2 -f 5000M " CUDA VISIBLE DEVICES="GPU-849d5a8d-610e-eeea-1fd4-81ff44a23794,GPU-18ef14e9-dec6-1d7e-1284-3010c6ce98b1,GPU-96de99c9-d68f-84c8-424c-7c75e59cc0a0,GPU-2428d171-8684-5b64-830c-435cd972ec4a,GPU-6d2a57c9-7783-44bb-9f53-13f36282830a,GPU-f8e5a624-2c7e-417c-e647-b764d26d4733,GPU-f9ca790e-683e-3d56-00ba-8f654e977e02" Pipe output to other shell utilities $ nvisel --newline -O uuid -C 6 -f 8GiB GPU-849d5a8d-610e-eeea-1fd4-81ff44a23794 GPU-18ef14e9-dec6-1d7e-1284-3010c6ce98b1 GPU-96de99c9-d68f-84c8-424c-7c75e59cc0a0 GPU-2428d171-8684-5b64-830c-435cd972ec4a GPU-6d2a57c9-7783-44bb-9f53-13f36282830a GPU-f8e5a624-2c7e-417c-e647-b764d26d4733 $ nvisel -0 -O uuid -c 2 -f 4GiB | xargs -0 -I {} nvidia-smi --id={} --query-gpu=index,memory.free --format=csv CUDA VISIBLE DEVICES="GPU-849d5a8d-610e-eeea-1fd4-81ff44a23794,GPU-18ef14e9-dec6-1d7e-1284-3010c6ce98b1,GPU-96de99c9-d68f-84c8-424c-7c75e59cc0a0,GPU-2428d171-8684-5b64-830c-435cd972ec4a,GPU-6d2a57c9-7783-44bb-9f53-13f36282830a,GPU-f8e5a624-2c7e-417c-e647-b764d26d4733,GPU-f9ca790e-683e-3d56-00ba-8f654e977e02" index, memory.free MiB 6, 11018 MiB index, memory.free MiB 5, 11018 MiB index, memory.free MiB 4, 11018 MiB index, memory.free MiB 3, 11018 MiB index, memory.free MiB 2, 11018 MiB index, memory.free MiB 1, 11018 MiB index, memory.free MiB 0, 11018 MiB Normalize the CUDA VISIBLE DEVICES environment variable e.g. convert UUIDs to indices or get full UUIDs for an abbreviated form $ nvisel -i "GPU-18ef14e9,GPU-849d5a8d" -S 5,6 $ nvisel -i "GPU-18ef14e9,GPU-849d5a8d" -S -O uuid --newline GPU-18ef14e9-dec6-1d7e-1284-3010c6ce98b1 GPU-849d5a8d-610e-eeea-1fd4-81ff44a23794 You can also integrate nvisel into your training script like this: python Put this at the top of the Python script import os from nvitop import select devices os.environ 'CUDA VISIBLE DEVICES' = ','.join select devices format='uuid', min count=4, min free memory='8GiB' Type nvisel --help for more command options: usage: nvisel --help --version --inherit CUDA VISIBLE DEVICES --account-as-free USERNAME ... --min-count N --max-count N --count N --min-free-memory SIZE --min-total-memory SIZE --max-gpu-utilization RATE --max-memory-utilization RATE --tolerance TOL --format FORMAT --sep SEP | --newline | --null --no-sort CUDA visible devices selection tool. options: --help, -h Show this help message and exit. --version, -V Show nvisel's version number and exit. constraints: --inherit CUDA VISIBLE DEVICES , -i CUDA VISIBLE DEVICES Inherit the given CUDA VISIBLE DEVICES . If the argument is omitted, use the value from the environment. This means selecting a subset of the currently CUDA-visible devices. --account-as-free USERNAME ... Account the used GPU memory of the given users as free memory. If this option is specified but without argument, $USER will be used. --min-count N, -c N Minimum number of devices to select. default: 0 The tool will fail exit non-zero if the requested resource is not available. --max-count N, -C N Maximum number of devices to select. default: all devices --count N, -n N Overriding both --min-count N and --max-count N . --min-free-memory SIZE, -f SIZE Minimum free memory of devices to select. example value: 4GiB If this constraint is given, check against all devices. --min-total-memory SIZE, -t SIZE Minimum total memory of devices to select. example value: 10GiB If this constraint is given, check against all devices. --max-gpu-utilization RATE, -G RATE Maximum GPU utilization rate of devices to select. example value: 30 If this constraint is given, check against all devices. --max-memory-utilization RATE, -M RATE Maximum memory bandwidth utilization rate of devices to select. example value: 50 If this constraint is given, check against all devices. --tolerance TOL, --tol TOL The constraints tolerance in percentage . default: 0, i.e., strict This option can loose the constraints if the requested resource is not available. For example, set --tolerance=20 will accept a device with only 4GiB of free memory when set --min-free-memory=5GiB . formatting: --format FORMAT, -O FORMAT The output format of the selected device identifiers. default: index If any MIG device found, the output format will be fallback to uuid . --sep SEP, --separator SEP, -s SEP Separator for the output. default: ',' --newline Use newline character as separator for the output, equivalent to --sep=$'\n' . --null, -0 Use null character '\x00' as separator for the output. This option corresponds to the -0 option of xargs . --no-sort, -S Do not sort the device by memory usage and GPU utilization. nvitop can be easily integrated into other applications. You can use nvitop to make your own monitoring tools. The full API references host at https://nvitop.readthedocs.io https://nvitop.readthedocs.io . Runnable reference scripts live in examples/ https://github.com/XuehaiPan/nvitop/blob/main/examples . A browser dashboard example built on top of nvitop.collect in background . A minimal script to monitor the GPU devices based on APIs from nvitop : python from nvitop import Device devices = Device.all or Device.cuda.all to use CUDA ordinal instead for device in devices: processes = device.processes type: Dict int, GpuProcess sorted pids = sorted processes.keys print device print f' - Fan speed: {device.fan speed }%' print f' - Temperature: {device.temperature }C' print f' - GPU utilization: {device.gpu utilization }%' print f' - Total memory: {device.memory total human }' print f' - Used memory: {device.memory used human }' print f' - Free memory: {device.memory free human }' print f' - Processes {len processes } : {sorted pids}' for pid in sorted pids: print f' - {processes pid }' print '-' 120 Another more advanced approach with coloring: python import time from nvitop import Device, GpuProcess, NA, colored print colored time.strftime '%a %b %d %H:%M:%S %Y' , color='red', attrs= 'bold', devices = Device.cuda.all or Device.all to use NVML ordinal instead separator = False for device in devices: processes = device.processes type: Dict int, GpuProcess print colored str device , color='green', attrs= 'bold', print colored ' - Fan speed: ', color='blue', attrs= 'bold', + f'{device.fan speed }%' print colored ' - Temperature: ', color='blue', attrs= 'bold', + f'{device.temperature }C' print colored ' - GPU utilization: ', color='blue', attrs= 'bold', + f'{device.gpu utilization }%' print colored ' - Total memory: ', color='blue', attrs= 'bold', + f'{device.memory total human }' print colored ' - Used memory: ', color='blue', attrs= 'bold', + f'{device.memory used human }' print colored ' - Free memory: ', color='blue', attrs= 'bold', + f'{device.memory free human }' if len processes 0: processes = GpuProcess.take snapshots processes.values , failsafe=True processes.sort key=lambda process: process.username, process.pid print colored f' - Processes {len processes } :', color='blue', attrs= 'bold', fmt = ' {pid:<5} {username:<8} {cpu: 5} {host memory: 8} {time: 8} {gpu memory: 8} {sm: 3} {command:<}'.format print colored fmt pid='PID', username='USERNAME', cpu='CPU%', host memory='HOST-MEM', time='TIME', gpu memory='GPU-MEM', sm='SM%', command='COMMAND' , attrs= 'bold', for snapshot in processes: print fmt pid=snapshot.pid, username=snapshot.username :7 + '+' if len snapshot.username 8 else snapshot.username 7:8 , cpu=snapshot.cpu percent, host memory=snapshot.host memory human, time=snapshot.running time human, gpu memory= snapshot.gpu memory human if snapshot.gpu memory human is not NA else 'WDDM:N/A' , sm=snapshot.gpu sm utilization, command=snapshot.command else: print colored ' - No Running Processes', attrs= 'bold', if separator: print '-' 120 separator = True An example monitoring script built with APIs from nvitop . nvitop provides a helper function take snapshots https://nvitop.readthedocs.io/en/latest/api/collector.html nvitop.take snapshots to retrieve the status of both GPU devices and GPU processes at once. You can type help nvitop.take snapshots in Python REPL for detailed documentation. python In 1 : from nvitop import take snapshots, Device ...: import os ...: os.environ 'CUDA DEVICE ORDER' = 'PCI BUS ID' ...: os.environ 'CUDA VISIBLE DEVICES' = '1,0' comma-separated integers or UUID strings In 2 : take snapshots equivalent to take snapshots Device.all Out 2 : SnapshotResult devices= DeviceSnapshot real=Device index=0, ... , ... , ... , gpu processes= GpuProcessSnapshot real=GpuProcess pid=xxxxxx, device=Device index=0, ... , ... , ... , ... In 3 : device snapshots, gpu process snapshots = take snapshots Device.all type: Tuple List DeviceSnapshot , List GpuProcessSnapshot In 4 : device snapshots, = take snapshots gpu processes=False ignore process snapshots In 5 : take snapshots Device.cuda.all use CUDA device enumeration Out 5 : SnapshotResult devices= CudaDeviceSnapshot real=CudaDevice cuda index=0, nvml index=1, ... , ... , CudaDeviceSnapshot real=CudaDevice cuda index=1, nvml index=0, ... , ... , , gpu processes= GpuProcessSnapshot real=GpuProcess pid=xxxxxx, device=CudaDevice cuda index=0, ... , ... , ... , ... In 6 : take snapshots Device.cuda 1 <CUDA 1 only Out 6 : SnapshotResult devices= CudaDeviceSnapshot real=CudaDevice cuda index=1, nvml index=0, ... , ... , gpu processes= GpuProcessSnapshot real=GpuProcess pid=xxxxxx, device=CudaDevice cuda index=1, ... , ... , ... , ... Please refer to section Low-level APIs low-level-apis for more information. ResourceMetricCollector https://nvitop.readthedocs.io/en/latest/api/collector.html nvitop.ResourceMetricCollector is a class that collects resource metrics for host, GPUs and processes running on the GPUs. All metrics will be collected in an asynchronous manner. You can type help nvitop.ResourceMetricCollector in Python REPL for detailed documentation. python In 1 : from nvitop import ResourceMetricCollector, Device ...: import os ...: os.environ 'CUDA DEVICE ORDER' = 'PCI BUS ID' ...: os.environ 'CUDA VISIBLE DEVICES' = '3,2,1,0' comma-separated integers or UUID strings In 2 : collector = ResourceMetricCollector log all devices and descendant processes of the current process on the GPUs In 3 : collector = ResourceMetricCollector root pids={1} log all devices and all GPU processes In 4 : collector = ResourceMetricCollector devices=Device 0 , root pids={1} log <GPU 0 and all GPU processes on <GPU 0 In 5 : collector = ResourceMetricCollector devices=Device.cuda.all use the CUDA ordinal In 6 : with collector tag='<tag ' : ...: Do something ...: collector.collect - Dict str, float key - '<tag /<scope /<metric unit /<mean/min/max ' { '<tag /host/cpu percent % /mean': 8.967849777683456, '<tag /host/cpu percent % /min': 6.1, '<tag /host/cpu percent % /max': 28.1, ..., '<tag /host/memory percent % /mean': 21.5, '<tag /host/swap percent % /mean': 0.3, '<tag /host/memory used GiB /mean': 91.0136418208109, '<tag /host/load average % 1 min /mean': 10.251427386878328, '<tag /host/load average % 5 min /mean': 10.072539414569503, '<tag /host/load average % 15 min /mean': 11.91126970422139, ..., '<tag /cuda:0 gpu:3 /memory used MiB /mean': 3.875, '<tag /cuda:0 gpu:3 /memory free MiB /mean': 11015.562499999998, '<tag /cuda:0 gpu:3 /memory total MiB /mean': 11019.437500000002, '<tag /cuda:0 gpu:3 /memory percent % /mean': 0.0, '<tag /cuda:0 gpu:3 /gpu utilization % /mean': 0.0, '<tag /cuda:0 gpu:3 /memory utilization % /mean': 0.0, '<tag /cuda:0 gpu:3 /fan speed % /mean': 22.0, '<tag /cuda:0 gpu:3 /temperature C /mean': 25.0, '<tag /cuda:0 gpu:3 /power usage W /mean': 19.11166264116916, ..., '<tag /cuda:1 gpu:2 /memory used MiB /mean': 8878.875, ..., '<tag /cuda:2 gpu:1 /memory used MiB /mean': 8182.875, ..., '<tag /cuda:3 gpu:0 /memory used MiB /mean': 9286.875, ..., '<tag /pid:12345/host/cpu percent % /mean': 151.34342772112265, '<tag /pid:12345/host/host memory MiB /mean': 44749.72373447514, '<tag /pid:12345/host/host memory percent % /mean': 8.675082352111717, '<tag /pid:12345/host/running time min ': 336.23803206741576, '<tag /pid:12345/cuda:1 gpu:4 /gpu memory MiB /mean': 8861.0, '<tag /pid:12345/cuda:1 gpu:4 /gpu memory percent % /mean': 80.4, '<tag /pid:12345/cuda:1 gpu:4 /gpu memory utilization % /mean': 6.711118172407917, '<tag /pid:12345/cuda:1 gpu:4 /gpu sm utilization % /mean': 48.23283397736476, ..., '<tag /duration s ': 7.247399162035435, '<tag /timestamp': 1655909466.9981883 } The results can be easily logged into TensorBoard https://github.com/tensorflow/tensorboard or a CSV file. For example: python import os import torch import torch.nn as nn import torch.nn.functional as F from torch.utils.tensorboard import SummaryWriter from nvitop import CudaDevice, ResourceMetricCollector def add scalar dict writer, main tag, tag scalar dict, global step=None, walltime=None : for tag, scalar in tag scalar dict.items : writer.add scalar f'{main tag}/{tag}', scalar, global step=global step, walltime=walltime Build networks and prepare datasets ... Logger and status collector writer = SummaryWriter collector = ResourceMetricCollector devices=CudaDevice.all , log all visible CUDA devices and use the CUDA ordinal root pids={os.getpid }, only log the descendant processes of the current process interval=1.0, snapshot interval for background daemon thread Start training global step = 0 for epoch in range num epoch : with collector tag='train' : for batch in train dataset: with collector tag='batch' : metrics = train net, batch global step += 1 add scalar dict writer, 'train', metrics, global step=global step add scalar dict writer, 'resources', tag='resources/train/batch/...' collector.collect , global step=global step, add scalar dict writer, 'resources', tag='resources/train/...' collector.collect , global step=epoch, with collector tag='validate' : metrics = validate net, validation dataset add scalar dict writer, 'validate', metrics, global step=epoch add scalar dict writer, 'resources', tag='resources/validate/...' collector.collect , global step=epoch, Another example for logging into a CSV file: python import datetime import time import pandas as pd from nvitop import ResourceMetricCollector Log all devices and all GPU processes collector = ResourceMetricCollector root pids={1}, interval=2.0 df = pd.DataFrame with collector tag='resources' : for in range 60 : Do something time.sleep 60 metrics = collector.collect df metrics = pd.DataFrame.from records metrics, index= len df df = pd.concat df, df metrics , ignore index=True Flush to CSV file ... df.insert 0, 'time', df 'resources/timestamp' .map datetime.datetime.fromtimestamp df.to csv 'results.csv', index=False You can also daemonize the collector in the background using collect in background https://nvitop.readthedocs.io/en/latest/api/collector.html nvitop.collect in background or ResourceMetricCollector.daemonize https://nvitop.readthedocs.io/en/latest/api/collector.html nvitop.ResourceMetricCollector.daemonize with callback functions. python from nvitop import Device, ResourceMetricCollector, collect in background logger = ... def on collect metrics : will be called periodically if logger.is closed : closed manually by user return False logger.log metrics return True def on stop collector : will be called only once at stop if not logger.is closed : logger.close cleanup Record metrics to the logger in the background every 5 seconds. It will collect 5-second mean/min/max for each metric. collect in background on collect, ResourceMetricCollector Device.cuda.all , interval=5.0, on stop=on stop, or simply: ResourceMetricCollector Device.cuda.all .daemonize on collect, interval=5.0, on stop=on stop, The full API references can be found at https://nvitop.readthedocs.io https://nvitop.readthedocs.io . The device module https://nvitop.readthedocs.io/en/latest/api/device.html provides: | Device https://nvitop.readthedocs.io/en/latest/api/device.html nvitop.Device index, uuid, bus id | Live class of the GPU devices, different from the device snapshots. | | PhysicalDevice https://nvitop.readthedocs.io/en/latest/api/device.html nvitop.PhysicalDevice index, uuid, bus id | Class for physical devices. | | MigDevice https://nvitop.readthedocs.io/en/latest/api/device.html nvitop.MigDevice index, uuid, bus id | Class for MIG devices. | | CudaDevice https://nvitop.readthedocs.io/en/latest/api/device.html nvitop.CudaDevice cuda index, nvml index, uuid | Class for devices enumerated over the CUDA ordinal. | | CudaMigDevice https://nvitop.readthedocs.io/en/latest/api/device.html nvitop.CudaMigDevice cuda index, nvml index, uuid | Class for CUDA devices that are MIG devices. | | parse cuda visible devices https://nvitop.readthedocs.io/en/latest/api/device.html nvitop.parse cuda visible devices ... | Parse the given CUDA VISIBLE DEVICES value into a list of NVML device indices. | | normalize cuda visible devices https://nvitop.readthedocs.io/en/latest/api/device.html nvitop.normalize cuda visible devices ... | Parse the given CUDA VISIBLE DEVICES value and convert it into a comma-separated string of UUIDs. | In 1 : from nvitop import ...: host, ...: Device, PhysicalDevice, CudaDevice, ...: parse cuda visible devices, normalize cuda visible devices ...: HostProcess, GpuProcess, ...: NA, ...: ...: import os ...: os.environ 'CUDA DEVICE ORDER' = 'PCI BUS ID' ...: os.environ 'CUDA VISIBLE DEVICES' = '9,8,7,6' comma-separated integers or UUID strings In 2 : Device.driver version Out 2 : '525.60.11' In 3 : Device.cuda driver version the maximum CUDA version supported by the driver can be different from the CUDA Runtime version Out 3 : '12.0' In 4 : Device.cuda runtime version the CUDA Runtime version Out 4 : '11.8' In 5 : Device.count Out 5 : 10 In 6 : CudaDevice.count or Device.cuda.count Out 6 : 4 In 7 : all devices = Device.all all devices on board physical device ...: nvidia0, nvidia1 = Device.from indices 0, 1 from physical device indices ...: all devices Out 7 : PhysicalDevice index=0, name="GeForce RTX 2080 Ti", total memory=11019MiB , PhysicalDevice index=1, name="GeForce RTX 2080 Ti", total memory=11019MiB , PhysicalDevice index=2, name="GeForce RTX 2080 Ti", total memory=11019MiB , PhysicalDevice index=3, name="GeForce RTX 2080 Ti", total memory=11019MiB , PhysicalDevice index=4, name="GeForce RTX 2080 Ti", total memory=11019MiB , PhysicalDevice index=5, name="GeForce RTX 2080 Ti", total memory=11019MiB , PhysicalDevice index=6, name="GeForce RTX 2080 Ti", total memory=11019MiB , PhysicalDevice index=7, name="GeForce RTX 2080 Ti", total memory=11019MiB , PhysicalDevice index=8, name="GeForce RTX 2080 Ti", total memory=11019MiB , PhysicalDevice index=9, name="GeForce RTX 2080 Ti", total memory=11019MiB In 8 : NOTE: The function results might be different between calls when the CUDA VISIBLE DEVICES environment variable has been modified ...: cuda visible devices = Device.from cuda visible devices from the CUDA VISIBLE DEVICES environment variable ...: cuda0, cuda1 = Device.from cuda indices 0, 1 from CUDA device indices might be different from physical device indices if CUDA VISIBLE DEVICES is set ...: cuda visible devices = CudaDevice.all shortcut to Device.from cuda visible devices ...: cuda visible devices = Device.cuda.all Device.cuda is aliased to CudaDevice ...: cuda visible devices Out 8 : CudaDevice cuda index=0, nvml index=9, name="NVIDIA GeForce RTX 2080 Ti", total memory=11019MiB , CudaDevice cuda index=1, nvml index=8, name="NVIDIA GeForce RTX 2080 Ti", total memory=11019MiB , CudaDevice cuda index=2, nvml index=7, name="NVIDIA GeForce RTX 2080 Ti", total memory=11019MiB , CudaDevice cuda index=3, nvml index=6, name="NVIDIA GeForce RTX 2080 Ti", total memory=11019MiB In 9 : nvidia0 = Device 0 from device index or Device index=0 ...: nvidia0 Out 9 : PhysicalDevice index=0, name="GeForce RTX 2080 Ti", total memory=11019MiB In 10 : nvidia1 = Device uuid='GPU-01234567-89ab-cdef-0123-456789abcdef' from UUID string or just Device 'GPU-xxxxxxxx-...' ...: nvidia2 = Device bus id='00000000:06:00.0' from PCI bus ID ...: nvidia1 Out 10 : PhysicalDevice index=1, name="GeForce RTX 2080 Ti", total memory=11019MiB In 11 : cuda0 = CudaDevice 0 from CUDA device index equivalent to CudaDevice cuda index=0 ...: cuda1 = CudaDevice nvml index=8 from physical device index ...: cuda3 = CudaDevice uuid='GPU-xxxxxxxx-...' from UUID string ...: cuda4 = Device.cuda 4 Device.cuda is aliased to CudaDevice ...: cuda0 Out 11 : CudaDevice cuda index=0, nvml index=9, name="NVIDIA GeForce RTX 2080 Ti", total memory=11019MiB In 12 : nvidia0.memory used in bytes Out 12 : 9293398016 In 13 : nvidia0.memory used human Out 13 : '8862MiB' In 14 : nvidia0.gpu utilization in percentage Out 14 : 5 In 15 : nvidia0.processes type: Dict int, GpuProcess Out 15 : { 52059: GpuProcess pid=52059, gpu memory=7885MiB, type=C, device=PhysicalDevice index=0, name="GeForce RTX 2080 Ti", total memory=11019MiB , host=HostProcess pid=52059, name='ipython3', status='sleeping', started='14:31:22' , 53002: GpuProcess pid=53002, gpu memory=967MiB, type=C, device=PhysicalDevice index=0, name="GeForce RTX 2080 Ti", total memory=11019MiB , host=HostProcess pid=53002, name='python', status='running', started='14:31:59' } In 16 : nvidia1 snapshot = nvidia1.as snapshot ...: nvidia1 snapshot Out 16 : PhysicalDeviceSnapshot real=PhysicalDevice index=1, name="GeForce RTX 2080 Ti", total memory=11019MiB , bus id='00000000:05:00.0', compute mode='Default', clock infos=ClockInfos graphics=1815, sm=1815, memory=6800, video=1680 , in MHz clock speed infos=ClockSpeedInfos current=ClockInfos graphics=1815, sm=1815, memory=6800, video=1680 , max=ClockInfos graphics=2100, sm=2100, memory=7000, video=1950 , in MHz cuda compute capability= 7, 5 , current driver model='N/A', decoder utilization=0, in percentage display active='Disabled', display mode='Disabled', encoder utilization=0, in percentage fan speed=22, in percentage gpu utilization=17, in percentage NOTE: this is the utilization rate of SMs, i.e. GPU percent index=1, max clock infos=ClockInfos graphics=2100, sm=2100, memory=7000, video=1950 , in MHz memory clock=6800, in MHz memory free=10462232576, in bytes memory free human='9977MiB', memory info=MemoryInfo total=11554717696, free=10462232576, used=1092485120 in bytes memory percent=9.5, in percentage NOTE: this is the percentage of used GPU memory memory total=11554717696, in bytes memory total human='11019MiB', memory usage='1041MiB / 11019MiB', memory used=1092485120, in bytes memory used human='1041MiB', memory utilization=7, in percentage NOTE: this is the utilization rate of GPU memory bandwidth mig mode='N/A', name='GeForce RTX 2080 Ti', pcie rx throughput=1000, in KiB/s pcie rx throughput human='1000KiB/s', pcie throughput=ThroughputInfo tx=1000, rx=1000 , in KiB/s pcie tx throughput=1000, in KiB/s pcie tx throughput human='1000KiB/s', performance state='P2', persistence mode='Disabled', power limit=250000, in milliwatts mW power status='66W / 250W', in watts W power usage=66051, in milliwatts mW sm clock=1815, in MHz temperature=39, in Celsius total volatile uncorrected ecc errors='N/A', utilization rates=UtilizationRates gpu=17, memory=7, encoder=0, decoder=0 , in percentage uuid='GPU-01234567-89ab-cdef-0123-456789abcdef', In 17 : nvidia1 snapshot.memory percent snapshot uses properties instead of function calls Out 17 : 9.5 In 18 : nvidia1 snapshot 'memory info' snapshot also supports getitem by string Out 18 : MemoryInfo total=11554717696, free=10462232576, used=1092485120 In 19 : nvidia1 snapshot.bar1 memory info snapshot will automatically retrieve not presented attributes from real Out 19 : MemoryInfo total=268435456, free=257622016, used=10813440 NOTE: Some entry values may be 'N/A' type: NaType https://nvitop.readthedocs.io/en/latest/index.html nvitop.NaType , a subclass of str when the corresponding resources are not applicable. The NA https://nvitop.readthedocs.io/en/latest/index.html nvitop.NA value supports arithmetic operations. It acts like math.nan: float . python from nvitop import NA NA 'N/A' 'memory usage: {}'.format NA NA is an instance of str 'memory usage: N/A' NA.lower NA is an instance of str 'n/a' NA.ljust 5 NA is an instance of str 'N/A ' NA + 'str' string contamination if the operand is a string 'N/Astr' float NA explicit conversion to float math.nan nan NA + 1 auto-casting to float if the operand is a number nan NA 1024 auto-casting to float if the operand is a number nan NA / 1024 1024 auto-casting to float if the operand is a number nan You can use entry = 'N/A' conditions to avoid exceptions. It's safe to use float entry for numbers while NaType will be converted to math.nan . For example: memory used: Union int, NaType = device.memory used memory usage in bytes or 'N/A' memory used in mib: float = float memory used / 1 << 20 memory usage in Mebibytes MiB or math.nan It's safe to compare NaType with numbers, but NaType is always larger than any number: devices by used memory = sorted Device.all , key=Device.memory used, reverse=True it's safe to compare 'N/A' with numbers devices by free memory = sorted Device.all , key=Device.memory free, reverse=True please add memory free = 'N/A' checks if sort in descending order here See nvitop.NaType https://nvitop.readthedocs.io/en/latest/apis/index.html nvitop.NaType documentation for more details. The process module https://nvitop.readthedocs.io/en/latest/api/process.html provides: | HostProcess https://nvitop.readthedocs.io/en/latest/api/process.html nvitop.HostProcess pid | Represents an OS process with the given PID. | | GpuProcess https://nvitop.readthedocs.io/en/latest/api/process.html nvitop.GpuProcess pid, device , gpu memory, ... | Represents a process with the given PID running on the given GPU device. | | command join https://nvitop.readthedocs.io/en/latest/api/process.html nvitop.command join cmdline | Returns a shell-escaped string from command line arguments. | In 20 : processes = nvidia1.processes type: Dict int, GpuProcess ...: processes Out 20 : { 23266: GpuProcess pid=23266, gpu memory=1031MiB, type=C, device=Device index=1, name="GeForce RTX 2080 Ti", total memory=11019MiB , host=HostProcess pid=23266, name='python3', status='running', started='2021-05-10 21:02:40' } In 21 : process = processes 23266 ...: process Out 21 : GpuProcess pid=23266, gpu memory=1031MiB, type=C, device=Device index=1, name="GeForce RTX 2080 Ti", total memory=11019MiB , host=HostProcess pid=23266, name='python3', status='running', started='2021-05-10 21:02:40' In 22 : process.status GpuProcess will automatically inherit attributes from GpuProcess.host Out 22 : 'running' In 23 : process.cmdline type: List str Out 23 : 'python3', 'rllib train.py' In 24 : process.command type: str Out 24 : 'python3 rllib train.py' In 25 : process.cwd GpuProcess will automatically inherit attributes from GpuProcess.host Out 25 : '/home/xxxxxx/Projects/xxxxxx' In 26 : process.gpu memory human Out 26 : '1031MiB' In 27 : process.as snapshot Out 27 : GpuProcessSnapshot real=GpuProcess pid=23266, gpu memory=1031MiB, type=C, device=PhysicalDevice index=1, name="GeForce RTX 2080 Ti", total memory=11019MiB , host=HostProcess pid=23266, name='python3', status='running', started='2021-05-10 21:02:40' , cmdline= 'python3', 'rllib train.py' , command='python3 rllib train.py', compute instance id='N/A', cpu percent=98.5, in percentage device=PhysicalDevice index=1, name="GeForce RTX 2080 Ti", total memory=11019MiB , gpu encoder utilization=0, in percentage gpu decoder utilization=0, in percentage gpu instance id='N/A', gpu memory=1081081856, in bytes gpu memory human='1031MiB', gpu memory percent=9.4, in percentage NOTE: this is the percentage of used GPU memory gpu memory utilization=5, in percentage NOTE: this is the utilization rate of GPU memory bandwidth gpu sm utilization=0, in percentage NOTE: this is the utilization rate of SMs, i.e. GPU percent host=HostProcessSnapshot real=HostProcess pid=23266, name='python3', status='running', started='2021-05-10 21:02:40' , cmdline= 'python3', 'rllib train.py' , command='python3 rllib train.py', cpu percent=98.5, in percentage host memory=9113627439, in bytes host memory human='8691MiB', is running=True, memory percent=1.6849018430285683, in percentage name='python3', running time=datetime.timedelta days=1, seconds=80013, microseconds=470024 , running time human='46:13:33', running time in seconds=166413.470024, status='running', username='panxuehai', , host memory=9113627439, in bytes host memory human='8691MiB', is running=True, memory percent=1.6849018430285683, in percentage NOTE: this is the percentage of used host memory name='python3', pid=23266, running time=datetime.timedelta days=1, seconds=80013, microseconds=470024 , running time human='46:13:33', running time in seconds=166413.470024, status='running', type='C', 'C' for Compute / 'G' for Graphics / 'C+G' for Both username='panxuehai', In 28 : process.uids GpuProcess will automatically inherit attributes from GpuProcess.host Out 28 : puids real=1001, effective=1001, saved=1001 In 29 : process.kill GpuProcess will automatically inherit attributes from GpuProcess.host In 30 : list map Device.processes, all devices all processes Out 30 : { 52059: GpuProcess pid=52059, gpu memory=7885MiB, type=C, device=PhysicalDevice index=0, name="GeForce RTX 2080 Ti", total memory=11019MiB , host=HostProcess pid=52059, name='ipython3', status='sleeping', started='14:31:22' , 53002: GpuProcess pid=53002, gpu memory=967MiB, type=C, device=PhysicalDevice index=0, name="GeForce RTX 2080 Ti", total memory=11019MiB , host=HostProcess pid=53002, name='python', status='running', started='14:31:59' }, {}, {}, {}, {}, {}, {}, {}, { 84748: GpuProcess pid=84748, gpu memory=8975MiB, type=C, device=PhysicalDevice index=8, name="GeForce RTX 2080 Ti", total memory=11019MiB , host=HostProcess pid=84748, name='python', status='running', started='11:13:38' }, { 84748: GpuProcess pid=84748, gpu memory=8341MiB, type=C, device=PhysicalDevice index=9, name="GeForce RTX 2080 Ti", total memory=11019MiB , host=HostProcess pid=84748, name='python', status='running', started='11:13:38' } In 31 : this = HostProcess os.getpid ...: this Out 31 : HostProcess pid=35783, name='python', status='running', started='19:19:00' In 32 : this.cmdline type: List str Out 32 : 'python', '-c', 'import IPython; IPython.terminal.ipapp.launch new instance ' In 33 : this.command not simply ' '.join cmdline but quotes are added Out 33 : 'python -c "import IPython; IPython.terminal.ipapp.launch new instance "' In 34 : this.memory info Out 34 : pmem rss=83988480, vms=343543808, shared=12079104, text=8192, lib=0, data=297435136, dirty=0 In 35 : import cupy as cp ...: x = cp.zeros 10000, 1000 ...: this = GpuProcess os.getpid , cuda0 construct from GpuProcess pid, device explicitly rather than calling device.processes ...: this Out 35 : GpuProcess pid=35783, gpu memory=N/A, type=N/A, device=CudaDevice cuda index=0, nvml index=9, name="NVIDIA GeForce RTX 2080 Ti", total memory=11019MiB , host=HostProcess pid=35783, name='python', status='running', started='19:19:00' In 36 : this.update gpu status update used GPU memory from new driver queries Out 36 : 267386880 In 37 : this Out 37 : GpuProcess pid=35783, gpu memory=255MiB, type=C, device=CudaDevice cuda index=0, nvml index=9, name="NVIDIA GeForce RTX 2080 Ti", total memory=11019MiB , host=HostProcess pid=35783, name='python', status='running', started='19:19:00' In 38 : id this == id GpuProcess os.getpid , cuda0 IMPORTANT: the instance will be reused while the process is running Out 38 : True Host inherited from psutil https://github.com/giampaolo/psutil In 39 : host.cpu count Out 39 : 88 In 40 : host.cpu percent Out 40 : 18.5 In 41 : host.cpu times Out 41 : scputimes user=2346377.62, nice=53321.44, system=579177.52, idle=10323719.85, iowait=28750.22, irq=0.0, softirq=11566.87, steal=0.0, guest=0.0, guest nice=0.0 In 42 : host.load average Out 42 : 14.88, 17.8, 19.91 In 43 : host.virtual memory Out 43 : svmem total=270352478208, available=192275968000, percent=28.9, used=53350518784, free=88924037120, active=125081112576, inactive=44803993600, buffers=37006450688, cached=91071471616, shared=23820632064, slab=8200687616 In 44 : host.memory percent Out 44 : 28.9 In 45 : host.swap memory Out 45 : sswap total=65534947328, used=475136, free=65534472192, percent=0.0, sin=2404139008, sout=4259434496 In 46 : host.swap percent Out 46 : 0.0 Example output of nvitop -1 : Example output of nvitop : | Full | Compact | Tree-view screen shortcut: t for GPU processes and their ancestors: NOTE: The process tree is built in backward order recursively back to the tree root . Only GPU processes along with their children and ancestors parents and grandparents ... will be shown. Not all running processes will be displayed. Environment variable screen shortcut: e : Spectrum-like bar charts with option --colorful : See CHANGELOG.md https://github.com/XuehaiPan/nvitop/blob/HEAD/CHANGELOG.md . The source code of nvitop is dual-licensed by the Apache License, Version 2.0 Apache-2.0 and GNU General Public License, Version 3 GPL-3.0 . The nvitop CLI is released under the GPL-3.0 license while the remaining part of nvitop is released under the Apache-2.0 license. The license files can be found at LICENSE https://github.com/XuehaiPan/nvitop/blob/HEAD/LICENSE Apache-2.0 and COPYING https://github.com/XuehaiPan/nvitop/blob/HEAD/COPYING GPL-3.0 . The source code is organized as: nvitop GPL-3.0 ├── init .py Apache-2.0 ├── version.py Apache-2.0 ├── api Apache-2.0 │ ├── LICENSE Apache-2.0 │ └── Apache-2.0 ├── select.py Apache-2.0 ├── main .py GPL-3.0 ├── cli.py GPL-3.0 └── tui GPL-3.0 ├── COPYING GPL-3.0 └── GPL-3.0 Please feel free to use nvitop as a dependency for your own projects. The following Python import statements are permitted: python import nvitop import nvitop as alias import nvitop.api as api import nvitop.device as device from nvitop import from nvitop.api import from nvitop import Device, ResourceMetricCollector The public APIs from nvitop are released under the Apache License, Version 2.0 Apache-2.0 . The original license files can be found at LICENSE https://github.com/XuehaiPan/nvitop/blob/HEAD/LICENSE , nvitop/api/LICENSE https://github.com/XuehaiPan/nvitop/blob/HEAD/nvitop/api/LICENSE . The CLI of nvitop is released under the GNU General Public License, Version 3 GPL-3.0 . The original license files can be found at COPYING https://github.com/XuehaiPan/nvitop/blob/HEAD/COPYING and nvitop/tui/COPYING https://github.com/XuehaiPan/nvitop/blob/HEAD/nvitop/tui/COPYING . If you dynamically load the source code of nvitop 's CLI or TUI: python from nvitop import cli from nvitop import tui import nvitop.cli import nvitop.tui your source code should also be released under the GPL-3.0 License. If you want to add or modify some features of nvitop 's CLI, or copy some source code of nvitop 's CLI into your own code, the source code should also be released under the GPL-3.0 License as nvitop contains some modified source code from ranger https://github.com/ranger/ranger under the GPL-3.0 License .