{"slug": "gpu-retired-pages-py-dump-retired-nvidia-gpu-memory-pages-with-retirement", "title": "gpu_retired_pages.py -- dump retired NVIDIA GPU memory pages WITH retirement timestamps", "summary": "A developer released gpu_retired_pages.py, a read-only Python script that uses NVML to dump retired GPU memory pages with retirement timestamps, a feature nvidia-smi lacks. The tool auto-detects timestamp units, warns when ECC is disabled, and is intended for vetting used datacenter GPUs, though it does not support Ampere and later GPUs, which use row remapping instead.", "body_md": "| #!/usr/bin/env python3 | |\n| \"\"\" | |\n| gpu_retired_pages.py -- dump retired GPU memory pages WITH retirement timestamps. | |\n| nvidia-smi reports retired page *counts* and *addresses* but not *when* each page | |\n| was retired. NVML's nvmlDeviceGetRetiredPages_v2 does return timestamps. This | |\n| script calls it via ctypes against libnvidia-ml.so.1, so it needs no Python | |\n| packages -- only the installed NVIDIA driver. | |\n| Retired pages live in the GPU's InfoROM and persist for the life of the board, | |\n| so timestamps predating a purchase date show the memory was already degrading | |\n| before you owned it. That makes this useful for vetting used datacenter GPUs. | |\n| SCOPE -- this reads dynamic page retirement, a PRE-AMPERE mechanism: | |\n| Supported : Tesla K20 and higher, Quadro 5000 and higher, with framebuffer | |\n| ECC enabled. Covers K80, P40, P100, V100, T4 and similar. | |\n| Excluded : GeForce and GRID -- no page retirement at all. | |\n| Ampere+ : A100, H100, L40, Blackwell and later replaced page retirement | |\n| with ROW REMAPPING. This script reports \"not supported\" on them. | |\n| The equivalent calls are nvmlDeviceGetRemappedRows and | |\n| nvmlDeviceGetRowRemapperHistogram (banks, not pages). | |\n| READ-ONLY -- this script only issues NVML query calls. It never changes ECC | |\n| mode, resets counters, resets the GPU, or touches the InfoROM. The sole thing | |\n| it writes is the optional -o output file on local disk. Safe to run against | |\n| production GPUs and against hardware you are evaluating but do not own. | |\n| CRITICAL CAVEAT -- pages are only retired while ECC is ENABLED. A card running | |\n| with ECC off silently accumulates memory errors and retires nothing. An empty | |\n| table on such a card means \"nothing was being watched\", NOT \"healthy memory\". | |\n| The script reports ECC mode per GPU and warns whenever it is off. | |\n| Usage: | |\n| python3 gpu_retired_pages.py # all GPUs | |\n| python3 gpu_retired_pages.py -i 0 # one GPU by index | |\n| python3 gpu_retired_pages.py -i 0 -o out.txt # also write to file | |\n| python3 gpu_retired_pages.py --csv # machine-readable | |\n| \"\"\" | |\n| import argparse | |\n| import ctypes | |\n| import datetime | |\n| import sys | |\n| NVML_SUCCESS = 0 | |\n| NVML_ERROR_NOT_SUPPORTED = 3 | |\n| NVML_ERROR_INSUFFICIENT_SIZE = 7 | |\n| CAUSE_SBE = 0 # multiple single-bit ECC errors on the same address | |\n| CAUSE_DBE = 1 # a single double-bit ECC error | |\n| CAUSE_LABEL = { | |\n| CAUSE_SBE: \"SBE (2+ single-bit)\", | |\n| CAUSE_DBE: \"DBE (double-bit)\", | |\n| } | |\n| # Sanity window for interpreting timestamps: 2000-01-01 .. 2050-01-01 | |\n| _EPOCH_MIN = 946_684_800 | |\n| _EPOCH_MAX = 2_524_608_000 | |\n| def load_nvml(): | |\n| for name in (\"libnvidia-ml.so.1\", \"libnvidia-ml.so\"): | |\n| try: | |\n| return ctypes.CDLL(name) | |\n| except OSError: | |\n| continue | |\n| sys.exit(\"Could not load libnvidia-ml.so.1 -- is the NVIDIA driver installed?\") | |\n| def err_str(nvml, rc): | |\n| nvml.nvmlErrorString.restype = ctypes.c_char_p | |\n| try: | |\n| return nvml.nvmlErrorString(rc).decode() | |\n| except Exception: | |\n| return f\"NVML error {rc}\" | |\n| def check(nvml, rc, what): | |\n| if rc != NVML_SUCCESS: | |\n| sys.exit(f\"{what} failed: {err_str(nvml, rc)}\") | |\n| def get_string(fn, handle, length=96): | |\n| buf = ctypes.create_string_buffer(length) | |\n| rc = fn(handle, buf, ctypes.c_uint(length)) | |\n| if rc != NVML_SUCCESS: | |\n| return \"n/a\" | |\n| return buf.value.decode(errors=\"replace\") | |\n| def decode_timestamp(raw): | |\n| \"\"\"NVML documents these as Unix timestamps, but the unit has varied across | |\n| driver versions. Auto-detect the scale that lands in a plausible date range.\"\"\" | |\n| if raw == 0: | |\n| return None, \"zero\" | |\n| for divisor, unit in ((1, \"s\"), (1e3, \"ms\"), (1e6, \"us\"), (1e9, \"ns\")): | |\n| secs = raw / divisor | |\n| if _EPOCH_MIN < secs < _EPOCH_MAX: | |\n| return datetime.datetime.fromtimestamp(secs), unit | |\n| return None, \"unrecognized\" | |\n| def get_retired_pages(nvml, handle, cause): | |\n| \"\"\"Returns (list of (address, raw_timestamp), status_string).\"\"\" | |\n| fn = getattr(nvml, \"nvmlDeviceGetRetiredPages_v2\", None) | |\n| if fn is None: | |\n| return [], \"nvmlDeviceGetRetiredPages_v2 unavailable in this driver\" | |\n| capacity = 1024 # spec caps the InfoROM table well below this | |\n| while True: | |\n| count = ctypes.c_uint(capacity) | |\n| addrs = (ctypes.c_ulonglong * capacity)() | |\n| stamps = (ctypes.c_ulonglong * capacity)() | |\n| rc = fn(handle, ctypes.c_uint(cause), ctypes.byref(count), addrs, stamps) | |\n| if rc == NVML_SUCCESS: | |\n| return [(addrs[i], stamps[i]) for i in range(count.value)], \"ok\" | |\n| if rc == NVML_ERROR_INSUFFICIENT_SIZE: | |\n| capacity = max(count.value, capacity * 2) | |\n| continue | |\n| if rc == NVML_ERROR_NOT_SUPPORTED: | |\n| return [], (\"page retirement not supported -- Ampere and later use \" | |\n| \"row remapping instead (see nvidia-smi -q -d ROW_REMAPPER)\") | |\n| return [], err_str(nvml, rc) | |\n| def report_gpu(nvml, index, csv_mode, out): | |\n| handle = ctypes.c_void_p() | |\n| rc = nvml.nvmlDeviceGetHandleByIndex_v2(ctypes.c_uint(index), ctypes.byref(handle)) | |\n| check(nvml, rc, f\"nvmlDeviceGetHandleByIndex_v2({index})\") | |\n| name = get_string(nvml.nvmlDeviceGetName, handle, 96) | |\n| serial = get_string(nvml.nvmlDeviceGetSerial, handle, 32) | |\n| uuid = get_string(nvml.nvmlDeviceGetUUID, handle, 96) | |\n| # Page retirement only happens while ECC is on. Without this, an empty | |\n| # table on an ECC-disabled card reads as \"healthy\" when it means \"blind\". | |\n| cur, pend = ctypes.c_uint(), ctypes.c_uint() | |\n| if nvml.nvmlDeviceGetEccMode(handle, ctypes.byref(cur), | |\n| ctypes.byref(pend)) == NVML_SUCCESS: | |\n| ecc = \"enabled\" if cur.value else \"DISABLED\" | |\n| if cur.value != pend.value: | |\n| ecc += f\" (pending: {'enabled' if pend.value else 'disabled'} -- needs reboot)\" | |\n| else: | |\n| ecc = \"unknown\" | |\n| rows = [] | |\n| notes = [] | |\n| for cause in (CAUSE_DBE, CAUSE_SBE): | |\n| pages, status = get_retired_pages(nvml, handle, cause) | |\n| if status != \"ok\": | |\n| notes.append(f\"{CAUSE_LABEL[cause]}: {status}\") | |\n| for addr, raw in pages: | |\n| dt, unit = decode_timestamp(raw) | |\n| rows.append((dt, unit, raw, addr, CAUSE_LABEL[cause])) | |\n| # Sort oldest first; undated entries last. | |\n| rows.sort(key=lambda r: (r[0] is None, r[0] or datetime.datetime.min)) | |\n| if csv_mode: | |\n| ecc_field = ecc.replace(\",\", \";\") | |\n| if not rows: | |\n| # Still emit one row so ECC state is visible for every GPU. | |\n| out(f\"{index},{serial},{uuid},{ecc_field},,,,\") | |\n| for dt, unit, raw, addr, cause in rows: | |\n| out(f\"{index},{serial},{uuid},{ecc_field},{cause},0x{addr:016x},\" | |\n| f\"{raw},{dt.isoformat() if dt else ''}\") | |\n| return | |\n| out(\"\") | |\n| out(f\"GPU {index}: {name}\") | |\n| out(f\" Serial : {serial}\") | |\n| out(f\" UUID : {uuid}\") | |\n| out(f\" ECC mode : {ecc}\") | |\n| out(f\" Retired pages: {len(rows)}\") | |\n| for note in notes: | |\n| out(f\" Note: {note}\") | |\n| # Fires whether or not pages exist. A populated table on an ECC-disabled | |\n| # card is the more misleading case: it looks like a complete history, but | |\n| # records only the periods when ECC happened to be on. | |\n| if ecc.startswith(\"DISABLED\"): | |\n| out(\"\") | |\n| out(\" WARNING: ECC is disabled on this GPU.\") | |\n| out(\" No pages can be retired and no new memory errors are\") | |\n| out(\" detected while ECC is off. Any history below covers\") | |\n| out(\" only periods when ECC was enabled, and absence of\") | |\n| out(\" entries is not evidence of healthy memory.\") | |\n| if not rows: | |\n| out(\" (no retired pages)\") | |\n| return | |\n| out(\"\") | |\n| out(f\" {'Retired at':<21} {'Address':<20} {'Cause':<20} {'Raw timestamp'}\") | |\n| out(f\" {'-' * 21} {'-' * 20} {'-' * 20} {'-' * 20}\") | |\n| for dt, unit, raw, addr, cause in rows: | |\n| when = dt.strftime(\"%Y-%m-%d %H:%M:%S\") if dt else f\"unparsed ({unit})\" | |\n| out(f\" {when:<21} 0x{addr:016x} {cause:<20} {raw}\") | |\n| dated = [r[0] for r in rows if r[0] is not None] | |\n| if dated: | |\n| out(\"\") | |\n| out(f\" Earliest retirement: {min(dated)}\") | |\n| out(f\" Latest retirement : {max(dated)}\") | |\n| def main(): | |\n| ap = argparse.ArgumentParser(description=__doc__, | |\n| formatter_class=argparse.RawDescriptionHelpFormatter) | |\n| ap.add_argument(\"-i\", \"--index\", type=int, default=None, | |\n| help=\"GPU index (default: all GPUs)\") | |\n| ap.add_argument(\"-o\", \"--output\", help=\"also write output to this file\") | |\n| ap.add_argument(\"--csv\", action=\"store_true\", | |\n| help=\"CSV output: index,serial,uuid,cause,address,raw_ts,iso_ts\") | |\n| args = ap.parse_args() | |\n| lines = [] | |\n| def out(line=\"\"): | |\n| lines.append(line) | |\n| print(line) | |\n| nvml = load_nvml() | |\n| check(nvml, nvml.nvmlInit_v2(), \"nvmlInit_v2\") | |\n| try: | |\n| dbuf = ctypes.create_string_buffer(80) | |\n| driver = (dbuf.value.decode(errors=\"replace\") | |\n| if nvml.nvmlSystemGetDriverVersion(dbuf, ctypes.c_uint(80)) == NVML_SUCCESS | |\n| else \"n/a\") | |\n| if args.csv: | |\n| out(\"index,serial,uuid,ecc_mode,cause,address,raw_timestamp,retired_at_iso\") | |\n| else: | |\n| out(f\"Collected : {datetime.datetime.now().isoformat(timespec='seconds')}\") | |\n| out(f\"Driver version : {driver}\") | |\n| if args.index is not None: | |\n| report_gpu(nvml, args.index, args.csv, out) | |\n| else: | |\n| count = ctypes.c_uint() | |\n| check(nvml, nvml.nvmlDeviceGetCount_v2(ctypes.byref(count)), | |\n| \"nvmlDeviceGetCount_v2\") | |\n| for i in range(count.value): | |\n| report_gpu(nvml, i, args.csv, out) | |\n| finally: | |\n| nvml.nvmlShutdown() | |\n| if args.output: | |\n| with open(args.output, \"w\") as fh: | |\n| fh.write(\"\\n\".join(lines) + \"\\n\") | |\n| print(f\"\\nWritten to {args.output}\", file=sys.stderr) | |\n| if __name__ == \"__main__\": | |\n| main() |", "url": "https://wpnews.pro/news/gpu-retired-pages-py-dump-retired-nvidia-gpu-memory-pages-with-retirement", "canonical_source": "https://gist.github.com/samteezy/a788dcf430deb448ae48bbe17c369241", "published_at": "2026-08-21 12:22:52+00:00", "updated_at": "2026-08-21 15:14:39.210618+00:00", "lang": "en", "topics": ["developer-tools", "ai-infrastructure"], "entities": ["NVIDIA", "NVML", "gpu_retired_pages.py"], "alternates": {"html": "https://wpnews.pro/news/gpu-retired-pages-py-dump-retired-nvidia-gpu-memory-pages-with-retirement", "markdown": "https://wpnews.pro/news/gpu-retired-pages-py-dump-retired-nvidia-gpu-memory-pages-with-retirement.md", "text": "https://wpnews.pro/news/gpu-retired-pages-py-dump-retired-nvidia-gpu-memory-pages-with-retirement.txt", "jsonld": "https://wpnews.pro/news/gpu-retired-pages-py-dump-retired-nvidia-gpu-memory-pages-with-retirement.jsonld"}}