NVIDIA has open-sourced the tool it uses to work out why distributed AI jobs fail. It is called NVIDIA Attribution. In a November 2, 2026 post on the NVIDIA Technical Blog, the company says it has run Attribution in production for years and that it analyzes millions of workloads a week across hundreds of thousands of GPUs.
The problem it targets is familiar to anyone who has trained on a cluster. One bad GPU or link produces a flood of errors on other ranks. The exception in your log shows where the job noticed trouble, not where the trouble began. Attribution is built to walk that trail backwards and name the node and the cause.
TL;DR: what NVIDIA released #
| Question | Answer |
|---|---|
| What is it? | A workload-centric failure-attribution system that returns a structured root-cause diagnosis |
| Who made it? | NVIDIA's AI infrastructure resilience teams; the post is by Gargi Prasad, Jason Sewall, Aaron Blakeman, Ajay Hotchandani and Ashwin Poojary |
| Is it open source? | Yes. The repository is dsx-ai-factory/attribution on GitHub. Check the repository for its license before use |
| Does it use an LLM? | No. It uses deterministic rules, so results can be reproduced |
| Which schedulers work out of the box? | Slurm and Kueue, plus static inputs |
| Where does evidence come from? | Loki, local files, Kubernetes pod logs and PromQL metrics |
| How do I try it? | Run the Docker-based quickstart on the sample workloads in the repo; you need Docker and the uv package manager |
| What scale does NVIDIA claim? | Millions of workloads a week across hundreds of thousands of GPUs, on NVIDIA's own infrastructure |
Why is the first error in a distributed job usually not the cause? #
Large training runs use many processes, called ranks, that talk to each other through collective operations in NCCL. If one GPU hits a hard fault, every other rank waiting on it can time out or report a broken connection. The scheduler then marks the job failed, and thousands of messages land in the logs.
NVIDIA's post puts it plainly: "The reported exception shows where the workload stopped, not where the failure began." Engineers who read the loudest error first can chase the wrong node, restart onto the same bad hardware, or file a communication-library bug for what is really a hardware fault.
This is the gap Attribution fills. It does not repair anything. It tells you which node, which domain and which evidence, so the next action is the right one.
How does NVIDIA Attribution work? #
Attribution analyzes one workload attempt at a time. NVIDIA describes four stages.
- Scope the workload. The Blueprint identifies the attempt from scheduler identifiers, or from an explicit time window and node allocation.
- Collect observations. Log and metric connectors create workload-scoped Observation records from application logs, syslogs, GPU driver and Fabric Manager messages, scheduler history, and node and network metrics.
- Extract and correlate events. The Service finds significant events such as GPU XIDs, NCCL failures, scheduler transitions and metric conditions. Deterministic rules judge their timing and interaction, separate downstream symptoms from the event that caused the failure, and pick the root cause.
- Return a structured result. The Blueprint combines the verdict with the workload outcome (
terminal_state), the attributed domain (sub_domain), the symptoms, the affected nodes, and a record of which collectors covered what and which errored.
Downstream systems read terminal_state and sub_domain instead of parsing product-specific logs. That makes the output usable by incident tools, recovery systems and fleet dashboards.
Worked example: an NVLink fault that looks like an NCCL problem #
NVIDIA walks through a synthetic multi-rank training job. One GPU on node-02 hits a hard fault, XID 149, and NCCL errors follow on other ranks.
- Extraction. Application logs contain three different NCCL errors. System logs on node-02 contain XID 149 with error words that decode to an NVLink signal integrity issue.
- Early filtering. A rule named
HandleXidBurstEventsreads the XID burst, decodes the error words, and emits an attribution event for the signal integrity issue on node-02. The three NCCL events stay unchanged at this step. - Final refinement. The framework turns events into per-node symptoms, scrubs them per node, and merges the survivors into a job-wide set. A rule named
Xid149AndNvlinkFilteringkeeps the signal integrity symptom and demotes the others. A rule namedGb200PriorityRootCausenarrows the active set. Outcome mapping then reports a hardware failure.
The result names node-02 and the signal integrity issue. The NCCL reports are not thrown away. Demotion keeps their original sources marked as "interesting," so an engineer can still inspect them. The job is reported as a hardware failure instead of three unrelated communication errors.
The example is synthetic. NVIDIA says so. It shows the logic, not a measured accuracy figure.
Service and Blueprint: what you deploy #
The engine has two deployable pieces that share one core.
Attribution Service. A self-hosted HTTP wrapper over the attribution engine. It takes normalized observations and returns one verdict per workload.
Attribution Blueprint. A Python reference orchestrator. It finds workloads, resolves the ExecutionContext, runs collectors, calls the Service, enriches the verdict, writes an AttributionBlueprintResult, and offers a read-only dashboard. NVIDIA calls it a reference implementation for you to customize.
You can run one Service per Blueprint, or share one Service among several Blueprints. NVIDIA suggests running Blueprints near each cluster's data sources. That split lets one rule set apply across many clusters while connectors differ per site.
| Layer | Included connectors |
|---|---|
| Execution context | Slurm, Kueue, static inputs |
| Logs and metrics | Loki, local files, Kubernetes pod logs, PromQL metrics |
| Output | stdout, SQLite |
How do I try it? #
NVIDIA's quickstart uses sample evidence that ships in the repository, so you can follow the analysis before you touch a live cluster. You need Docker and the uv package manager. The steps, as the post gives them:
git clone https://github.com/dsx-ai-factory/attribution.git nvidia-attribution
cd nvidia-attribution
Then build the Blueprint web image from blueprint/docker/Dockerfile, start the container with ports 8000 and 8080 bound to localhost, and run the nvidia-attribution-blueprint command against blueprint/examples/static/configs, pointing the attribution connector at http://127.0.0.1:8000 and the output at a SQLite file. Open http://127.0.0.1:8080 to see each workload's diagnosis, affected nodes, supporting symptoms and evidence trail. Copy the exact commands from the NVIDIA post rather than from this summary, since flags may change.
For your own cluster, NVIDIA points to the Blueprint configuration reference for scheduler and telemetry connectors.
How do you add your own failure patterns? #
Rules are the core extension point. NVIDIA shows a production extractor that recognizes an invalid Pyxis argument in an application log. It uses a regular expression matching an srun error about a bad --container option, emits a bad_node event with the symptom "srun: invalid pyxis argument," and carries a test case next to the rule.
Three details matter for operators:
- Test cases sit beside the rule. At startup the Service loads the bundled catalog and evaluates the inline cases.
- A rule change is committed and shipped with the Service. Rules are not uploaded dynamically through the HTTP API.
- The repository includes a rule-authoring guide for choosing a rule form, registering symptoms, adding tests and verifying the full catalog.
That design favors reviewable, version-controlled knowledge over a black box. It also means a new failure mode needs a code change and a redeploy.
What does NVIDIA claim about results, and what is verified? #
| Claim | Source | Status |
|---|---|---|
| Used in production for years across NVIDIA AI infrastructure | NVIDIA Technical Blog | NVIDIA's own statement; not independently checked |
| Millions of workloads a week across hundreds of thousands of GPUs | NVIDIA Technical Blog | NVIDIA's own figure; no breakdown by cluster or failure type |
| In one large RDMA-backed cluster it correlated port errors, connection resets and scheduler events to find patterns around specific host and switch ports | NVIDIA Technical Blog | A single anecdote; no numbers on repairs or time saved |
| Deterministic rules make results reproducible | NVIDIA Technical Blog | Consistent with the open design; you can check by running the rules |
| Diagnosis quality depends on telemetry quality | NVIDIA Technical Blog | NVIDIA states this limitation itself |
The post gives no accuracy rate, false-attribution rate or time-to-diagnosis comparison against manual triage. The code is open, so teams can measure those on their own failure history before trusting it.
How does it fit with NVIDIA Recovery? #
Attribution arrived on the same day as another NVIDIA resilience release. Our coverage of NVIDIA Recovery explains the tool that detects crashed, hung and slow jobs and restarts them from checkpoints. NVIDIA positions the two as parts of one stack: Attribution explains why a workload failed and gives evidence for the next action, while recovery tooling acts on it.
NVIDIA's own walkthrough ends by using its recovery guidance to pick the next step for the affected node. A reasonable reading is a pipeline: monitor, attribute, then repair or restart. The post does not say how tightly the two are integrated in the open-source code, so check the repositories.
The same engineering group's Dynamo shadow engine work on faster recovery tackles the serving side of resilience. Together they show NVIDIA treating uptime as a software problem at the scale of tens of thousands of GPUs.
What this means if you run a GPU cluster #
- Lower the cost of a bad node. If you can name the node and cause quickly, you can drain it instead of restarting onto it. At large scale each blind restart wastes GPU hours.
- Keep the evidence. The demotion model, which retains symptoms as "interesting," is a good habit even without this tool.
- Start with your own logs. The sample workloads show the format. Before you wire a live cluster, replay a few past failures through the Blueprint and compare the output with what your engineers concluded.
- Check scheduler fit. Slurm and Kueue are covered. If you use something else, plan to write an execution-context connector.
- Mind the rule coverage. NVIDIA's rules include names like
Gb200PriorityRootCause, which suggests hardware-specific logic for NVIDIA systems. Failures on other accelerators or fabrics may need new rules. - Treat "ambiguous" as an honest answer. A tool that says several causes fit is more useful than one that guesses.
For teams sizing hardware and clusters, our reporting on Rubin NVL72 throughput and cost and Google Cloud's A5x Vera Rubin NVL72 cluster shows how large the machines have become. At that size, per-node failure is routine, and diagnosis tooling matters more.
Limits and open questions #
- No public accuracy data. There is no benchmark for correct attribution rates.
- Rules are static. New failure modes need code changes shipped with the Service.
- Telemetry dependence. Missing or partial logs reduce quality, and NVIDIA says the tool records ambiguity instead of forcing an answer.
- Hardware focus. The walkthrough centers on NVLink, XIDs and NCCL, so it fits NVIDIA GPU fleets best.
- Reference code. The Blueprint is a reference implementation, not a managed product, so expect to adapt it.
- License and support. The post does not state a support commitment. Read the repository license and issue tracker before relying on it in production.
Related reading #
Details are accurate as of November 3, 2026 and come from NVIDIA's own post. Repository contents, flags and supported connectors may change.