NVIDIA Recovery documentation

When a training run fails, you need to know what happened, whether another attempt makes sense, and whether it actually started. These guides follow that path from a local demonstration to running and operating recovery on your cluster. New to the project? Start with why NVIDIA Recovery exists.

I want to…	Start here	What I will learn
Evaluate recovery locally	Try the local demo	Watch a simulated failure produce Advice and a restarted Slurm attempt.
Run a job on an installed service	Run jobs	Opt in, follow attempts, and understand recovery decisions.
Deploy to my infrastructure	Install NVIDIA Recovery	Build images, install the backend, and connect Slurm or Slinky.
Change or contribute to the project	Set up development	Find the owning component and use its edit/test loop.
Understand the design	Architecture	Follow a job from submission through observed recovery.

Understand the model

The model separates the work you want to finish from the executions and decisions along the way:

Jobs, workflows, and attempts: follow one training run without losing the history of a failed execution.
Advice: understand what should happen next and why that decision is recorded separately from a scheduler command.
Scheduler integration: see how a controller turns that intent into actions and checks their outcome.

These explain how recovery decisions stay separate from scheduler commands. The FAQ covers fit, requirements, and limitations.

Operate and troubleshoot

After the shared installation, follow the guide for conventional Slurm or Slinky. Continue with validation and operations before enabling actions. Use troubleshooting when jobs, signals, or recovery actions are missing.

Detailed references remain available:

Helm charts and their component configuration.
Log-query backends and Slurm log collection.
Slurm stdout path discovery, including active and finalized accounting fallback, supported patterns, and retry behavior.
API Gateway HTTP and gRPC.
NVIDIA Attribution and its Adapter HTTP contract.
Slurm Advisor Controller internals.
Testing, engineering quality, and architecture decisions.

Participate and get help

Read Contributing for the contribution and CI approval process, Support for help, and Security for private vulnerability reporting. Governance, Maintainers, the Code of Conduct, and the release policy define the project’s policies.

NVIDIA Recovery is experimental and not ready for production use. Attribution is enabled by default and currently requires authorized upstream source access. The local tutorial explicitly selects degraded OFF mode to evaluate recovery without private provider credentials.