> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/recovery/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/recovery/_mcp/server.

# NVIDIA Recovery documentation

When a training run fails, you need to know what happened, whether another
attempt makes sense, and whether it actually started. These guides follow that
path from a local demonstration to running and operating recovery on your
cluster. New to the project? Start with [why NVIDIA Recovery exists](/overview).

| I want to…                          | Start here                               | What I will learn                                                       |
| ----------------------------------- | ---------------------------------------- | ----------------------------------------------------------------------- |
| Evaluate recovery locally           | [Try the local demo](/getting-started)   | Watch a simulated failure produce Advice and a restarted Slurm attempt. |
| Run a job on an installed service   | [Run jobs](/running-jobs)                | Opt in, follow attempts, and understand recovery decisions.             |
| Deploy to my infrastructure         | [Install NVIDIA Recovery](/installation) | Build images, install the backend, and connect Slurm or Slinky.         |
| Change or contribute to the project | [Set up development](/development)       | Find the owning component and use its edit/test loop.                   |
| Understand the design               | [Architecture](/architecture)            | Follow a job from submission through observed recovery.                 |

## Understand the model

The model separates the work you want to finish from the executions and
decisions along the way:

* [Jobs, workflows, and attempts](/jobs-and-workflows): follow one
  training run without losing the history of a failed execution.
* [Advice](/advice): understand what should happen next and why that
  decision is recorded separately from a scheduler command.
* [Scheduler integration](/scheduler-integration): see how a
  controller turns that intent into actions and checks their outcome.

These explain how recovery decisions stay separate from scheduler commands.
The [FAQ](/faq) covers fit, requirements, and limitations.

## Operate and troubleshoot

After the shared installation, follow the guide for [conventional Slurm](/slurm)
or [Slinky](/slinky). Continue with [validation and operations](/operations)
before enabling actions. Use [troubleshooting](/troubleshooting)
when jobs, signals, or recovery actions are missing.

Detailed references remain available:

* [Helm charts](https://github.com/dsx-ai-factory/NVIDIA-Recovery/blob/main/deploy/helm/README.md) and their component configuration.
* [Log-query backends](https://github.com/dsx-ai-factory/NVIDIA-Recovery/blob/main/docs/ops/configure-log-query-backends.md) and
  [Slurm log collection](https://github.com/dsx-ai-factory/NVIDIA-Recovery/blob/main/docs/ops/loki-slurm-runai-logging.md).
* [Slurm stdout path discovery](/slurm#log-path-discovery), including
  active and finalized accounting fallback, supported patterns, and retry behavior.
* [API Gateway HTTP and gRPC](/api-gateway).
* [NVIDIA Attribution](/optional-attribution) and its
  [Adapter HTTP contract](/attribution-adapter).
* [Slurm Advisor Controller internals](https://github.com/dsx-ai-factory/NVIDIA-Recovery/blob/main/docs/architecture/slurm-advisor-controller.md).
* [Testing](/testing), [engineering quality](https://github.com/dsx-ai-factory/NVIDIA-Recovery/blob/main/docs/quality/go-engineering.md),
  and [architecture decisions](https://github.com/dsx-ai-factory/NVIDIA-Recovery/blob/main/docs/decisions/README.md).

## Participate and get help

Read [Contributing](https://github.com/dsx-ai-factory/NVIDIA-Recovery/blob/main/CONTRIBUTING.md) for the contribution and CI approval
process, [Support](https://github.com/dsx-ai-factory/NVIDIA-Recovery/blob/main/SUPPORT.md) for help, and [Security](https://github.com/dsx-ai-factory/NVIDIA-Recovery/blob/main/SECURITY.md) for
private vulnerability reporting. [Governance](https://github.com/dsx-ai-factory/NVIDIA-Recovery/blob/main/GOVERNANCE.md),
[Maintainers](https://github.com/dsx-ai-factory/NVIDIA-Recovery/blob/main/MAINTAINERS.md), the [Code of Conduct](https://github.com/dsx-ai-factory/NVIDIA-Recovery/blob/main/CODE_OF_CONDUCT.md),
and the [release policy](https://github.com/dsx-ai-factory/NVIDIA-Recovery/blob/main/RELEASE.md) define the project's policies.

NVIDIA Recovery is experimental and not ready for production use. Attribution
is enabled by default and currently requires authorized upstream source access.
The [local tutorial](/getting-started) explicitly selects degraded OFF mode
to evaluate recovery without private provider credentials.