> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/docsfab/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/docsfab/_mcp/server.
# DocsFab Recovery State Reference
> Look up the durable state included in a deployment recovery bundle, its storage backend, and its authority boundaries. For procedures, follow Back Up and Restore Local State or Restore a Deployment.
Look up the durable state included in a deployment recovery bundle, its storage backend, and its authority boundaries. For procedures, follow [Back Up and Restore Local State](/docsfab/manage/recover-local-state) or [Restore a Deployment](/docsfab/manage/restore-deployment).
The first pilot-host recovery drill has not run. The checked-in implementation
and its local tests do not establish that `docsfab.nvidia.com` has a usable external
backup. The latest deployed pilot run verified the physical base backup and
then stopped while validating compensated forge outbox history in the logical
export. No complete bundle was published. A fresh run after reviewed deployment
must verify the corrected export against the source host.
## Understand the Recovery Boundary
The recovery bundle combines independently verifiable physical and logical
artifacts. The physical artifact is an encrypted PostgreSQL base backup. Each
logical artifact is a sanitized, provider-neutral export for one project.
The following table describes how DocsFab treats each state class.
| State Class | Recovery Treatment |
| --- | --- |
| Coordination journal and deployment-epoch history | Preserve ordered, digest-bound logical history for each project without the provider's physical project identifiers. Keep the encrypted physical source image inert. |
| Host jobs, context threads and links, capsule history, reconciliation history, and forge event, delivery, and outbox history | Preserve sanitized, digest-bound service history for each project. Each host-job entry keeps a fixed-size packet reference instead of the packet body, so the logical object grows with retained metadata rather than with packet bytes. |
| Forge payload store (content-addressed request payloads under the project custody root) | Do not export and do not back up today. Forge outbox history keeps each payload digest and locator only, and the physical backup covers the PostgreSQL cluster only. |
| Product Reference Model, source threads, and PRD, SRD, and POR projections | Recover from the project's authoritative Git repository. The recovery bundle does not replace Git as semantic authority. |
| Interrupted work | Record the prior attempt and lease identities as history, return the work to `ready`, and require fresh packet, capsule, attempt, lease, and fence identities. |
| Runtime bindings, live agents, claims, leases, and old capsule authority | Do not import as active authority. A new deployment epoch makes every old proof stale. |
| Slack route map and activation state | Preserve as digest-verified `host-operator-file` objects at `/etc/docsfab/slack-pilot-map.json` (`root:docsfab` mode `0640`) and `/etc/docsfab/activation-state.json` (`root:docsfab` mode `0440`), encrypted with the recovery encryption adapter before they enter the bundle. Older bundles hold them in plaintext and restore unchanged. Restore runs as the recovery identity, which cannot write `/etc/docsfab`, so it authenticates them and stages the manifest and the still-encrypted objects. A separate root step, `docsfab-restore --place-host-operator-files`, checks the staged manifest against the manifest digest and each object against that manifest, has the recovery identity decrypt them, and writes both paths or neither with that owner, group, and mode. It repairs owner, group, and mode when the digest already matches. Credentials under `/etc/docsfab/credentials` stay excluded. A bundle whose manifest omits them is not a complete restore of operator state. |
| Raw private prompts, general channel bodies, private persona or profile memory, worktrees, caches, and process logs | Exclude. |
| Unapproved large objects | Exclude. Include only objects approved for the configured recovery backend. |
Forge outbox history preserves all six database-valid lifecycle states:
`pending`, `claimed`, `suppressed`, `compensation-pending`, `cancelled`, and
`delivered`. It retains claim identity digests, lifecycle timestamps,
cancellation outcomes, delivery state, payload digests, and payload locators.
The export marks `pending`, `claimed`, and `compensation-pending` records for
fresh reconciliation. It rejects any lifecycle tuple that does not match the
source database contract. Restoring this history does not reactivate a claim or
fabricate delivery or cancellation evidence.
## Keep the Logical Object Bounded
Every host-job history entry carries a packet reference, not a packet body. The
exporter still validates the complete stored packet, then retains a fixed-size
record of it with the following fields:
- `requestKey`, the same identity as the history record id.
- `packetSha256`, calculated from the complete validated stored packet.
- `packetBytes`, the byte length of that same serialization.
- `packetFingerprintSha256`, copied from the stored packet.
- `workflow` and `status`.
- `retainedIn`, which names where the referenced bytes are retained.
Nothing is deleted. The referenced bytes stay in the source cluster and travel
in the same bundle inside the encrypted PostgreSQL image, which
`pg_verifybackup` has already passed. Restore never reinstates a stored packet
as active work in any shape: interrupted work requires a fresh packet, and
logical import writes service history only to the immutable recovery ledger.
The bundle therefore stays lossless, but the logical object on its own does
not. A consumer that reads only the logical object no longer sees packet
bodies: that includes `recovery-cli verify`, `recovery-cli inspect-project`,
`recovery-cli restore --mode verify-only`, and the restored deployment's
`recovery_project_history` ledger. Recovering a packet body after this change
means starting the staged PostgreSQL data directory that `docsfab-restore`
publishes, which is deliberately marked `DOCSFAB_DO_NOT_START`, and reading the
source tables. No DocsFab command performs that step today. Treat the logical
object as an index into the physical image rather than as a self-sufficient
forensic record.
This keeps the logical object a function of retained metadata. Before the
change, one project's object grew by the size of every job packet, and the
pilot's export reached 94.6 % service history, of which 76.9 % was the inlined
packet. A 10,000-job project measures 7,901,004 bytes as references against
23,857,230 bytes inlined, and the per-entry cost is now constant.
The export still does not retain every host-job row. Job history accumulates
one row per host job, so the logical object can grow past the legacy 16 MiB
bound even with packet references. The exporter keeps the newest
`recovery.serviceHistoryMaxHostJobs` host-job rows, leaves every other history
kind untouched, and writes a warning that names the catalogued limit, the
observed count, and that older rows remain in the physical image. It does not
refuse the backup. Restore of those clipped rows means reading the source
tables from the staged PostgreSQL data directory, which no DocsFab command
performs today.
Each history entry declares its own shape, through the packet's `kind`. The
logical object's top-level schema stays the same closed 13-key set it has
always had, because the storage adapter that publishes every bundle validates
exactly those keys and is refreshed separately from the release. An object
whose entries mix the two shapes is refused, and an unfamiliar recovery packet
kind is refused by name rather than misread. DocsFab still verifies and
restores objects in the original inline shape.
The copied packet fingerprint is source provenance. You cannot recompute it
from the reference because the packet body is absent. The logical artifact
independently recomputes `recordSha256`, `sourceStateSha256`, and
`projectSha256` over the retained projection. Logical restore writes this
record only to immutable recovery history. It does not restore Slack effects,
resume a live-agent session, or recreate an active host job.
The excluded-word screen still refuses credential-shaped values, private-key
material, host paths, and private prompts. Ordinary documentation words are
allowed only in already validated host-job packet `sourceBranch` and
`targetBranch` fields, which exist only in the original inline shape. A packet
reference carries no branch prose and receives no exception. The exception
applies during closed host-job history parse and during the final
project-export scan of parsed top-level host-job entries. Journal inputs,
manifests, restore receipts, and nested look-alike paths receive no exception.
Local fixtures prove this correction. They do not prove that a live backup or
restore has succeeded.
The manifest binds the release revision, coordination and persistence schema
versions, forge-event schema, pinned coordination-provider commit, service
catalog fingerprint, source deployment and epoch, object inventory, artifact
sizes and SHA-256 digests, the locked project-registry snapshot digest, storage
locators, the logical-object bound the producer assumed, and explicit
exclusions. Verify the expected manifest digest and every listed object before
restoring state.
## Use the Official Pilot Backend
The pilot uses a private DocsFab recovery project,
project ID `485025`. This is a storage adapter for the portable recovery
contract, not a GitLab dependency in the recovery model.
The backend separates its artifact classes across the following stores:
- Protected Git history stores sanitized manifests, logical project exports,
and content-free terminal restore-completion receipts.
- The project's Generic Package Registry stores client-encrypted PostgreSQL
backups and other approved large or private objects.
- Git records digest-bound locators for registry objects. It does not contain
their plaintext content or a credential-bearing download URL.
The protected sink and source adapters must upload the correct artifact class,
read it back, and bind the observed digest to their receipt. Encrypt a physical
backup before it leaves private recovery staging. Keep adapter credentials in
the host credential store, not in a manifest, command argument, issue,
repository, prompt, run record, or log.
The repository ships one reviewed authenticated-encryption adapter, an
AES-256-GCM implementation that reads its key only from the protected host
credential path. Bootstrap installs no adapter by default. You stage the
adapter you choose and name it explicitly, and key custody stays outside the
repository and outside the host. The encryption key must remain outside the
recovery bundle, recovery Git history, Package Registry object, configuration
values, and operator command arguments. The adapter streams plaintext before
the authentication tag is checked, so restore revalidates the decrypted archive
and re-runs `pg_verifybackup`.
Backup packages and local staging bundles expire after
`recovery.backupRetentionDays`, 30 days. Git history in the recovery repository
is not pruned yet: manifests, receipts, and logical exports (Git LFS objects and
historical inline JSON) stay until an authorized history cleanup, which #1790
tracks. After each backup that passes verification and whose external copy is
published and read back, the backup tool deletes package versions and local
staging bundles older than the bound. It always keeps the newest verified backup
and every backup published within the bound, and it deletes nothing after a
failed or local-only backup. New manifests record `expire-after-retention-days`
with the bound. Manifests from the earlier mode that never deleted backups record
no bound and still verify and restore.
## Separate Database Authority
The locked `docsfab-recovery` operating-system service uses 3 database
identities. None of the following identities can assume a
coordination-administration role:
- `docsfab-backup` has `LOGIN` and `REPLICATION`, no role memberships, and no
DDL or DML grants. The physical backup service uses this identity.
- `docsfab-recovery-export` has `LOGIN` and can set only the non-owning,
read-only `docsfab_recovery_export` role in the source cluster.
- `docsfab-recovery-import` exists only in the separately provisioned clean
destination cluster. It can set only the non-owning
`docsfab_recovery_import` role there.
Before physical capture, the source must be a dedicated DocsFab cluster with
exactly `docsfab`, `postgres`, `template0`, and `template1`. The cluster guard
checks every database row, including non-connectable and template databases,
and rejects extra databases, custom templates, unrecognized roles, or stored
password verifiers. The export identity also opens `postgres` and `template1`
and rejects non-system schemas, relations, routines, types, extensions, large
objects, foreign mappings, subscriptions, publications, event triggers, and
database-role settings. `template0` remains non-connectable and is bounded by
its exact owner, flags, tablespace, and access-control row; its local catalog
cannot be inspected by the online non-superuser check, so cluster provisioning
and superuser custody remain part of the trusted deployment boundary.
Peer authentication grants `docsfab-recovery-export` exactly 3 local database
routes: `docsfab`, `postgres`, and `template1`. Each route uses the same
`docsfab` identity map and must precede any broader matching local rule. The
bootstrap rejects a missing route, changed map, or shadowing earlier rule. It
does not allow `all`, `template0`, another database, TCP, or `trust`.
The source and destination cannot be 2 databases in one PostgreSQL cluster.
PostgreSQL role membership is cluster-wide, so that layout would expose source
import authority. The pilot uses `/var/run/postgresql` for the source socket
and `/run/docsfab-recovery-destination` for the clean destination socket. Both
connections use exact peer mappings from the `docsfab-recovery` operating
system identity and contain no password. The destination socket directory is
root-owned and is not writable by the recovery service or human operators.
The source bootstrap removes memberships from the obsolete combined
`docsfab-recovery` database login and makes it non-login. It also audits the
source export and physical backup sessions. Clean-destination provisioning is
a separate deployment action. It grants import authority only in that
destination cluster.
Deploy and rollback hold `/srv/docsfab/deployment.lock` while an installed,
narrow wrapper changes from an operator to the `docsfab-recovery` service. The
wrapper preserves only the required read-only lock descriptor, clears the
environment, and invokes the fixed backup command. Its exact sudo rule grants
no shell, arbitrary command, or environment-setting authority. A failed
transition or backup prevents deployment state from changing.
## Interpret Current Status
The repository contains the provider-neutral artifact contracts and a pilot-host
operator implementation in the internal deployment tree, which the public source
does not include. Local tests exercise
sanitization, digest verification, valid SCM branch metadata, all six valid
forge outbox lifecycle states, two-project isolation, fresh epoch and execution
identities, empty-destination staging, and stale-proof rejection.
A source-bound PostgreSQL 17 harness also runs a real streamed `pg_basebackup`,
validates it with `pg_verifybackup`, performs a test-only authenticated
encrypt/decrypt round trip, and verifies the extracted backup again. It then
exercises the exact CLI DSN path, source-custody preflight, injected
second-project rollback, clean-destination rejection, project isolation, a
pending merge-reconciliation projection, stale-proof rejection, and completion
authority. The test-only cipher is evidence for the protected command seam. It
is not the shipped recovery adapter and it does not decide pilot-host key custody.
This is local container evidence, not a backup or restore of the pilot-host VM.
The pilot is not operationally recoverable until an operator completes and
records a live drill. That drill runs on the installed PostgreSQL major, which
is 16 on the current pilot host, because a base backup taken from one major does
not start on another. The drill includes the protected GitLab backend,
an encrypted Package Registry copy, a clean destination, at least two isolated
projects, project and swarm re-registration, semantic reconciliation, and a
successful resumed work item.
## Related Topics
The following pages provide related information:
- [DocsFab Implementation Status](/docsfab/reference/status-and-configuration/implementation-status)
- [Runtime CLI and Configuration in DocsFab](/docsfab/reference/status-and-configuration/runtime-cli-and-configuration)
- [Documentation Convergence and Freshness in DocsFab](/docsfab/reference/knowledge-and-delivery/delivery/documentation-convergence)
## Moved Topic Links
Earlier section bookmarks lead to the canonical topic below.
Refer to [Configure the Recovery Object Bound in DocsFab](/docsfab/manage/configure-recovery-bound#configure-the-logical-object-bound).
Refer to [Restore a DocsFab Deployment](/docsfab/manage/restore-deployment#preserve-project-isolation).
Refer to [Back Up and Restore Local DocsFab State](/docsfab/manage/recover-local-state#recover-a-local-cli-installation).
Refer to [Restore a DocsFab Deployment](/docsfab/manage/restore-deployment#restore-without-reviving-old-authority).
Refer to [Configure the Recovery Object Bound in DocsFab](/docsfab/manage/configure-recovery-bound#watch-headroom-before-the-bound-refuses).