DocsFab Recovery State Reference

Look up the durable state included in a deployment recovery bundle, its storage backend, and its authority boundaries. For procedures, follow Back Up and Restore Local State or Restore a Deployment.

The first pilot-host recovery drill has not run. The checked-in implementation and its local tests do not establish that docsfab.nvidia.com has a usable external backup. The latest deployed pilot run verified the physical base backup and then stopped while validating compensated forge outbox history in the logical export. No complete bundle was published. A fresh run after reviewed deployment must verify the corrected export against the source host.

Understand the Recovery Boundary

The recovery bundle combines independently verifiable physical and logical artifacts. The physical artifact is an encrypted PostgreSQL base backup. Each logical artifact is a sanitized, provider-neutral export for one project.

The following table describes how DocsFab treats each state class.

State Class	Recovery Treatment
Coordination journal and deployment-epoch history	Preserve ordered, digest-bound logical history for each project without the provider’s physical project identifiers. Keep the encrypted physical source image inert.
Host jobs, context threads and links, capsule history, reconciliation history, and forge event, delivery, and outbox history	Preserve sanitized, digest-bound service history for each project. Each host-job entry keeps a fixed-size packet reference instead of the packet body, so the logical object grows with retained metadata rather than with packet bytes.
Forge payload store (content-addressed request payloads under the project custody root)	Do not export and do not back up today. Forge outbox history keeps each payload digest and locator only, and the physical backup covers the PostgreSQL cluster only.
Product Reference Model, source threads, and PRD, SRD, and POR projections	Recover from the project’s authoritative Git repository. The recovery bundle does not replace Git as semantic authority.
Interrupted work	Record the prior attempt and lease identities as history, return the work to `ready`, and require fresh packet, capsule, attempt, lease, and fence identities.
Runtime bindings, live agents, claims, leases, and old capsule authority	Do not import as active authority. A new deployment epoch makes every old proof stale.
Slack route map and activation state	Preserve as digest-verified `host-operator-file` objects at `/etc/docsfab/slack-pilot-map.json` (`root:docsfab` mode `0640`) and `/etc/docsfab/activation-state.json` (`root:docsfab` mode `0440`), encrypted with the recovery encryption adapter before they enter the bundle. Older bundles hold them in plaintext and restore unchanged. Restore runs as the recovery identity, which cannot write `/etc/docsfab`, so it authenticates them and stages the manifest and the still-encrypted objects. A separate root step, `docsfab-restore --place-host-operator-files`, checks the staged manifest against the manifest digest and each object against that manifest, has the recovery identity decrypt them, and writes both paths or neither with that owner, group, and mode. It repairs owner, group, and mode when the digest already matches. Credentials under `/etc/docsfab/credentials` stay excluded. A bundle whose manifest omits them is not a complete restore of operator state.
Raw private prompts, general channel bodies, private persona or profile memory, worktrees, caches, and process logs	Exclude.
Unapproved large objects	Exclude. Include only objects approved for the configured recovery backend.

Forge outbox history preserves all six database-valid lifecycle states: pending, claimed, suppressed, compensation-pending, cancelled, and delivered. It retains claim identity digests, lifecycle timestamps, cancellation outcomes, delivery state, payload digests, and payload locators. The export marks pending, claimed, and compensation-pending records for fresh reconciliation. It rejects any lifecycle tuple that does not match the source database contract. Restoring this history does not reactivate a claim or fabricate delivery or cancellation evidence.

Keep the Logical Object Bounded

Every host-job history entry carries a packet reference, not a packet body. The exporter still validates the complete stored packet, then retains a fixed-size record of it with the following fields:

requestKey, the same identity as the history record id.
packetSha256, calculated from the complete validated stored packet.
packetBytes, the byte length of that same serialization.
packetFingerprintSha256, copied from the stored packet.
workflow and status.
retainedIn, which names where the referenced bytes are retained.

Nothing is deleted. The referenced bytes stay in the source cluster and travel in the same bundle inside the encrypted PostgreSQL image, which pg_verifybackup has already passed. Restore never reinstates a stored packet as active work in any shape: interrupted work requires a fresh packet, and logical import writes service history only to the immutable recovery ledger.

The bundle therefore stays lossless, but the logical object on its own does not. A consumer that reads only the logical object no longer sees packet bodies: that includes recovery-cli verify, recovery-cli inspect-project, recovery-cli restore --mode verify-only, and the restored deployment’s recovery_project_history ledger. Recovering a packet body after this change means starting the staged PostgreSQL data directory that docsfab-restore publishes, which is deliberately marked DOCSFAB_DO_NOT_START, and reading the source tables. No DocsFab command performs that step today. Treat the logical object as an index into the physical image rather than as a self-sufficient forensic record.

This keeps the logical object a function of retained metadata. Before the change, one project’s object grew by the size of every job packet, and the pilot’s export reached 94.6 % service history, of which 76.9 % was the inlined packet. A 10,000-job project measures 7,901,004 bytes as references against 23,857,230 bytes inlined, and the per-entry cost is now constant.

The export still does not retain every host-job row. Job history accumulates one row per host job, so the logical object can grow past the legacy 16 MiB bound even with packet references. The exporter keeps the newest recovery.serviceHistoryMaxHostJobs host-job rows, leaves every other history kind untouched, and writes a warning that names the catalogued limit, the observed count, and that older rows remain in the physical image. It does not refuse the backup. Restore of those clipped rows means reading the source tables from the staged PostgreSQL data directory, which no DocsFab command performs today.

Each history entry declares its own shape, through the packet’s kind. The logical object’s top-level schema stays the same closed 13-key set it has always had, because the storage adapter that publishes every bundle validates exactly those keys and is refreshed separately from the release. An object whose entries mix the two shapes is refused, and an unfamiliar recovery packet kind is refused by name rather than misread. DocsFab still verifies and restores objects in the original inline shape.

The copied packet fingerprint is source provenance. You cannot recompute it from the reference because the packet body is absent. The logical artifact independently recomputes recordSha256, sourceStateSha256, and projectSha256 over the retained projection. Logical restore writes this record only to immutable recovery history. It does not restore Slack effects, resume a live-agent session, or recreate an active host job.

The excluded-word screen still refuses credential-shaped values, private-key material, host paths, and private prompts. Ordinary documentation words are allowed only in already validated host-job packet sourceBranch and targetBranch fields, which exist only in the original inline shape. A packet reference carries no branch prose and receives no exception. The exception applies during closed host-job history parse and during the final project-export scan of parsed top-level host-job entries. Journal inputs, manifests, restore receipts, and nested look-alike paths receive no exception. Local fixtures prove this correction. They do not prove that a live backup or restore has succeeded.

The manifest binds the release revision, coordination and persistence schema versions, forge-event schema, pinned coordination-provider commit, service catalog fingerprint, source deployment and epoch, object inventory, artifact sizes and SHA-256 digests, the locked project-registry snapshot digest, storage locators, the logical-object bound the producer assumed, and explicit exclusions. Verify the expected manifest digest and every listed object before restoring state.

Use the Official Pilot Backend

The pilot uses a private DocsFab recovery project, project ID 485025. This is a storage adapter for the portable recovery contract, not a GitLab dependency in the recovery model.

The backend separates its artifact classes across the following stores:

Protected Git history stores sanitized manifests, logical project exports, and content-free terminal restore-completion receipts.
The project’s Generic Package Registry stores client-encrypted PostgreSQL backups and other approved large or private objects.
Git records digest-bound locators for registry objects. It does not contain their plaintext content or a credential-bearing download URL.

The protected sink and source adapters must upload the correct artifact class, read it back, and bind the observed digest to their receipt. Encrypt a physical backup before it leaves private recovery staging. Keep adapter credentials in the host credential store, not in a manifest, command argument, issue, repository, prompt, run record, or log.

The repository ships one reviewed authenticated-encryption adapter, an AES-256-GCM implementation that reads its key only from the protected host credential path. Bootstrap installs no adapter by default. You stage the adapter you choose and name it explicitly, and key custody stays outside the repository and outside the host. The encryption key must remain outside the recovery bundle, recovery Git history, Package Registry object, configuration values, and operator command arguments. The adapter streams plaintext before the authentication tag is checked, so restore revalidates the decrypted archive and re-runs pg_verifybackup.

Backup packages and local staging bundles expire after recovery.backupRetentionDays, 30 days. Git history in the recovery repository is not pruned yet: manifests, receipts, and logical exports (Git LFS objects and historical inline JSON) stay until an authorized history cleanup, which #1790 tracks. After each backup that passes verification and whose external copy is published and read back, the backup tool deletes package versions and local staging bundles older than the bound. It always keeps the newest verified backup and every backup published within the bound, and it deletes nothing after a failed or local-only backup. New manifests record expire-after-retention-days with the bound. Manifests from the earlier mode that never deleted backups record no bound and still verify and restore.

Separate Database Authority

The locked docsfab-recovery operating-system service uses 3 database identities. None of the following identities can assume a coordination-administration role:

docsfab-backup has LOGIN and REPLICATION, no role memberships, and no DDL or DML grants. The physical backup service uses this identity.
docsfab-recovery-export has LOGIN and can set only the non-owning, read-only docsfab_recovery_export role in the source cluster.
docsfab-recovery-import exists only in the separately provisioned clean destination cluster. It can set only the non-owning docsfab_recovery_import role there.

Before physical capture, the source must be a dedicated DocsFab cluster with exactly docsfab, postgres, template0, and template1. The cluster guard checks every database row, including non-connectable and template databases, and rejects extra databases, custom templates, unrecognized roles, or stored password verifiers. The export identity also opens postgres and template1 and rejects non-system schemas, relations, routines, types, extensions, large objects, foreign mappings, subscriptions, publications, event triggers, and database-role settings. template0 remains non-connectable and is bounded by its exact owner, flags, tablespace, and access-control row; its local catalog cannot be inspected by the online non-superuser check, so cluster provisioning and superuser custody remain part of the trusted deployment boundary.

Peer authentication grants docsfab-recovery-export exactly 3 local database routes: docsfab, postgres, and template1. Each route uses the same docsfab identity map and must precede any broader matching local rule. The bootstrap rejects a missing route, changed map, or shadowing earlier rule. It does not allow all, template0, another database, TCP, or trust.

The source and destination cannot be 2 databases in one PostgreSQL cluster. PostgreSQL role membership is cluster-wide, so that layout would expose source import authority. The pilot uses /var/run/postgresql for the source socket and /run/docsfab-recovery-destination for the clean destination socket. Both connections use exact peer mappings from the docsfab-recovery operating system identity and contain no password. The destination socket directory is root-owned and is not writable by the recovery service or human operators.

The source bootstrap removes memberships from the obsolete combined docsfab-recovery database login and makes it non-login. It also audits the source export and physical backup sessions. Clean-destination provisioning is a separate deployment action. It grants import authority only in that destination cluster.

Deploy and rollback hold /srv/docsfab/deployment.lock while an installed, narrow wrapper changes from an operator to the docsfab-recovery service. The wrapper preserves only the required read-only lock descriptor, clears the environment, and invokes the fixed backup command. Its exact sudo rule grants no shell, arbitrary command, or environment-setting authority. A failed transition or backup prevents deployment state from changing.

Interpret Current Status

The repository contains the provider-neutral artifact contracts and a pilot-host operator implementation in the internal deployment tree, which the public source does not include. Local tests exercise sanitization, digest verification, valid SCM branch metadata, all six valid forge outbox lifecycle states, two-project isolation, fresh epoch and execution identities, empty-destination staging, and stale-proof rejection.

A source-bound PostgreSQL 17 harness also runs a real streamed pg_basebackup, validates it with pg_verifybackup, performs a test-only authenticated encrypt/decrypt round trip, and verifies the extracted backup again. It then exercises the exact CLI DSN path, source-custody preflight, injected second-project rollback, clean-destination rejection, project isolation, a pending merge-reconciliation projection, stale-proof rejection, and completion authority. The test-only cipher is evidence for the protected command seam. It is not the shipped recovery adapter and it does not decide pilot-host key custody. This is local container evidence, not a backup or restore of the pilot-host VM.

The pilot is not operationally recoverable until an operator completes and records a live drill. That drill runs on the installed PostgreSQL major, which is 16 on the current pilot host, because a base backup taken from one major does not start on another. The drill includes the protected GitLab backend, an encrypted Package Registry copy, a clean destination, at least two isolated projects, project and swarm re-registration, semantic reconciliation, and a successful resumed work item.

The following pages provide related information:

Moved Topic Links

Earlier section bookmarks lead to the canonical topic below.

Refer to Configure the Recovery Object Bound in DocsFab.

Refer to Restore a DocsFab Deployment.

Refer to Back Up and Restore Local DocsFab State.