Skip to navigation
Manage DocsFabDeployment Recovery

Configure the Recovery Object Bound in DocsFab

Dev
View as Markdown

Configure the logical-object size bound and inspect recovery headroom before a backup refuses.

Prerequisites

Use an authorized deployment configuration change and the generated recovery limits mirror.

Steps

Use the command and checks below.

Configure the Logical-Object Bound

A verifier refuses a logical project object larger than recovery.logicalObjectMaxBytes. Declare this value in config/limits.yaml, then regenerate the config/limits.json recovery mirror. The TypeScript verifier and GitLab storage adapter read that mirror; DOCSFAB_LIMITS_FILE selects it explicitly. Export producers, recorded manifest bounds, and the adapter use that same selected document. They refuse at bound plus one with the Recover class, the coded name, observed and configured values, and one next action. They never clamp. Bundle manifests use the separate recovery.manifestMaxBytes bound. An unavailable configuration refuses the operation.

The logical-object default is 536,870,887 bytes, 25 bytes below 512 MiB. This removes the former 256 MiB heap margin. The verifier converts the complete object to a JavaScript string before parsing it. On qualified 64-bit Node, MAX_STRING_LENGTH is 536,870,888 UTF-16 code units; worst-case ASCII JSON needs one code unit per byte. The loader accepts values from 1 MiB up to one less than that ceiling.

Restore also needs memory for buffers, parsed objects, and canonical serialization. The allowance does not reserve that memory or establish near-limit restore capacity. Literal 512 MiB requires changing the whole-string representation throughout export, verification, serialization, and restore.

The pre-deploy and pre-rollback backup gate does not read that file only from the outgoing release. Rolling back onto a revision that predates config/limits.json would otherwise make the next gate fall back to 16 MiB and refuse. The gate resolves the bound from the newest eligible installed release that carries config/limits.json: the incoming staged release when DOCSFAB_BACKUP_TO names one, the current release, or previous. Before selection, each candidate must occupy the deployment release layout, name its exact Git revision, pass sealed release verification, provide a valid commit time, and pass the canonical closed-key limits parser. An invalid purported candidate refuses the operation. The gate falls back to the generated legacy bound only when no eligible candidate carries the file, and it names that fallback on stderr. An operator bypass of the backup hook is not the normal recovery path.

docsfab-status prints a Recovery bound section that names the effective validated bound, boundSource (configured or legacy-fallback), and the release path that supplied config/limits.json. Observed bytes and headroom remain unknown unless fresh verified manifest evidence is available through the existing recovery CLI or headroom artifact. Status does not infer them from an unsealed snapshot. A generic limits table row that only says config/limits.yaml does not identify which installed release the gate reads.

The same command prints a Recovery operator artefacts section. The producer list is the sealed Slack map and activation state. When ${DOCSFAB_STATUS_ROOT}/recovery-manifest.json is present, status fails if that manifest omits either host-operator-file object, so a restore would not silently skip reconstructing those files. When no manifest is present, status names the producer list and does not treat the omission as a permanent known gap.

Each bundle manifest records the bound its producer assumed, so a verifier knows what it was working to instead of guessing. A manifest sealed before that field existed assumed recovery.legacyLogicalObjectMaxBytes, which is 16 MiB. The effective bound is the smaller of the recorded value and the verifier’s own configured value, so a manifest cannot raise the limit it is checked against. A manifest that records more is clamped rather than refused, so lowering the configured value cannot retroactively make already-sealed bundles unreadable. A manifest that records less than recovery.minimumLogicalObjectMaxBytes is refused.

The configured value does not govern every bundle. Until the producer that seals a bundle records a bound, recovery.legacyLogicalObjectMaxBytes governs it, and raising recovery.logicalObjectMaxBytes changes nothing for that bundle. A refusal names whichever of the two actually binds, so the remediation you read is the one that can move the limit.

The bundle manifest, the object inventory, and the restore receipt keep their own bound, recovery.manifestMaxBytes. They grow with the project count rather than with job history, so they do not follow the logical object’s value.

Watch Headroom Before the Bound Refuses

The pre-deploy and pre-rollback backup gate refuses a logical object over the bound, which takes deploy and rollback away at the same moment. Reaching the bound must therefore never be a surprise.

Every backup run reports headroom, on the success path too, so the size, the bound, the share used, and the growth since the previous bundle are in the deploy transcript. recovery-cli headroom --manifest FILE [--previous FILE] produces the same record directly and exits 3 when a project is degraded. docsfab-status reprints the effective bound, boundSource, and the supplying release. Use the verified headroom artifact or the recovery CLI for observed object size and clearance evidence.

The report never fails a backup. It runs after the objects are already in the remote store, so it warns and continues rather than turning a backup that succeeded into one that reports failure. A bound that is genuinely exceeded has already failed the run much earlier, during verification.

Export producers (recovery-cli seal-project, store-project, and postgres-export) write a recovery-producer WARN line to stderr at the configured warnAt share of the selected recovery.logicalObjectMaxBytes. That warning is operator-visible on the success path and names the effective bound. Crossing the size bound is a Recover-class stop: raise the named limit or reduce the logical object, then retry the backup.

Backup staging also checks deploy.diskFreeMinBytes before it writes. Available bytes at that floor pass. One byte less refuses with deploy-disk-free-min-bytes-bound and tells you to free space on the recovery staging volume, then retry the backup. Catalog min is one logical object. A host can restate that floor in recovery.env as DOCSFAB_DEPLOY_DISK_FREE_MIN_BYTES. Install, systemd, the env -i backup transition, and backup start parse it fail-fast. A value below one logical object refuses. deploy.hostTimeoutStopSeconds is a sealed release invariant matching TimeoutStopSec on docsfab-host.service.

A project at or above recovery.headroomWarnRatio of the bound warns. At or above recovery.headroomDegradedRatio it is degraded. Both states name the remediation for the bound that actually governs. When the gate does refuse, its message names the object, its measured size, the bound, the share used, and the same remediation.

Growth is reported against the previous bundle, never assumed. A first bundle reports the growth as unknown rather than as zero.