Skip to navigation
ResourcesEvaluation

Documentation Source Coverage in DocsFab

Dev
View as Markdown

Use this retained evaluation record to distinguish deterministic fixture coverage from installed named-workflow qualification.

This page records the DocsFab source-coverage-v1 qualification matrix for five documentation configurations: plain Markdown, MkDocs, Docusaurus, Sphinx, and Fern. Each configuration is crossed with the named create-documentation, update-documentation, and review-documentation jobs. The matrix is evidence, not a quality threshold, release gate, or extra workflow entry point. For more information, refer to DocsFab Knowledge and Delivery Reference.

Run those jobs with the procedures in Run a Documentation Workflow in DocsFab. This page does not replace that guide.

What the Matrix Establishes

A deterministic fixture-only cell proves that the sealed corpus, collector, writer and validator receipts, preserved facts and routes, and named negative controls fail closed. It does not prove an installed docsfab run job.

An installed cell requires a real named-workflow run against a materialized candidate, with retained package, CLI, workflow, model, profile, and quality-pack identities, plus access and citation ledgers. Fixture-only, admission-only, and empty-model records cannot satisfy that cell.

A pending cell names the exact missing grant or toolchain. It is not a pass.

Collector Commands

From the repository root, collect the deterministic cases and the installed matrix identified by v1. This identifier versions the qualification matrix, not the DocsFab release:

node packages/runtime/scripts/source-coverage-qualification.mjs \
--mode deterministic \
--corpus packages/quality/benchmarks/runtime/evaluation/source-coverage-v1
node packages/runtime/scripts/source-coverage-qualification.mjs \
--mode installed --matrix v1 \
--corpus packages/quality/benchmarks/runtime/evaluation/source-coverage-v1 \
--output "${TMPDIR:-/tmp}/docsfab-source-coverage-v1"
git diff --check

Installed live execution still requires an absolute CLI outside this checkout, an absolute work root, and --acknowledge-installed-execution source-coverage-v1-installed-execution. That acknowledgment records operator intent. It does not mint model credentials, host authority, or a release.

Checked-in receipts live under the source-coverage-v1 qualification/ directory. They follow the private-v1 campaign pattern: a campaign index, retained report identities, and explicit pending or failed cells.

Choose a fresh, real directory for --output. The collector refuses a symlinked output directory and refuses to replace an existing report.json. These checks preserve an earlier report and prevent concurrent collectors from clobbering each other’s output.

Historical Snapshot From a Worktree Based on d5e74672

The following snapshot was collected on September 16, 2026, from a worktree based on origin/main d5e746729f629d6fcabb981640695fd7057907fb. The worktree contained the uncommitted collector changes that produced these receipts, so the commit is the base revision, not the exact collector tree. The receipts preserve this historical output with the source-coverage-v1 manifest digest cb68fd403c6f97a1f49e4acd69d0b404a811b1696178871fbbfd7f88aa4e1303. Publishing the qualification receipts resealed that corpus to 0b0a222e8a1d073b4d432dc7717707227b3307852472f7041bae6c5a9863341b. The evidence clarification and safety repairs resealed the current corpus to b61ca81723f9d48f850860bbc3b0fea7627bdd8863b7d06e5dbdfd266b226941. The retained reports keep the collection-time digest. The following table lists the result for each lane.

The following table lists the result and evidence status for each lane.

LaneResultEvidence status
15 primary cells, deterministic15 passed, 0 failedfixture-only
12 named negative controls12 rejected at the named conditionfixture-only
15 primary cells, installed named workflows15 pending, 0 passedpending
11 declared variants11 source assertions passedfixture-only
Variant writer outputs4 pending (root-readme-monorepo, mkdocs-automatic-nav, docusaurus-autogenerated, sphinx-myst)not installed

installedQualified is false. Installed named-workflow qualification remains incomplete.

Later Installed Attempt

On September 16, 2026, a packed CLI outside the checkout started installed qualification from MR revision 4f2536d9 with the required execution acknowledgment. The first cell, plain-markdown-create, failed closed with duplicate-node-identity: Duplicate or empty anchor in README.md at line 1. No installed report was written, and the other 14 cells did not run. Refer to the issue #839 attempt record.

This attempt does not change the historical pending receipt. It establishes a newer limitation: zero installed cells are qualified, the first cell has a known failure, and the remaining cells are unrun.

Primary Cells

The primary matrix is five configurations times CREATE, multi-page UPDATE, and comprehensive REVIEW, as the following table shows.

ConfigurationCREATEUPDATEREVIEW
Plain MarkdownDeterministic passed. Installed pending.Deterministic passed. Installed pending.Deterministic passed. Installed pending.
MkDocsDeterministic passed. Installed pending.Deterministic passed. Installed pending.Deterministic passed. Installed pending.
DocusaurusDeterministic passed. Installed pending.Deterministic passed. Installed pending.Deterministic passed. Installed pending.
SphinxDeterministic passed. Installed pending.Deterministic passed. Installed pending.Deterministic passed. Installed pending.
FernDeterministic passed. Installed pending.Deterministic passed. Installed pending.Deterministic passed. Installed pending.

The historical installed receipt records this pending reason on every primary cell: no granted installed CLI outside this checkout, work root, or --acknowledge-installed-execution was supplied. Live named workflows were not invoked. No model credentials were present in the collector environment. The later packed CLI attempt failed as described in Later Installed Attempt.

Declared Variants

Variants are extra materializable source profiles. They are not additional primary journey cells. The following table lists the variants.

The following table lists the declared variants.

VariantConfigurationFixture assertionsWriter outputsLimitation
root-readme-monorepoplain MarkdownPassedPending installed writerNone declared. Root README and nested package pages are the input shape.
mkdocs-automatic-navMkDocsPassedPending installed writerAutomatic navigation is an input shape only; it does not claim parser or generator execution.
docusaurus-autogeneratedDocusaurusPassedPending installed writerThe TypeScript configuration is inert and is not evaluated.
sphinx-mystSphinxPassedPending installed writerMyST syntax is fixture evidence and no Sphinx build is run.
malformed-mixednonePassedNot applicableThis negative fixture makes no successful discovery, route, navigation, or build claim.

Historical Evidence Input Families

In the historical snapshot, these lanes prove that sealed corpus bytes exist. HTML, PDF, and DOCX also have registered host extractors. None of the families is an installed named-workflow read-cite cell. The following table lists each family.

The following table lists the historical evidence input families.

FamilyCorpus assertionsHost extractorInstalled read-cite
Code, tests, and JSON settingsPassedNone registeredPending
OpenAPI and JSON SchemaPassedNone registeredPending
NotebookPassedNone registeredPending
HTML exportPasseddocsfab-html-text (text/html)Pending
PDFPasseddocsfab-pdf-text (application/pdf)Pending
DOCXPasseddocsfab-docx-block (WordprocessingML)Pending

Current Supplier Update

After the historical snapshot, structured repository source support from #836 landed on main at b23825bc. A new collection against current main reports docsfab-json-pointer for the OpenAPI and JSON Schema lane and docsfab-notebook-text for the notebook lane. The Python files in the mixed code, tests, and settings lane still have no registered extractor.

This supplier update does not change the retained receipts or qualify an installed cell. A new installed run must still produce the required source access, citation, writer, validator, and reader-fact evidence.

Supplier Status

Source-coverage suppliers that were present in the historical worktree do not fill an installed cell. The following table lists their historical status.

The following table lists the historical status of each supplier.

SupplierHistorical status
Portable corpus and C2 collectorLanded. Deterministic 15/15 and 12 negative controls passed.
Shared source evidence and tracked HTML workflow (#835)Landed. decodeSourceEvidenceFinal and the registered HTML extractor are available to installed collection.
Structured repository sources and notebook locators (#836)Not present. Notebook, OpenAPI, code, and configuration lanes have no dedicated extractor in this snapshot.
PDF adapter (#837)Landed and registered. Composite-font ToUnicode detection remains the documented PDF limitation from #795.
DOCX adapter (#838)Landed and registered.
Installed named-workflow matrixIncomplete. The retained receipt is pending; the later packed CLI attempt failed on cell 1/15 before a report was written.
Native MkDocs, Docusaurus, Sphinx, or Fern site buildsNot claimed. Bounded adapters do not execute those toolchains.

How to Read a Cell

Use this order when you interpret a later receipt:

  1. Confirm the corpus id source-coverage-v1 and the recorded manifest digest.
  2. Confirm the repository identity. In the historical receipts, testedRevision is the observed worktree HEAD and is only a base revision. A new receipt has exact revision attribution only when testedRevision is non-null and testedWorktreeDirty is false.
  3. Treat evidenceStatus: fixture-only as collector proof only.
  4. Treat status: pending as an incomplete installed claim.
  5. Treat installedQualified: true only when every primary cell has status: passed and evidenceStatus: installed.

Do not promote this matrix to a release record, quality score, or extra docsfab entry point.

See the following related topics:

Source Acquisition Proof

Offline doctor does not acquire sources and does not call a model. A context-v1-source-acquisition collection distinguishes deterministic fixture producers from live provider evidence collected through the installed CLI. Fixture results cannot stand in for an installed proof. Today that installed-CLI coverage is partial: only the local provider’s run and do cells (PDF, DOCX) actually collect live evidence through the installed CLI; the local provider’s HTML live-agent cell is not invoked (the installed live-agent requires an attached interactive terminal, which this collector does not allocate) and stays pending, as do the GitLab, HTTPS, and Brave live cells, in the collection’s own report until their providers are wired (#1062 N7). /sources search remains search-unavailable until a live searcher is wired; GitLab discovery beyond the first page uses the shipped sources commands, not search.