Testing and Documentation

This page describes the validation and documentation checks expected for repository changes.

Rust and Python examples, plugin test fixtures, and benchmark tools are workspace members. Dependency versions are locked in the repository-root Cargo.lock and uv.lock; npm workspaces use the root package-lock.json. Run Python example commands with uv run --package <example-package> to select the example while sharing the root environment.

Quick Start Contracts

The quick starts store commands and expected output in their MDX pages. Sample files are committed beside the pages and displayed with Fern’s Code component. The checker reads the referenced files, so displayed and tested content share the same source. From the repository root, validate contracts and run the checker unit tests. Install jq first; a missing executable fails these checks:

just check-quickstarts

The existing CI Check / Run job runs this command without inference credentials. Every guide in docs/getting-started/quick-start/, except index.mdx, must have a contract. Missing contracts and an empty guide directory fail validation.

These lightweight checks run whenever the CI Check / Run job runs. Changes to shared dependencies or CI tooling can affect validation without changing a guide. Full Quick Start execution remains a separate manually triggered workflow.

The target explicitly selects the contract-checker and jq fixture tests. It does not discover execution tests or run shell/Python code extracted from the guides. Document execution and its regression tests belong to the separate manual Quick Start execution workflow.

The committed scripts/quickstarts/schema.json defines scenario and block metadata, including field types, required fields, and permitted values. Unknown fields fail validation. The schema: 1 field versions this testing format, not the Relay release; ordinary Relay version bumps do not change it. The Python checker additionally validates relationships such as referenced files, command IDs, shell variants, and lifecycle assertions. It reports schema errors together and continues checking other guides after a page fails. Invalid metadata can prevent further checks within that page.

The JSON metadata is embedded in each guide; commands, expected output, and referenced samples come from the displayed documentation. Review assertions in the same PR as a guide change. Static validation catches broken structure and references, but cannot establish that assertions fully describe the prose. Execution tests compare actual results with those assertions; consistency with the guide’s intent still requires review.

After installing the dev dependencies with the command above, use uv run --no-sync python scripts/check-quickstarts.py extract <page> --shell bash to inspect its commands. Replace <page> with the guide’s MDX path.

The preflight mode rejects stale artifacts. The verify mode checks expected output and artifacts after a successful run. These modes do not execute commands, provision credentials, or test operating-system compatibility.

Review documentation in the same PR as a version bump, including early development bumps. Ask a coding agent to review affected installation instructions and examples, then have a maintainer review its proposed changes. Choose documentation versions according to the intended audience and available packages; contract checks do not require them to match the Cargo workspace version. The documentation maintenance process is described in RELEASING.md.

The unit tests also exercise the four agent-runtime guides’ copyable JSON verification expressions against complete and invalid fixtures. They use the installed jq executable.

These are document, checker, and JSON-expression tests, not quickstart execution tests. A passing check does not establish that packages have been published, that a provider accepts the example, or that the guide runs on every supported system. Live execution and cross-platform quickstart validation remain separate work.

Testing Rule

Run tests for every language surface whose observable behavior can change. Start with the smallest focused test that proves the change, then run the canonical suite for each directly affected surface before handoff. A Rust implementation change does not automatically require every binding suite. Add a binding suite when the shared change affects that binding’s public or observable contract.

Run the affected targets directly:

just test-rust
# Run Rust tests with code coverage reporting, requires installing cargo-llvm-cov with `cargo install cargo-llvm-cov`
just ci=true test-rust
just test-python
just test-go
just test-node
just test-openclaw

The Rust-backed test recipes use cargo-nextest; install it with cargo install cargo-nextest --version 0.9.133 --locked before running them.

The Rust, Python, and Go recipes build the native and worker dynamic-plugin test fixtures once before starting their test runners. Before a focused raw test that uses those fixtures, prepare them explicitly:

just build-test-plugin-fixtures

Keep normal hermetic test cases below five seconds. Do not compile packages or fixtures inside a test case, keep real-time synchronization guards at five seconds or less, and use virtual time when validating timer behavior. Package builds and opt-in live end-to-end checks remain separate validation steps with their own runtime budgets.

Run integration, documentation, packaging, FFI, generated-output, or build-only checks only when those surfaces change or the relevant test does not build them. Documentation, comments, mechanical metadata, and other reversible low-impact changes do not require language suites or tests that merely mirror the implementation. CI remains the broader cross-platform backstop.

Use the matching build recipes when you need explicit build-only passes:

just build-rust
just build-python
just build-go
just build-node

Common Commands

These commands cover the most common language-specific validation loops.

Rust

Run the Rust validation loop when a change touches the core runtime or Rust-facing API surface.

just test-rust

Python

Run the Python validation loop when a change touches the wrapper package, tests, or docs tooling.

uv sync
just build-test-plugin-fixtures
uv run pytest

Node.js

Run the Node.js validation loop when a change touches the NAPI binding or JavaScript package surface. Run the OpenClaw target when Node changes can affect the OpenClaw plugin or when touching integrations/openclaw.

npm install --ignore-scripts
npm test --workspace=nemo-relay-node
just test-openclaw

Documentation Checklist

If your change affects public behavior, bindings, examples, or workspace structure, update the corresponding docs in the same branch.

Before opening a PR, confirm:

1. README.md still matches the repo structure
2. Relevant reference docs are updated for public API changes.
3. Relevant package or crate READMEs are updated when needed.
4. Examples and snippets stay aligned with supported bindings.
5. Docs build cleanly.

Docs Verification

Use just docs to validate the Fern documentation site after relevant docs changes. It regenerates the ignored Python, Node.js, and Rust API reference pages before checking links and navigation. It also compiles and runs the Rust examples that share source files with the guides. just docs-linkcheck is the compatibility entry point for a link-focused run; the recipes currently perform the same validation, so do not run both.

just docs

To check only the shared Rust examples, run just docs-examples.

Fern Publishing

NeMo Relay uses a dual-branch Fern publishing model. Authors edit documentation on main in docs/ and fern/. The generated Fern content on docs-website is managed by CI and must not be edited by hand. The branch root .gitignore, README.md, and .github/workflows/publish-fern-docs.yml are branch-local maintenance files and can be updated directly on docs-website.

The Fern workflow uses the FERN_TOKEN GitHub environment secret from the fern environment. Pull-request branches under pull-request/** get Fern previews when docs change, and merged pull requests clean up their preview deployment. Pushes to main sync the generated fern/pages-dev/ layout to docs-website and publish the dev docs. Raw SemVer tags such as 0.1.0, 0.1.0-beta.1, and 0.1.0-rc.1 create public docs versions displayed with a leading v. Prerelease indicators are stripped from the public docs version path, so 0.1.0-beta.1, 0.1.0-rc.1, and 0.1.0 all target v0.1.0. Stable tags use availability: stable and update the default Latest version. Beta and release-candidate tags use availability: beta, replace the same base version snapshot, and do not update Latest. Alpha tags are not published.

The branch sync and version snapshot logic lives in scripts/docs/sync_fern_docs_branch.py.

Licensing and Headers

All source files must include SPDX headers and remain under Apache 2.0 expectations. Reviewers check this during normal review even when hooks do not enforce it automatically.