Development

View as Markdown

This page collects developer-facing reference material for working on RMS itself. For building and testing, see Getting Started; for adding hardware, see Adding Support for New Hardware.

Project structure

.
├── Cargo.toml # workspace manifest
├── Dockerfile # multi-stage build (builder + release)
├── docker-compose.yml # local Postgres for the persistence tests
├── Justfile # build / docker helpers
└── crates/ # all workspace crates
├── rackmanagementservice/ # core RMS service crate
│ ├── build.rs # tonic-build proto compilation
│ ├── src/ # CLI, runtime, domain, transport, and API modules
│ ├── tests/ # integration test binaries (see Testing)
│ └── benches/ # 100-10k node Criterion benchmarks
├── redfish_test_support/ # in-process Redfish simulator + fixtures
├── nvue_client/ # NVUE/NVOS REST client crate
├── redfish_client/ # RMS-focused Redfish client crate
└── rust_nvfwupd/ # embedded nvfwupd firmware-update crate + CLI

See Workspace Crates for the supporting crates and Architecture: Internal View for how the layers fit together.

Key dependencies

CratePurpose
tokioAsync runtime
tonic / prostgRPC server + protobuf
reqwestAsync HTTP client (Redfish, NVUE)
russh / russh-sftpAsync SSH + SFTP (switch firmware)
nvfwupdEmbedded firmware-update workflow library and CLI binary
serde / serde_jsonJSON serialization
thiserrorError type derivation
tracing / tracing-subscriberStructured logging (logfmt)
clapCLI argument parsing (--config)
figment / tomlTOML runtime configuration loading
tokio-utilCancellationToken for graceful shutdown
uuidJob ID generation
async-traitAsync methods in trait objects
secrecyZero-on-drop credential storage
sqlxAsync Postgres driver + compile-time-embedded migrations
chronoTimestamps in persistence types
mimallocGlobal allocator tuned for high async fan-out

Adding a persistence domain

The crates/rackmanagementservice/src/persistence/ module is per-domain by design: each new domain (e.g. inventory) gets its own trait file, two implementation files, and a migration. The shared infrastructure (pool builder, error type, migrations runner, CI Postgres service, docker-compose) is not touched.

To add a domain inventory:

  1. Domain types + trait - create crates/rackmanagementservice/src/persistence/inventory.rs with the entity types and a Send + Sync trait whose methods take and return only domain types.
  2. Migration - add crates/rackmanagementservice/src/persistence/postgres/migrations/000N_inventory.sql with the typed schema. Use sqlx migrate add --source crates/rackmanagementservice/src/persistence/postgres/migrations <name> to generate the timestamped file.
  3. Memory impl - create crates/rackmanagementservice/src/persistence/memory/inventory.rs with a struct holding RwLock-protected collections, mirroring the Postgres semantics so the same behavioral tests pass against either backend.
  4. Postgres impl - create crates/rackmanagementservice/src/persistence/postgres/inventory.rs. Reuse DatabaseError from super::error and the shared PgPool. Use sqlx::query_as (no ! macro - CI builds don’t need a live DB).
  5. Wire into Backends - add a pub inventory: Arc<dyn InventoryStore> field to the Backends struct in crates/rackmanagementservice/src/persistence/mod.rs.
  6. Tests - add scenarios to crates/rackmanagementservice/tests/persistence.rs as async fn<S: InventoryStore>(store: &S) and wrap each with the memory_test! and postgres_test! macros.

sqlx-cli defaults to looking for a top-level migrations/ directory. RMS’s live under crates/rackmanagementservice/src/persistence/postgres/, so always pass --source.

Adding hardware support

Adding a new node type or rack generation touches API identity, endpoint policy, rack routing, node construction, firmware policy, and tests. This has its own detailed guide: Adding Support for New Hardware.

Building standalone nvfwupd

The nvfwupd CLI can be built as a standalone, portable binary. A plain build inherits the host sysroot glibc baseline; the docker/nvfwupd-standalone/ Dockerfiles produce binaries with a glibc 2.17 baseline for broad portability.

$# Host-arch build (inherits host glibc baseline)
$cargo build --release -p nvfwupd
$
$# Portable x86_64 binary (glibc 2.17 baseline)
$docker buildx build --platform linux/amd64 --target artifact \
> --output type=local,dest=target/x86_64-unknown-linux-gnu/release \
> -f docker/nvfwupd-standalone/Dockerfile.x86_64-glibc217 .
$
$# Portable arm64 binary (glibc 2.17 baseline, Zig cross-linking from x86)
$docker buildx build --platform linux/amd64 --target artifact \
> --output type=local,dest=target/aarch64-unknown-linux-gnu/release \
> -f docker/nvfwupd-standalone/Dockerfile.arm64-glibc217 .
$
$# Cross-compile arm64 directly from an x86 host (inherits host aarch64 glibc)
$rustup target add aarch64-unknown-linux-gnu
$CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER=aarch64-linux-gnu-gcc \
> RUSTFLAGS="-C link-arg=-fuse-ld=bfd" \
> cargo build --release -p nvfwupd --target aarch64-unknown-linux-gnu
$
$# Verify a produced artifact does not require newer glibc symbols
$readelf --version-info target/x86_64-unknown-linux-gnu/release/nvfwupd \
> | grep -o 'GLIBC_[0-9]\+\.[0-9]\+' | sort -Vu | tail

The release container image also bundles nvfwupd; extract it with docker cp from a created container if you need the binary that shipped in a specific image.

Building multi-arch release images

The release image can be built for arm64 from an x86 host with Buildx plus QEMU/binfmt on the Docker daemon:

$docker buildx build --platform linux/arm64 --target release --load -t rms-release:arm64 \
> --build-arg VERGEN_GIT_SHA="$(git rev-parse HEAD)" \
> --build-arg VERGEN_GIT_DESCRIBE="$(git describe --tags --always --dirty)" \
> .

CI/CD

GitLab CI runs on every push and merge request, using the multi-stage Dockerfile to build, test, and package the service.

StageJobWhenWhat
buildbuild-baseEvery push/MRBuild the Rust project inside Docker
testfmt-checkEvery push/MRcargo fmt --all -- --check
testclippy-lintEvery push/MRcargo clippy --workspace --all-targets -- -D warnings
testdoc-checkEvery push/MRcargo doc --workspace --no-deps with RUSTDOCFLAGS="-D warnings"
testdeny-checkEvery push/MRcargo deny check bans sources
testrun-testsEvery push/MRcargo test --workspace --release
testcoverage-reportmain; MRs with ci-coverage in the commit messagecargo-llvm-cov coverage (Cobertura XML for MR diffs)
testcoverage-rackmanagementservicemain; MRs with ci-coveragepackage-level RMS coverage summary and threshold
testcoverage-nvfwupdmain; MRs with ci-coveragepackage-level nvfwupd coverage summary and threshold
testrun-benchmarksmain; MRs with ci-benchmarkscargo bench with Criterion HTML artifacts
simulation-teststratumsim-testsDefault branch or commit message containing ci-sim-testsRun replacement-grade GB200, GB300, and VR NVL72 ATP through the orchestrator-owned CI driver
nico-teststratumsim-nico-testsCommit message containing ci-sim-nico-testsRun NICo ATP through the orchestrator-owned CI driver
buildbuild-release-image-mrEvery MRBuild the minimal release image on native-arch runners
buildbuild-release-imagemain, release/*, tags, or ci-publishBuild amd64/arm64 release images from x86 via Buildx
securitytag-image-for-nspect-scanningmain, release/*, tags, or ci-publishTag image for NVIDIA security scanning
promotepush-to-registrymain, release/*, tags, or ci-publishPush the release image to nvcr.io/0837451325059433/rms-dev/rms-api

Merges to main are gated on fmt-check, clippy-lint, doc-check, deny-check, and run-tests all passing. Coverage jobs run on main and are opt-in for MRs via ci-coverage in the commit message. Release consumers should pull a versioned image rather than latest.

The job checks out pinned StratumSim and rms-sim-orchestrator sources and builds their runtime images. It then calls rms_ci/run.py from rms-sim-orchestrator. That driver owns network allocation, combined fixture generation, rack target rendering, Compose execution, failure logs, and cleanup. RMS CI retains source authentication, revision pins, and the RMS image build without duplicating orchestrator topology or lifecycle scripts. The NICo job follows the same boundary: GitLab checks out authenticated sources and prepares images, then calls rms_ci/nico.py. The orchestrator driver owns NICo runtime volumes, network adaptation, NICo ATP, diagnostics, and cleanup.