CI/CD#
NeMo RL uses GitHub Actions for continuous integration, testing, and release automation. The CI pipeline implements a tiered testing system that balances thoroughness with resource efficiency.
Test Levels#
Tests are organized into levels of increasing scope and cost:
Level |
What runs |
When |
|---|---|---|
docs |
Lint + Sphinx docs build only; no container, no GPU |
|
L0 |
CPU config checks + unit tests (one parallel GPU job per |
|
L1 |
Unit tests + functional tests (GPU) |
|
L2 |
Same as L1 today (plan to add convergence tests) |
|
Lfast |
Fast unit + functional tests, reuses pre-built main container (skips build); PRs to a release branch (e.g. |
|
Defaults:
PRs do not run tests unless a CI label is applied.
Pushes to
mainand merge-group events force L1.Nightly scheduled runs (09:00 UTC) run the full suite.
Doc-only changes are auto-detected and skip unnecessary tests.
When Lfast missed a failure#
CI:Lfast runs a subset of CI:L1. If L1 catches a failure that Lfast missed, promote the test that caught it into Lfast so the next PR sees it earlier:
Unit test: narrow or remove the entry that excludes it in
tests/unit/excluded_unit_tests.sh. Tests underresearch/*/tests/unitnever run in Lfast.Functional test: change its
run_test <cmd>line intests/functional/L1_Functional_Tests_*.shtorun_test fast <cmd>. Lfast runs only the H100 scripts, so a test in anL1_Functional_Tests_GB200_*.shscript, or a failure that reproduces only on GB200, cannot be promoted this way.
Triggering CI on Pull Requests#
The CI label is set for you. On PRs to
main, theSet CI labelworkflow picksCI:docs,CI:LfastorCI:L1from the changed files (rules inskills/cicd/SKILL.md). Only repo maintainers can change it, for example toCI:L1for a large refactor or broad changes to core code; ask one if the label is wrong.Comment
/ok to test <commit-sha>— a bot will acknowledge with a thumbs-up and start CI.If you are an external contributor, you will need an internal NVIDIA developer to comment this on your PR to trigger CI.
The
Skip CICDlabel bypasses tests entirely (except onmain/merge-group). Only repo maintainers can set it.
Required Checks#
All PRs must pass these checks before merging:
Lint: ruff + pyrefly via pre-commit
Branch freshness: PR branch must be at most 10 commits behind the base branch
Semantic PR title: must follow conventional commit format
DCO sign-off: all commits must be signed with
--signoff(see CONTRIBUTING.md)Secrets detection: scans for accidentally committed secrets
Submodule validation: Automodel submodule must be fast-forwarded from the base branch
CI Pipeline Architecture#
The main pipeline (cicd-main.yml) runs through these stages:
Pre-flight: determines test level from PR labels, changed files, and event type
Container build: Docker image built on GPU runners (skipped for
Lfastor when a pre-builtimage_tagis provided viaworkflow_dispatch)Tests: unit and functional tests run in containers on GPU runners using the custom
test-templateaction. Doctests no longer run in CI; run them locally withtests/docs/Docs_Tests.sh(see documentation). The CPU config checks (test_recipes_and_test_suites.py,test_config_validation.py) run on a GitHub-hosted CPU runner after pre-flight and, on PR runs, lint and the test approval queue pass, alongside the container build when there is one. No test job waits for them; the QA gate requires them, and on a PR run a failure cancels the rest of the run.Coverage: aggregated from unit-tests and e2e; uploaded to Codecov
QA Gate: aggregates all job results into a single pass/fail status
Re-running a Failed Run#
To retry the same commit, re-run only the failed jobs (Re-run jobs → Re-run failed jobs, or gh run rerun <run-id> --failed). This also re-runs cancelled jobs, so it recovers a run that the CPU config checks cancelled. Re-run all jobs repeats every GPU job that already passed. See CONTRIBUTING.md.
Nightly Runs#
Full test suite runs daily at 09:00 UTC on
main. Failures send Slack alerts.Nightly docs are published at 10:00 UTC to a separate “nightly” version (does not overwrite stable “latest” docs).
Release Process#
All release workflows are manual (workflow_dispatch) with dry-run defaults:
Workflow |
Purpose |
|---|---|
|
Create release branch and version bump |
|
Build wheel, create GitHub release, generate changelog |
|
Publish docs to S3 + Akamai CDN (versioned and/or “latest”) |
|
Auto-publish to TestPyPI on main/release pushes (dry-run by default) |
Infrastructure#
VM health checks (
healthcheck_vms.yml): daily GPU health checks (07:00 UTC) on self-hosted runners. Auto-reboots degraded VMs and alerts via Slack on persistent failures.Merge queue retry (
merge-queue-retry.yml): auto-retries PRs dequeued due to CI timeout (max 3 retries before alerting).Stale cleanup (
close-inactive-issue-pr.yml): daily auto-close of inactive issues and PRs.Cherry-pick (
cherry-pick-release-commit.yml): auto-creates cherry-pick PRs from release branches back to main.Community bot (
community-bot.yml): syncs issues and comments to a GitHub Project board for tracking.
Workflow Reference#
Workflow |
Trigger |
Purpose |
|---|---|---|
|
push, PR, schedule, dispatch |
Main CI pipeline |
|
push main/r** |
Wheel build + TestPyPI |
|
dispatch |
Full release |
|
dispatch |
Code freeze |
|
dispatch, callable |
Publish docs to S3/CDN |
|
schedule (10:00 UTC) |
Nightly docs publish |
|
PR |
Secrets scanning |
|
PR |
PR title validation |
|
PR |
Auto-label by file path |
|
PR (dependency paths) |
uv.lock freshness vs submodule pyprojects |
|
schedule (07:00 UTC), dispatch |
GPU runner health |
|
PR |
Submodule validation |
|
PR dequeued (timeout) |
Auto-retry merge queue |
|
push main |
Release cherry-picks |
|
schedule (01:30 UTC) |
Stale issue/PR cleanup |
|
issues, comments |
Project board sync |
|
PR |
Post submodule check results |