Development Setup

This guide covers everything you need to set up your development environment and submit contributions to NeMo Gym.

Quick Start

# Clone and set up development environment
git clone https://github.com/NVIDIA-NeMo/Gym.git
cd Gym
curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env
uv venv --python 3.13.14
source .venv/bin/activate
uv sync --extra dev
# Install pre-commit hooks (required for contributors)
pre-commit install

To work on servers that use Ray, or run the Ray-dependent unit tests, add --extra ray to uv sync.

Cloning over HTTPS needs no GitHub credentials. If you have an SSH key registered with GitHub, git clone git@github.com:NVIDIA-NeMo/Gym.git also works.

Optional component dependencies

Each component is an independent project with a committed pyproject.toml and uv.lock. Setup uses uv sync --locked --inexact; stale committed locks fail setup. Use uv 0.13.4 or newer, which includes the corrected certificate trust store. If an existing checkout uses an older uv release, run uv self update (or rerun the uv installer above) and confirm uv --version reports at least 0.13.4 before syncing. Older releases are rejected by the managed projects’ required-version setting; upgrading uv does not require regenerating the committed locks. Custom components and new scaffolds without a lock retain dependency resolution from their pyproject.toml or requirements.txt. Select optional extras per component in the environment configuration (paths are <server_type>/<component>):

component_extras:
  resources_servers/interactive_browser: [lexmount]
  responses_api_agents/haystack_agent: [example-tools]

Only the named component receives these extras. Setup preserves additional packages already installed in the environment. To remove them, recreate the component venv or run an exact uv sync --locked with the selected extras.

Parent OpenAI, Ray, provider, and telemetry requirements are verified after sync. When the committed lock cannot satisfy the parent’s installed requirements, setup resolves the component with those inherited requirements and rechecks the result. This compatibility path leaves the committed lock unchanged and reuses the compatible environment on later starts while the lock and parent contract match. Sync failures, inaccessible sources, and stale locks do not trigger this path.

UV_INDEX_URL, UV_EXTRA_INDEX_URL, UV_DEFAULT_INDEX, and UV_INDEX are supported: setup validates and exports the committed lock, then installs its pinned versions through the configured indexes. Mirror users must provide the locked distributions.

After changing root dependencies or resolver policy, update all managed locks:

python scripts/component_locks.py --update
python scripts/component_locks.py

Component exceptions to the root uv policy are recorded in scripts/component-lock-policy.json. The COMET-based wmt_translation and longmt_eval components use Transformers 4 and Hugging Face Hub below 1 to preserve COMET 2.2.7’s supported encoder and download APIs. These isolated components explicitly replace the shared Transformers 5 floor; other components retain it. CI checks every lock on Linux and macOS, and Nemo_CICD_Test requires both checks to succeed.

Development Commands

Run NeMo Gym Tests:

gym dev test                                   # Run all NeMo Gym core tests
gym env test --resources-server example_single_tool_call    # Test a single resources server
gym env test --all                             # Run all server tests (slow: one venv per server)

View Test Coverage:

coverage html                                  # Generate HTML coverage report

Configuration Debugging:

gym env resolve                                # Dump config as NeMo Gym sees it

CI/CD Requirements

All contributions must pass these automated checks:

Required Checks:

• Unit Tests: All existing tests must pass
• Build Docs: Documentation must build without errors
• Copyright Check: All files must have proper copyright headers
• DCO Sign-off: All commits must include Signed-off-by (git commit -s)
• Pre-commit Hooks: Code formatting and linting

Test Requirements:

• At least one test per server you contribute
• Tests must run using gym env test --resources-server your_server
• Use pytest for async testing patterns

CI Not Starting (/ok to test)

The mirrored Gym CICD and Fern workflows run automatically for verified commits from NVIDIA-NeMo members. If your commits are unverified, or you are an external contributor, a qualified NVIDIA-NeMo member must comment /ok to test <full commit SHA> on the PR before these workflows start. Use the full 40-character SHA (git rev-parse HEAD), not an abbreviation. After the head commit changes, the new full SHA needs a new comment. The same approval also triggers suggested reviewers from the # suggest: lines in .github/CODEOWNERS. See Continuous Integration in CONTRIBUTING.md.

Build Docs CI Failures

If the build-docs check fails:

1. Test documentation locally: Follow the Fern docs quickstart.

   • For local documentation previews, ensure Node.js 22+ is installed.
   • From the repository root, run make docs-login once to provision and authenticate Fern.
   • From the repository root, run make docs to generate the API reference and start the local preview.
   • After the initial generation, prose-only changes can use cd fern && npm run dev.

2. Common issues:

   • Missing docstrings in public functions
   • Broken markdown links in README or tutorials
   • Invalid MDX syntax

Contributing documentation (including external contributors)

You do not need Fern dashboard org membership to contribute docs.

1. Edit MDX under fern/versions/latest/pages/.
2. If you add a page that is not auto-discovered, update the bleeding-edge nav in fern/versions/main.yml. Folders mounted with title-source: frontmatter usually pick up new pages automatically; otherwise add a - page: entry under the right section. See the Fern docs README and the nemo-gym-docs skill.
3. Open a pull request from a fork or branch.
4. Rely on CI: fern check and the automatic Fern preview comment use the repository DOCS_FERN_TOKEN, including for fork PRs.

Optional local preview follows the Fern README. Full make docs / library autodoc (fern docs md generate) may require membership in the Fern nvidia organization if you hit HTTP 403: User does not belong to organization.

Requesting Fern nvidia org access (optional): open a GitHub issue on NVIDIA-NeMo/Gym. Suggested title: Fern nvidia org access request. Add the community-request label if you can, and say you need Fern dashboard membership for local autodoc (make docs / fern docs md generate). Maintainers triage the request.

NVIDIA employees: if you already have NVIDIA Slack, you can also ask in #fern for org access or docs help. Slack #fern is not required for external contributors.

Public repository content

NeMo Gym’s repository and documentation site are public. Do not include NVIDIA internal or confidential information. Before merging, authors and reviewers must check documentation, code comments, scripts, configs, example data, logs, and screenshots for information that is not approved for public release:

• Keep internal hostnames and URLs, private repository and issue links, employee filesystem paths, deployment identifiers, and non-public dataset or checkpoint details in approved private systems. Use portable paths, placeholders, or documented public resources in repository examples.
• Do not name individuals or publish personal handles or email addresses as contacts for credentials, access approvals, restricted artifacts, or internal support. Use a team-owned support route approved for public use.
• Give readers a public entry point, such as maintainer issues, for general setup questions. Keep internal access instructions in private systems; exchange credentials and restricted policy contents through approved private channels.
• Use placeholders or environment-variable references for credentials, and review copied output and screenshots for secrets or internal details.

Preserve public contributor attribution and required license notices. Document access restrictions accurately without exposing the restricted material.

Copyright Header Errors

Error: “Found files with missing copyright”

Solution: Add this header to all new Python files:

# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

DCO and Commit Signing

All NeMo Gym contributions require DCO sign-off (git commit -s). Cryptographic commit signing (SSH/GPG -S) is optional and not enforced by CI.

Quick Setup (required)

# Set your identity (use your GitHub email)
git config --global user.name "Your Name"
git config --global user.email "your@github-email.com"

Every commit must be signed off with -s:

git add .
git commit -s -m "Your commit message"
# The -s flag adds this line automatically:
# Signed-off-by: Your Name <your@email.com>

In VS Code / Cursor, you can set "git.alwaysSignOff": true so the IDE adds the trailer for you.

Troubleshooting DCO

Problem: “DCO sign-off missing”

If you have not pushed yet:

# Fix the last commit
git commit --amend -s --no-edit
# Fix multiple local commits
git rebase --signoff HEAD~3  # Adjust number as needed

If you already pushed to your PR branch:

Force-pushing is disallowed on branches in the upstream NVIDIA-NeMo/Gym repo. If your PR branch is in a fork and your fork allows force pushes, you can re-sign commits locally and push with --force-with-lease; otherwise, push the signed history to a new branch and update the PR.

# Option 1: Re-sign commits locally, then push to a new branch and update your PR
git rebase --signoff origin/main
git push origin HEAD:your-user/your-feature-dco-fix
# Option 2: Squash into one signed commit on a fresh branch
git checkout -b your-feature-dco-fix origin/main
git merge --squash your-feature-branch
git commit -s -m "Your commit message"
git push -u origin your-feature-dco-fix
# Option 3 (forks only): Re-sign and force-push to the same branch
git rebase --signoff origin/main
git push --force-with-lease

Tip: Always use git commit -s when creating commits to avoid this issue.

Optional: Cryptographic commit signing (SSH or GPG)

Not required for merge. If you want GitHub to show commits as “Verified”:

# SSH signing (recommended)
git config --global commit.gpgsign true
git config --global gpg.format ssh
git config --global user.signingkey ~/.ssh/id_ed25519.pub

Add the same public key in GitHub Settings → SSH and GPG keys as a Signing Key, then commit with -s as usual (add -S only if you want to force a signed commit from the CLI).

For GPG instead of SSH: generate a key, set user.signingkey to the key ID, leave gpg.format unset, and upload the public key under GitHub GPG keys.

Common Issues

Testing Issues

Problem: Tests fail locally but not in CI

• Check Python version (3.13.14+ required)
• Ensure all dependencies installed: uv sync --extra dev
• Run in clean environment

Problem: Async test failures

• Use pytest-asyncio for async tests
• Mark async tests with @pytest.mark.asyncio
• Ensure proper fixture cleanup

Pre-commit Hook Failures

Problem: Pre-commit hooks fail

# Fix common issues
pre-commit run --all-files        # Run all hooks manually
pre-commit autoupdate             # Update hook versions

Common fixes:

• ruff check --fix . for linting
• ruff format . for formatting
• Add copyright headers to new files

Use of AI and LLM Tools

We encourage contributors to use AI coding assistants (Cursor, Claude, Codex, OpenCode, and similar) where they genuinely help. However, AI assistance does not replace human understanding, judgment, and accountability.

Guiding principle: Prefer contributions where your review and ownership clearly outweigh the maintainer review burden. You are responsible for every line of code you submit, regardless of whether you or an AI tool wrote it.

The quality bar for PRs (focused diffs, real rollouts for env/agent work, CI-aligned local checks, substantive tests) lives in AGENTS.md so coding agents and humans share one source of truth. See also Agent Skills.

When We Request Changes

When a pull request or issue appears to be a low-effort, AI-generated submission, maintainers will request changes and explain what needs to improve. If the feedback goes unaddressed — or the contribution is clearly unreviewed bulk output with no path to a quality bar — it may be closed. Common indicators that prompt a change request include:

• Boilerplate or generic code that ignores project conventions
• PRs that do not pass CI or pre-commit checks
• Descriptions or comments that are clearly unreviewed LLM output
• Bulk “improvements” with no corresponding issue or discussion

This is not about policing tool usage. It is about maintaining the quality bar that the community and maintainers depend on.

Development Workflow

For a small, self-contained change, open a single PR:

1. Create a feature branch: git checkout -b <username>/<feature>
2. Make changes with tests
3. Run local checks: gym dev test && pre-commit run --all-files
4. Commit with signoff: git commit -s -m "Your message"
5. Push and open a PR: ensure all CI checks pass
6. Address review feedback and iterate

Stacked Pull Requests

For larger changes, NeMo Gym uses GitHub Stacked PRs: a chain of branches where each PR targets the branch below it, so each layer is reviewed as its own focused diff. See GitHub’s Stacked PRs overview for the concept and PR UI; the mechanics of the gh stack CLI in this repo follow.

Prerequisites

Install the GitHub CLI (gh) v2.0 or later, the gh stack extension, and the agent skill:

gh extension install github/gh-stack
gh skill install github/gh-stack gh-stack   # vendored under .agents/skills/gh-stack/; this installs it for your agent

gh auth login   # GitHub.com → HTTPS → "Login with a web browser"

A stack is created in the repo you push to, and that repo must have the preview enabled — push your branches to the remote that points at NVIDIA-NeMo/Gym. With multiple remotes, set the target once (or pass --remote <name> on each command) and enable conflict memory so commands do not prompt:

git config remote.pushDefault upstream   # the remote pointing at NVIDIA-NeMo/Gym
git config rerere.enabled true

Create a Stack

Build the stack bottom-up, one logical change per branch, where each branch depends on the one below it. Keep a contribution self-contained — land code together with its tests in a single PR — and stack genuinely dependent follow-ups (for example, documentation) on top.

# Bottom layer: the contribution (resources server, agent config, and its tests) as one branch.
gh stack init -p <user>/my-env contribution
git add resources_servers/my_env/
git commit -s -m "feat: add my_env environment and tests"
# Next layer: a follow-up that depends on the contribution above.
gh stack add docs
git add fern/versions/latest/pages/environment-tutorials/my-env.mdx
git commit -s -m "docs: document my_env"
# Push every branch and open a draft PR per layer (--auto titles them).
gh stack submit --auto
# Inspect the result (always --json; bare `view` opens a TUI).
gh stack view --json

submit sets each PR’s base to the branch below it and links them as a stack on GitHub. Run every gh stack command non-interactively: pass branch names to init/add/checkout, --auto to submit, and --json to view. Without these flags the commands prompt or open a TUI and hang.

Update a Stack

Make a change on the branch it belongs to, then rebase the layers above it:

gh stack down              # or: gh stack checkout <branch|pr-number>
git add <files> && git commit -s -m "..."
gh stack rebase --upstack  # replay the layers above onto the change
gh stack push

After a PR merges, sync to fast-forward the trunk and rebase the rest of the stack:

gh stack sync --prune      # fetch, rebase, push, drop merged branches

On a rebase conflict (exit code 3), resolve the listed files, git add them, then gh stack rebase --continue (or --abort). Merging is done from the GitHub PR UI; CLI merge is not supported yet.

Command Reference

The vendored skill (.agents/skills/gh-stack/SKILL.md) carries the full reference, exit codes, and non-interactive rules for agents.

See the gh-stack documentation for the full feature set.

Questions? Check existing issues or create a new one for guidance.