Contributing

This guide gets you from a fresh clone to a mergeable pull request. You need Python 3.12 or 3.13 and Git; no API keys are required for the default development loop.

Set up a development environment

$
git clone https://github.com/NVIDIA/SkillEvaluator.git
$
cd SkillEvaluator

$
uv sync --python 3.13 --all-extras

$
pip install -e '.[all,dev]'

$
make lint && make test && make build

The Makefile wraps the standard tools, so you can run the underlying commands directly if you prefer:

Running tests

make test runs the fast suite. Two pytest markers are excluded by default, so slow and credentialed tests never block the everyday loop:

Harbor is the open-source agent evaluation framework used by Tier 3; the live marker exercises its in-process backend.

The dev extra includes pytest-xdist, so you can parallelize any of these runs:

$
pytest -n auto

Code style and dependencies

Ruff owns linting — make lint is the project’s lint gate. CI runs ruff check over the whole repository, so lint the files you touch even outside src and tests.

Linting is only the first gate. CI also runs an OSS source-boundary scan (python scripts/check_oss_boundary.py --root . --allowlist config/oss_boundary_allowlist.json), which fails closed if the source tree or built distributions contain private integrations. The packaging job repeats that scan over the wheel and sdist, installs the wheel into a clean environment, and smoke-tests the CLI — including skillevaluator models --help and a doctor run with placeholder nv_build credentials. Platform jobs round out the matrix: a macOS runner executes the Tier 3 local-mode, progress, and preflight test files, and a native-Windows runner asserts local mode fails closed, with doctor required to exit non-zero for every SKILLEVALUATOR_LOCAL_SANDBOX value.

Follow the existing Python typing and test patterns, keep changes scoped, and add regression coverage for behavior changes.

For dependency changes: update pyproject.toml, uv.lock, and THIRD_PARTY_NOTICES.md together — the notices file lists every direct dependency and its license, and uv.lock records the exact resolved set for the release.

Work on the docs

The documentation site is built with Fern from the MDX pages in docs/. You only need a local build when you are editing docs; publishing happens automatically from main.

$
npm install -g fern-api

$
fern docs dev

$
fern check

New pages must be registered under navigation in fern/docs.yml, or they will not appear on the site. The full authoring rules — MDX gotchas, link conventions, the page-adding checklist — live in docs/AGENTS.md.

Submit a change

Contributions flow through GitHub pull requests:

1. Discuss substantial changes in an issue first, before you invest in an implementation.
2. Keep each pull request focused. Use the pull request template and explain the user-visible behavior and the verification you performed.
3. Add focused tests for behavioral changes.
4. Update CHANGELOG.md when the change affects users.
5. Keep provider credentials, customer data, and private benchmark material out of the repository. Tier 3 live evaluation reads provider and agent credentials from your environment, never from the repo (with NVIDIA Build, one NVIDIA_API_KEY covers both roles) — see Providers & Credentials.

Project resources

If you use SkillEvaluator in academic or technical work, please cite it:

1
cff-version: 1.2.0
2
message: "If you use SkillEvaluator in academic or technical work, please cite this software."
3
title: "SkillEvaluator"
4
type: software
5
version: "0.1.0"
6
authors:
7
  - name: "NVIDIA Corporation"
8
license: Apache-2.0
9
repository-code: "https://github.com/NVIDIA/SkillEvaluator"