Contributing
This guide gets you from a fresh clone to a mergeable pull request. You need Python 3.12, 3.13, or 3.14 and Git; no API keys are required for the default development loop.
Set up a development environment
Install with all extras
Install uv first.
uv creates .venv; either activate it using the command for your shell, or
run Make through uv so python3 resolves to that environment:
Make is an additional dependency. Without Make, the equivalent commands are uv run python -m ruff check src tests,
uv run python -m pytest, and uv run python -m build instead.
If you prefer pip, make install runs the equivalent editable install:
The Makefile wraps the standard tools, so you can run the underlying commands directly if you prefer:
Running tests
make test runs the fast suite. Two pytest markers are excluded by default, so slow and credentialed tests never block the everyday loop:
Harbor is the open-source agent
evaluation framework used by Tier 3; the live marker exercises its in-process
backend.
The POSIX same-inode read-race test explicitly advances the modified file’s mtime: two same-size writes can otherwise share a filesystem clock tick.
The MCP-staging and legacy-summary restored-mtime regressions model an advanced ctime explicitly, so their source-change assertions do not depend on the host clock ticking.
The directory-snapshot race regression explicitly advances directory mtime when inserting its late entry, rather than assuming creation crosses a clock tick.
Git URL-rewrite tests point GIT_CONFIG_GLOBAL at their temporary configuration,
so caller-level Git isolation does not suppress the rewrite being tested.
The dev extra includes pytest-xdist, so you can parallelize any of these runs:
Check the HTML report in a browser
tests/reporting/browser_navigation_checks.py opens the HTML report in a headless browser and checks navigation, focus, themes, print view and layout at three screen widths. pytest doesn’t collect it, and it needs Playwright:
The script renders its own fixture reports, then writes screenshots and navigation-results.json to browser-checks/. To use an installed Chrome instead of Playwright’s Chromium, set SKILLEVALUATOR_TEST_BROWSER to the browser’s executable or pass --browser.
Documentation and starter changes
The terminal starter uses a public skill, task and input with real CLI output checks. For changes to it, run:
Choose a new output directory for each replay. To verify the released route,
also run the pinned command in the starter guide. Report the evaluator version,
platform and exact commands tested in your pull request. Keep generated reports
out of commits. Describe seeded failures and incomplete evidence honestly; do
not reuse historical scores as a fresh run or silently change expectations to
make a check green. A fresh-user review should ask someone unfamiliar with the
tool to explain the passing, regressed and incomplete outputs before showing
the walkthrough’s answers, and record what confused them in the pull request.
The focused test intentionally runs the checkout’s evaluator. When the
evaluator is the checkout’s own source, the runner checks the checkout scores
in examples/starter/expected.json instead of the pinned release’s scores. A
scoring change on main updates those checkout scores; the release scores and
the starter guide change only when the pinned release does. Review both results
before changing expectations.
Code style and dependencies
Ruff owns linting — make lint is the project’s lint gate. CI runs ruff check over the whole repository, so lint the files you touch even outside src and tests.
Linting is only the first gate. CI also runs an OSS source-boundary scan (python scripts/check_oss_boundary.py --root . --allowlist config/oss_boundary_allowlist.json), which fails closed if the source tree or built distributions contain private integrations. The packaging job repeats that scan over the wheel and sdist, installs the wheel into a clean environment, and smoke-tests the CLI — including skillevaluator models --help and a doctor run with placeholder nv_build credentials. Platform jobs round out the matrix: a macOS runner executes the Tier 3 local-mode, progress, and preflight test files, and a native-Windows runner asserts local mode fails closed, with doctor required to exit non-zero for every SKILLEVALUATOR_LOCAL_SANDBOX value.
Follow the existing Python typing and test patterns, keep changes scoped, and add regression coverage for behavior changes.
For dependency changes: update pyproject.toml, uv.lock, and THIRD_PARTY_NOTICES.md together — the notices file lists every direct dependency and its license, and uv.lock records the exact resolved set for the release.
Work on the docs
The documentation site is built with Fern from the MDX pages in docs/. You only need a local build when you are editing docs; publishing happens automatically from main.
Install the Fern CLI
Requires Node.js 22+ and npm 10+. fern/package-lock.json pins the CLI and every package
below it; npm ci reproduces that tree exactly, and the flags keep npm from running any
install-time scripts. This is the same install CI performs.
New pages must be registered under navigation in fern/docs.yml, or they will not appear on the site. The full authoring rules — MDX gotchas, link conventions, the page-adding checklist — live in docs/AGENTS.md.
Submit a change
Contributions flow through GitHub pull requests:
- Discuss substantial changes in an issue first, before you invest in an implementation.
- Keep each pull request focused. Use the pull request template and explain the user-visible behavior and the verification you performed.
- Add focused tests for behavioral changes.
- Update CHANGELOG.md when the change affects users.
- Keep provider credentials, customer data, and private benchmark material out of the repository. Tier 3 live evaluation reads provider and agent credentials from your environment, never from the repo (with NVIDIA Build, one
NVIDIA_API_KEYcovers both roles) — see Providers & Credentials.
Every commit needs a Developer Certificate of Origin sign-off — commit with git commit -s. Pull requests containing unsigned commits are not accepted. By contributing, you confirm you have the right to submit the work under the Apache License 2.0.
Project resources
If you use SkillEvaluator in academic or technical work, cite the methodology
paper listed under preferred-citation in CITATION.cff, and record the
SkillEvaluator version you used for reproducibility: