CI Regression Gate | NVIDIA NeMo Evaluator

Block merge requests that cause evaluation regressions.

GitLab CI

Include the eval template in your .gitlab-ci.yml:

include:
  - local: deploy/gitlab-ci-eval.yml

Or add the stages directly:

stages:
  - eval
  - regression
eval:candidate:
  stage: eval
  image: nemo-evaluator:latest
  script:
    - nel eval run --bench gsm8k --repeats 2 --max-problems 50 -o results/candidate --no-progress
  artifacts:
    paths: [results/candidate/]
  rules:
    - if: $CI_MERGE_REQUEST_IID
eval:baseline:
  stage: eval
  image: nemo-evaluator:latest
  script:
    - git checkout $CI_MERGE_REQUEST_TARGET_BRANCH_NAME
    - pip install -e ".[scoring]"
    - nel eval run --bench gsm8k --repeats 2 --max-problems 50 -o results/baseline --no-progress
  artifacts:
    paths: [results/baseline/]
  rules:
    - if: $CI_MERGE_REQUEST_IID
regression:check:
  stage: regression
  image: nemo-evaluator:latest
  needs: [eval:candidate, eval:baseline]
  script:
    - nel compare results/baseline/eval-*.json results/candidate/eval-*.json --max-drop 0.05 --strict
  artifacts:
    paths: [results/regression.json]
  rules:
    - if: $CI_MERGE_REQUEST_IID

How it works

Statistical significance

With scipy installed (pip install nemo-evaluator[stats]), nel compare includes McNemar significance testing, effect size confidence intervals, and power analysis. This distinguishes meaningful regressions from benchmark noise. See Comparing Evaluation Runs for details on interpreting the statistical output.

Threshold tuning

Scenario	Threshold	Repeats	Max problems
Quick smoke test	0.10	1	50
Standard gate	0.05	2	100
High-confidence	0.03	4	full dataset

Higher repeats reduce noise in pass@k estimation. More problems reduce sampling variance. P-values require at least 2 samples per run.

GitHub Actions

jobs:
  eval-baseline:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { ref: main }
      - run: pip install -e ".[scoring]"
      - run: nel eval run --bench gsm8k --repeats 2 --max-problems 50 -o results/baseline --no-progress
      - uses: actions/upload-artifact@v4
        with: { name: baseline, path: results/baseline/ }
  eval-candidate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -e ".[scoring]"
      - run: nel eval run --bench gsm8k --repeats 2 --max-problems 50 -o results/candidate --no-progress
      - uses: actions/upload-artifact@v4
        with: { name: candidate, path: results/candidate/ }
  regression:
    needs: [eval-baseline, eval-candidate]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/download-artifact@v4
      - run: pip install -e ".[scoring]"
      - run: nel compare baseline/eval-*.json candidate/eval-*.json --strict