CI Regression Gate | NVIDIA NeMo Evaluator

Block merge requests that cause evaluation regressions.

GitLab CI

Include the eval template in your .gitlab-ci.yml:

1 include:
2   - local: deploy/gitlab-ci-eval.yml

Or add the stages directly:

1 stages:
2   - eval
3   - regression
4 
5 eval:candidate:
6   stage: eval
7   image: nemo-evaluator:latest
8   script:
9     - nel eval run --bench gsm8k --repeats 2 --max-problems 50 -o results/candidate --no-progress
10   artifacts:
11     paths: [results/candidate/]
12   rules:
13     - if: $CI_MERGE_REQUEST_IID
14 
15 eval:baseline:
16   stage: eval
17   image: nemo-evaluator:latest
18   script:
19     - git checkout $CI_MERGE_REQUEST_TARGET_BRANCH_NAME
20     - pip install -e ".[scoring]"
21     - nel eval run --bench gsm8k --repeats 2 --max-problems 50 -o results/baseline --no-progress
22   artifacts:
23     paths: [results/baseline/]
24   rules:
25     - if: $CI_MERGE_REQUEST_IID
26 
27 regression:check:
28   stage: regression
29   image: nemo-evaluator:latest
30   needs: [eval:candidate, eval:baseline]
31   script:
32     - nel compare results/baseline/eval-*.json results/candidate/eval-*.json --max-drop 0.05 --strict
33   artifacts:
34     paths: [results/regression.json]
35   rules:
36     - if: $CI_MERGE_REQUEST_IID

How it works

Statistical significance

With scipy installed (pip install nemo-evaluator[stats]), nel compare includes McNemar significance testing, effect size confidence intervals, and power analysis. This distinguishes meaningful regressions from benchmark noise. See Comparing Evaluation Runs for details on interpreting the statistical output.

Threshold tuning

Scenario	Threshold	Repeats	Max problems
Quick smoke test	0.10	1	50
Standard gate	0.05	2	100
High-confidence	0.03	4	full dataset

Higher repeats reduce noise in pass@k estimation. More problems reduce sampling variance. P-values require at least 2 samples per run.

GitHub Actions

1 jobs:
2   eval-baseline:
3     runs-on: ubuntu-latest
4     steps:
5       - uses: actions/checkout@v4
6         with: { ref: main }
7       - run: pip install -e ".[scoring]"
8       - run: nel eval run --bench gsm8k --repeats 2 --max-problems 50 -o results/baseline --no-progress
9       - uses: actions/upload-artifact@v4
10         with: { name: baseline, path: results/baseline/ }
11 
12   eval-candidate:
13     runs-on: ubuntu-latest
14     steps:
15       - uses: actions/checkout@v4
16       - run: pip install -e ".[scoring]"
17       - run: nel eval run --bench gsm8k --repeats 2 --max-problems 50 -o results/candidate --no-progress
18       - uses: actions/upload-artifact@v4
19         with: { name: candidate, path: results/candidate/ }
20 
21   regression:
22     needs: [eval-baseline, eval-candidate]
23     runs-on: ubuntu-latest
24     steps:
25       - uses: actions/download-artifact@v4
26       - run: pip install -e ".[scoring]"
27       - run: nel compare baseline/eval-*.json candidate/eval-*.json --strict