nemo_gym.eval.comparison.suite.coverage

Validate recorded and planned task coverage for suite comparisons.

Module Contents

Functions

Name	Description
`_check_paired_coverage`	-
`_task_indices`	Keep task indices useful for locating missing inputs without overwhelming reports.
`check_collection_coverage`	Return collection inconsistencies separately from each source’s own evidence.
`recorded_task_sets`	Read identities already recorded by aggregation; never derive repeat scores.
`run_coverage`	Describe a source’s own recorded and planned evidence independently of other runs.

API

nemo_gym.eval.comparison.suite.coverage._check_paired_coverage(
    baseline: nemo_gym.eval.comparison.suite.loading.LoadedSuiteRun,
    candidate: nemo_gym.eval.comparison.suite.loading.LoadedSuiteRun,
    baseline_coverage: nemo_gym.eval.comparison.suite.schema.SuiteCoverage,
    candidate_coverage: nemo_gym.eval.comparison.suite.schema.SuiteCoverage
) -> None

nemo_gym.eval.comparison.suite.coverage._task_indices(
    tasks: set[int]
) -> str

Keep task indices useful for locating missing inputs without overwhelming reports.

nemo_gym.eval.comparison.suite.coverage.check_collection_coverage(
    baseline: tuple[nemo_gym.eval.comparison.suite.loading.LoadedSuiteRun, ...],
    candidate: tuple[nemo_gym.eval.comparison.suite.loading.LoadedSuiteRun, ...],
    baseline_coverage: list[nemo_gym.eval.comparison.suite.schema.SuiteCoverage],
    candidate_coverage: list[nemo_gym.eval.comparison.suite.schema.SuiteCoverage]
) -> list[str]

Return collection inconsistencies separately from each source’s own evidence.

nemo_gym.eval.comparison.suite.coverage.recorded_task_sets(
    run: nemo_gym.eval.comparison.suite.loading.LoadedSuiteRun
) -> tuple[dict[int, frozenset[int]], list[str], bool]

Read identities already recorded by aggregation; never derive repeat scores.

nemo_gym.eval.comparison.suite.coverage.run_coverage(
    run: nemo_gym.eval.comparison.suite.loading.LoadedSuiteRun
) -> nemo_gym.eval.comparison.suite.schema.SuiteCoverage

Describe a source’s own recorded and planned evidence independently of other runs.