nemo_gym.eval.comparison.suite.loading

Read suite manifests and recorded artifacts without reconstructing repeat metrics.

Module Contents

Classes

Name	Description
`LoadedSuiteRun`	One selected aggregate entry and its optional planned task identities.

Functions

Name	Description
`_index`	-
`_object_rows`	-
`_read_planned_tasks`	Select the plan using the same server identity as aggregation, with unstamped-row fallback.
`_repeat_count`	Retain verified zero/single-repeat counts when no repeat estimates are stored.
`_select_entry`	Resolve optional selectors independently for one source’s aggregate file.
`load_suite_manifest`	Read and validate the explicit suite input contract.
`load_suite_run`	Select one aggregate entry by agent and/or server, retaining its recorded repeat rows.
`resolve_metric_definitions`	Validate explicit metric scales, using supplied catalog metadata for mean/reward.

API

class nemo_gym.eval.comparison.suite.loading.LoadedSuiteRun(
    rollouts: pathlib.Path,
    aggregate_metrics: pathlib.Path,
    materialized_inputs: pathlib.Path | None,
    agent: str,
    environment_server: str | None,
    repeat_rows: tuple[typing.Mapping[str, object], ...],
    groups: tuple[typing.Mapping[str, object], ...],
    agent_metrics: typing.Mapping[str, object],
    repeat_count: int,
    planned_tasks_by_repeat: typing.Mapping[int, frozenset[int]] | None,
    unscored_repeat_indices: tuple[int, ...] = ()
)

Dataclass

One selected aggregate entry and its optional planned task identities.

agent

str

agent_metrics

Mapping[str, object]

aggregate_metrics

Path

environment_server

str | None

groups

tuple[Mapping[str, object], ...]

materialized_inputs

Path | None

planned_tasks_by_repeat

Mapping[int, frozenset[int]] | None

repeat_count

int

repeat_rows

tuple[Mapping[str, object], ...]

rollouts

Path

unscored_repeat_indices

tuple[int, ...] = ()

nemo_gym.eval.comparison.suite.loading._index(
    value: object,
    context: str
) -> int

nemo_gym.eval.comparison.suite.loading._object_rows(
    value: object,
    field: str,
    path: pathlib.Path
) -> tuple[typing.Mapping[str, object], ...]

nemo_gym.eval.comparison.suite.loading._read_planned_tasks(
    path: pathlib.Path,
    agent: str,
    environment_server: str | None
) -> dict[int, frozenset[int]]

Select the plan using the same server identity as aggregation, with unstamped-row fallback.

nemo_gym.eval.comparison.suite.loading._repeat_count(
    rows: tuple[typing.Mapping[str, object], ...],
    groups: tuple[typing.Mapping[str, object], ...],
    agent_metrics: typing.Mapping[str, object],
    path: pathlib.Path,
    agent: str
) -> int

Retain verified zero/single-repeat counts when no repeat estimates are stored.

nemo_gym.eval.comparison.suite.loading._select_entry(
    entries: typing.Mapping[str, dict[str, object]],
    source: nemo_gym.eval.comparison.suite.schema.SuiteRunSource,
    path: pathlib.Path
) -> tuple[str, dict[str, object]]

Resolve optional selectors independently for one source’s aggregate file.

nemo_gym.eval.comparison.suite.loading.load_suite_manifest(
    path: pathlib.Path
) -> nemo_gym.eval.comparison.suite.schema.SuiteManifest

Read and validate the explicit suite input contract.

nemo_gym.eval.comparison.suite.loading.load_suite_run(
    source: nemo_gym.eval.comparison.suite.schema.SuiteRunSource,
    directory: pathlib.Path
) -> nemo_gym.eval.comparison.suite.loading.LoadedSuiteRun

Select one aggregate entry by agent and/or server, retaining its recorded repeat rows.

nemo_gym.eval.comparison.suite.loading.resolve_metric_definitions(
    benchmark: str,
    definition: nemo_gym.eval.comparison.suite.schema.SuiteBenchmarkDefinition,
    reward_metric_resolver: collections.abc.Callable[[str], nemo_gym.eval.comparison.suite.schema.SuiteMetricResult] | None = None
) -> list[nemo_gym.eval.comparison.suite.schema.SuiteMetricResult]

Validate explicit metric scales, using supplied catalog metadata for mean/reward.