Skip to navigation
API ReferenceFull Library ReferenceNemo GymNemo GymEvalComparisonSuite

nemo_gym.eval.comparison.suite.schema

View as Markdown

Manifest, configuration, and report contracts for benchmark-suite comparisons.

Module Contents

Classes

NameDescription
SuiteAggregateResult-
SuiteBenchmarkDefinitionMetrics averaged within each repeat of one benchmark.
SuiteCellResult-
SuiteComparisonConfigCompare baseline and candidate runs across a suite of models and benchmarks.
SuiteComparisonResultSeparate schema for suite_compare_report.json; run reports remain unchanged.
SuiteCoverageRecorded and planned task-coverage evidence for one selected agent.
SuiteFailureCountsSaved selected failure outcomes for the entire rollout file.
SuiteFailureSummaryPrecomputed accounting, without rescanning or certifying the source artifacts.
SuiteManifestExplicit benchmark definitions and a possibly sparse model-by-benchmark matrix.
SuiteMetricDefinitionA selected repeat metric and an optional complete theoretical scale override.
SuiteMetricResult-
SuiteRunPairNonempty source collections, accepting a single mapping as shorthand in YAML.
SuiteRunSourceOne run with companion artifacts discovered from its rollout path.
SuiteRunSummaryOne source or pooled repeats, with full per-source provenance under sources.
SuiteSampleSummary-
SuiteStatisticsCompleted numerical results; presentation only formats these fields.
SuiteVarianceContribution-
_SuiteModel-

Functions

NameDescription
summary_sourcesReturn constituent provenance, preserving a standalone source’s report contract.

Data

CoverageStatus

API

class nemo_gym.eval.comparison.suite.schema.SuiteAggregateResult()

Bases: _SuiteModel

cell_ids
list[str]
name
str
statistics
SuiteStatistics | None
weights
list[float]
class nemo_gym.eval.comparison.suite.schema.SuiteBenchmarkDefinition()

Bases: _SuiteModel

Metrics averaged within each repeat of one benchmark.

metrics
list[SuiteMetricDefinition] = Field(min_length=1)
class nemo_gym.eval.comparison.suite.schema.SuiteCellResult()

Bases: _SuiteModel

baseline
SuiteRunSummary
benchmark
str
candidate
SuiteRunSummary
eligible
bool
exclusion_reason
str | None = None
metrics
list[SuiteMetricResult]
model
str
model_weight
float = 0.0
overall_weight
float = 0.0
statistics
SuiteStatistics | None = None
class nemo_gym.eval.comparison.suite.schema.SuiteComparisonConfig()

Bases: BaseNeMoGymCLIConfig

Compare baseline and candidate runs across a suite of models and benchmarks.

Run paths in the suite YAML resolve relative to that file. Reports give delta = candidate - baseline on a 0-100 normalized scale with unpaired Welch intervals.

confidence_level
float
output_dirpath
str | None
report_format
ReportFormat
suite_fpath
str
class nemo_gym.eval.comparison.suite.schema.SuiteComparisonResult()

Bases: _SuiteModel

Separate schema for suite_compare_report.json; run reports remain unchanged.

assumptions
list[str]
cells
list[SuiteCellResult]
command
str
confidence_level
float
generated_at
str
nemo_gym_version
str
overall
SuiteAggregateResult
per_model
list[SuiteAggregateResult]
schema_version
Literal['1'] = '1'
suite_fpath
Path
warnings
list[str] = Field(default_factory=list)
class nemo_gym.eval.comparison.suite.schema.SuiteCoverage()

Bases: _SuiteModel

Recorded and planned task-coverage evidence for one selected agent.

notes
list[str] = Field(default_factory=list)
planned_task_count
int | None = None
recorded_task_count
int
status
CoverageStatus
class nemo_gym.eval.comparison.suite.schema.SuiteFailureCounts()

Bases: _SuiteModel

Saved selected failure outcomes for the entire rollout file.

by_class
dict[str, Annotated[int, Field(strict=True, ge=0)]]
counted_as_zero
int | None = Field(default=None, strict=True, ge=0)
final
int = Field(strict=True, ge=0)
final_zeros
int | None = Field(default=None, strict=True, ge=0)
not_final
int = Field(strict=True, ge=0)
total
int = Field(strict=True, ge=0)
class nemo_gym.eval.comparison.suite.schema.SuiteFailureSummary()

Bases: _SuiteModel

Precomputed accounting, without rescanning or certifying the source artifacts.

counts
SuiteFailureCounts | None = None
scope
Literal['rollout_file'] = 'rollout_file'
source
Path
unavailable_reason
str | None = None
class nemo_gym.eval.comparison.suite.schema.SuiteManifest()

Bases: _SuiteModel

Explicit benchmark definitions and a possibly sparse model-by-benchmark matrix.

benchmarks
dict[str, SuiteBenchmarkDefinition] = Field(min_length=1)
models
dict[str, dict[str, SuiteRunPair]] = Field(min_length=1)
schema_version
Literal[1] = 1
class nemo_gym.eval.comparison.suite.schema.SuiteMetricDefinition()

Bases: _SuiteModel

A selected repeat metric and an optional complete theoretical scale override.

high
float | None = None
higher_is_better
bool | None = None
low
float | None = None
name
str = Field(min_length=1)
class nemo_gym.eval.comparison.suite.schema.SuiteMetricResult()

Bases: _SuiteModel

baseline_raw
list[float] = Field(default_factory=list)
baseline_scaled
list[float] = Field(default_factory=list)
candidate_raw
list[float] = Field(default_factory=list)
candidate_scaled
list[float] = Field(default_factory=list)
definition_source
str
high
float
higher_is_better
bool
low
float
name
str
class nemo_gym.eval.comparison.suite.schema.SuiteRunPair()

Bases: _SuiteModel

Nonempty source collections, accepting a single mapping as shorthand in YAML.

baseline
list[SuiteRunSource] = Field(min_length=1)
candidate
list[SuiteRunSource] = Field(min_length=1)
nemo_gym.eval.comparison.suite.schema.SuiteRunPair.normalize_sources(
    value: object
) -> object
classmethod

Choose the source shape before validation so errors describe only that shape.

class nemo_gym.eval.comparison.suite.schema.SuiteRunSource()

Bases: _SuiteModel

One run with companion artifacts discovered from its rollout path.

agent
str | None = None
environment_server
str | None = None
rollouts
Path
class nemo_gym.eval.comparison.suite.schema.SuiteRunSummary()

Bases: _SuiteModel

One source or pooled repeats, with full per-source provenance under sources.

For pooled repeats, path and selector fields identify the first source for compatibility. Local repeat identities and saved failure accounting live under sources.

agent
str
aggregate_metrics
Path
coverage
SuiteCoverage
environment_server
str | None = None
failures
SuiteFailureSummary | None = None
materialized_inputs
Path | None = None
repeat_count
int | None
repeat_indices
list[int] = Field(default_factory=list)
repeat_scores
list[float] = Field(default_factory=list)
rollouts
Path
sources
list[SuiteRunSummary] = Field(default_factory=list)
summary
SuiteSampleSummary | None = None
unscored_repeat_indices
list[int] = Field(default_factory=list)
class nemo_gym.eval.comparison.suite.schema.SuiteSampleSummary()

Bases: _SuiteModel

count
int
mean
float
sample_variance
float
class nemo_gym.eval.comparison.suite.schema.SuiteStatistics()

Bases: _SuiteModel

Completed numerical results; presentation only formats these fields.

baseline_mean
float
candidate_mean
float
ci_high
float | None
ci_low
float | None
confidence_level
float
contributions
list[SuiteVarianceContribution]
degrees_of_freedom
float | None
delta
float
interval_unavailable_reason
str | None = None
standard_error
float
variance
float
class nemo_gym.eval.comparison.suite.schema.SuiteVarianceContribution()

Bases: _SuiteModel

baseline
float
candidate
float
weight
float
class nemo_gym.eval.comparison.suite.schema._SuiteModel()

Bases: BaseModel

model_config
= ConfigDict(extra='forbid')
nemo_gym.eval.comparison.suite.schema.summary_sources(
    summary: nemo_gym.eval.comparison.suite.schema.SuiteRunSummary
) -> tuple[nemo_gym.eval.comparison.suite.schema.SuiteRunSummary, ...]

Return constituent provenance, preserving a standalone source’s report contract.

nemo_gym.eval.comparison.suite.schema.CoverageStatus = Literal['verified', 'unverified', 'incomplete']