Skip to navigation

nemo_gym.eval.comparison.planning.schema

View as MarkdownOpen in Claude

Strict direct-input manifest and completed report contracts for repeat planning.

Module Contents

Classes

NameDescription
RepeatPlanningBaselineWhether baseline repeats are existing measurements or a planned investment.
RepeatPlanningCLIConfigPlan fixed repeat counts from a YAML of normalized repeat standard deviations.
RepeatPlanningCellOne normalized complete-repeat benchmark score for one model.
RepeatPlanningCellResult-
RepeatPlanningManifestOne fixed aggregate planned from direct normalized SD inputs.
RepeatPlanningResultCompleted numerical results and resolved inputs for JSON and Markdown reports.
RepeatPlanningTotals-
_PlanningModel-

Data

Name

StandardDeviation

API

class nemo_gym.eval.comparison.planning.schema.RepeatPlanningBaseline()

Bases: _PlanningModel

Whether baseline repeats are existing measurements or a planned investment.

mode
Literal['optimize', 'existing'] = 'optimize'
class nemo_gym.eval.comparison.planning.schema.RepeatPlanningCLIConfig()

Bases: BaseNeMoGymCLIConfig

Plan fixed repeat counts from a YAML of normalized repeat standard deviations.

output_dirpath
str | None
repeat_plan_fpath
str
report_format
ReportFormat
class nemo_gym.eval.comparison.planning.schema.RepeatPlanningCell()

Bases: _PlanningModel

One normalized complete-repeat benchmark score for one model.

For a multi-metric benchmark, normalize and equally average its metrics within each repeat before estimating this score’s SD across repeats. An omitted candidate SD defaults to the baseline SD.

baseline_std_points
StandardDeviation
benchmark
Name
candidate_std_points
StandardDeviation | None = None
existing_baseline_repeats
int | None = Field(default=None, strict=True, ge=2)
model
Name
repeat_cost
float
repeat_step
int = Field(default=1, strict=True, ge=1)
class nemo_gym.eval.comparison.planning.schema.RepeatPlanningCellResult()

Bases: _PlanningModel

baseline_repeats
int
baseline_std_points
float
benchmark
str
candidate_repeats_all_experiments
int
candidate_repeats_per_experiment
int
candidate_std_points
float
existing_baseline_repeats
int
model
str
new_baseline_repeats
int
new_repeat_cost
float
repeat_cost
float
repeat_step
int
weight
float
class nemo_gym.eval.comparison.planning.schema.RepeatPlanningManifest()

Bases: _PlanningModel

One fixed aggregate planned from direct normalized SD inputs.

benchmark selects one model/benchmark cell, model selects cells of a single model, and suite selects all declared cells. Every cell has equal aggregate weight and supplies the SD of one complete-repeat benchmark score.

aggregation_level
Literal['benchmark', 'model', 'suite']
alpha
float
baseline
RepeatPlanningBaseline = Field(default_factory=RepeatPlanningBaseline)
candidate_experiments
int = Field(default=1, strict=True, ge=1)
cells
list[RepeatPlanningCell] = Field(min_length=1)
expected_delta_points
float
margin_points
float
max_repeats
int = Field(default=20000, strict=True, ge=2)
schema_version
Literal[1]
seed
int = Field(default=20261109, strict=True, ge=0)
target_power
float
validation_trials
int = Field(default=10000, strict=True, ge=0)
nemo_gym.eval.comparison.planning.schema.RepeatPlanningManifest.validate_version_type(
value: object
) -> object
classmethod
class nemo_gym.eval.comparison.planning.schema.RepeatPlanningResult()

Bases: _PlanningModel

Completed numerical results and resolved inputs for JSON and Markdown reports.

allocation
RepeatAllocation
cells
list[RepeatPlanningCellResult]
command
str
generated_at
str
nemo_gym_version
str
plan_fpath
Path
schema_version
Literal['1'] = '1'
settings
RepeatPlanningManifest
simulation
SimulationResult | None
totals
RepeatPlanningTotals
class nemo_gym.eval.comparison.planning.schema.RepeatPlanningTotals()

Bases: _PlanningModel

candidate_passes_all_experiments
int
candidate_passes_per_experiment
int
existing_baseline_passes
int
new_baseline_passes
int
total_additional_passes
int
total_cost
float
class nemo_gym.eval.comparison.planning.schema._PlanningModel()

Bases: BaseModel

model_config
= ConfigDict(extra='forbid')
nemo_gym.eval.comparison.planning.schema.Name = Annotated[str, StringConstraints(strip_whitespace=True, min_length=1, strict=Tru...
nemo_gym.eval.comparison.planning.schema.StandardDeviation = Annotated[float, Field(strict=True, ge=0, allow_inf_nan=False)]