Skip to navigation
API ReferenceFull Library ReferenceNemo GymNemo GymEvalComparison

nemo_gym.eval.comparison.schema

View as Markdown

Config and result schema for gym eval compare.

Configuration lists follow source order, including agent selectors and aggregate overrides. Pooled candidate sources produce one result side; candidate-varying result fields follow ComparisonResult.candidates. Comparison currently supports one candidate side.

Module Contents

Classes

NameDescription
AgentComparison-
CandidateMetricValueA candidate’s reading, plus its movement from the baseline.
ComparisonConfigCompare baseline and candidate eval runs, optionally pooling independent collections.
ComparisonResultThe machine-readable artifact written as compare_report.json.
FlipSummary-
MetricRow-
MetricValueOne side’s reading of a metric, plus whatever uncertainty the run recorded for it.
RunFileParsed aggregates indexed by agent; pooled paths represent the first of authoritative sources.
RunSourceAn input artifact contributing independent repeats to a pooled side.
TaskFlip-

Data

MAX_CANDIDATES

MAX_FLIPS_SHOWN

ReportFormat

API

class nemo_gym.eval.comparison.schema.AgentComparison()

Bases: BaseModel

baseline_agent
str
baseline_repeat_count
Optional[int] = None
baseline_task_count
int
candidate_agents
List[str]
candidate_repeat_counts
List[Optional[int]] = Field(default_factory=list)
candidate_task_counts
List[int]
flips
List[FlipSummary] = Field(default_factory=list)
metrics
List[MetricRow] = Field(default_factory=list)
notes
List[str] = Field(default_factory=list)
class nemo_gym.eval.comparison.schema.CandidateMetricValue()

Bases: MetricValue

A candidate’s reading, plus its movement from the baseline.

delta
Optional[float] = None
delta_ci_high
Optional[float] = None
delta_ci_low
Optional[float] = None
delta_pct
Optional[float] = None
class nemo_gym.eval.comparison.schema.ComparisonConfig()

Bases: BaseNeMoGymCLIConfig

Compare baseline and candidate eval runs, optionally pooling independent collections.

Reads only each run’s <stem>_aggregate_metrics.json, derived from the rollouts JSONL path you pass, and writes compare_report.md and/or compare_report.json. The rollouts JSONL itself is never opened — it is the run’s identity and the handle from which the sibling path is derived. When a legacy aggregate lacks repeat statistics, they are reconstructed in memory from its embedded rollout summaries for this comparison.

Examples:

gym eval compare         --baseline outputs/run_a/rollouts.jsonl         --candidates outputs/run_b/rollouts.jsonl
gym eval compare         --baseline-runs outputs/b1/rollouts.jsonl,outputs/b2/rollouts.jsonl         --candidate-runs outputs/c1/rollouts.jsonl,outputs/c2/rollouts.jsonl

To point at metrics files that do not follow the <stem>_aggregate_metrics.json convention, use —baseline-agg-metrics, —baseline-runs-agg-metrics and —candidates-agg-metrics in the corresponding source order. Each pooled repeat is an equally weighted observation.

agent_name
Optional[str]
baseline_agent_name
Optional[str]
baseline_agent_names
Optional[List[str]]
baseline_aggregate_metrics_fpath
Optional[str]
baseline_aggregate_metrics_fpaths
Optional[List[str]]
baseline_rollouts_jsonl_fpath
Optional[str]
baseline_rollouts_jsonl_fpaths
Optional[List[str]]
candidate_agent_names
Optional[List[str]]
candidate_aggregate_metrics_fpaths
Optional[List[str]]
candidate_rollouts_jsonl_fpaths
List[str]
confidence_level
float
output_dirpath
Optional[str]
pooled_candidate_rollouts_jsonl_fpaths
Optional[List[str]]
report_format
ReportFormat
class nemo_gym.eval.comparison.schema.ComparisonResult()

Bases: BaseModel

The machine-readable artifact written as compare_report.json.

baseline
RunFile
candidates
List[RunFile]
command
str
comparisons
List[AgentComparison] = Field(default_factory=list)
confidence_level
float
generated_at
str
nemo_gym_version
str
schema_version
Literal['1'] = '1'
skipped_agents
Dict[str, List[str]] = Field(default_factory=dict)
warnings
List[str] = Field(default_factory=list)
class nemo_gym.eval.comparison.schema.FlipSummary()

Bases: BaseModel

baseline_only_task_count
int = 0
candidate_index
int
candidate_only_task_count
int = 0
common_task_count
int = 0
fail_to_pass_count
Optional[int] = None
field
str = 'reward'
flips
List[TaskFlip] = Field(default_factory=list)
mode
Literal['binary', 'continuous', 'unavailable']
net
Optional[int] = None
pass_to_fail_count
Optional[int] = None
reason
Optional[str] = None
tied_count
Optional[int] = None
unchanged_count
Optional[int] = None
class nemo_gym.eval.comparison.schema.MetricRow()

Bases: BaseModel

baseline
Optional[MetricValue] = None
candidates
List[Optional[CandidateMetricValue]] = Field(default_factory=list)
is_key_metric
bool
metric
str
present_in
List[str]
class nemo_gym.eval.comparison.schema.MetricValue()

Bases: BaseModel

One side’s reading of a metric, plus whatever uncertainty the run recorded for it.

ci_high
Optional[float] = None
ci_low
Optional[float] = None
mean_across_repeats
Optional[float] = None
se_across_repeats
Optional[float] = None
std_err_across_runs
Optional[float] = None
value
float
class nemo_gym.eval.comparison.schema.RunFile()

Bases: BaseModel

Parsed aggregates indexed by agent; pooled paths represent the first of authoritative sources.

agent_names
List[str]
aggregate_metrics_fpath
Path
entries_by_agent
Dict[str, Any]
index
int = 0
label
str

Short display name for the run: the directory its rollouts live in.

model_config
= ConfigDict(frozen=True)
role
Literal['baseline', 'candidate']
rollouts_jsonl_fpath
Path
sources
List[RunSource] = Field(default_factory=list)
class nemo_gym.eval.comparison.schema.RunSource()

Bases: BaseModel

An input artifact contributing independent repeats to a pooled side.

aggregate_metrics_fpath
Path
model_config
= ConfigDict(frozen=True)
repeat_counts_by_agent
Dict[str, Optional[int]]
rollouts_jsonl_fpath
Path
scored_repeat_counts_by_agent
Dict[str, int] = Field(default_factory=dict)
selected_agents
List[str] = Field(default_factory=list)
task_counts_by_agent
Dict[str, int] = Field(default_factory=dict)
class nemo_gym.eval.comparison.schema.TaskFlip()

Bases: BaseModel

baseline_rewards
Optional[List[float]] = None
baseline_score
float
candidate_rewards
Optional[List[float]] = None
candidate_score
float
delta
float
direction
Literal['pass_to_fail', 'fail_to_pass', 'changed']
task_index
int
nemo_gym.eval.comparison.schema.MAX_CANDIDATES = 1
 
nemo_gym.eval.comparison.schema.MAX_FLIPS_SHOWN = 10
 
nemo_gym.eval.comparison.schema.ReportFormat = Literal['md', 'json', 'both']