Skip to navigation

nemo_gym.eval.recovery.evidence

View as MarkdownOpen in Claude

The evidence collector of gym eval resubmit --recover: one self-contained Python script, run on the run’s host over the executor’s connection (or locally), that reads the job directory and prints one JSON object.

Decisions live in nemo_gym.eval.recovery.planner; this module holds the script and the constants it embeds.

Module Contents

Classes

NameDescription
CollectedEvidence-
ManagedRecoveryFactRead-only facts from the authoritative Store, not a second retry policy.
OperatorEpochWhere the last operator continuation started the chain’s accounting: its chain length and walltime baseline.

Functions

NameDescription
_identity-
_managed_source_digestHash one stable file with bounded memory, using the exporter’s existing identity checks.
_managed_trial_metadataSession trial metadata from the bound saved config and the Store’s own inputs, after Store selection.
select_managed_evidenceRead locally accessible Store history in this installed Gym process; never copy its selection rules.

Data

COLLECT_SCRIPT

EPISODE_DEADLINE_REASON

Identity

JUDGE_JOB_ACK_JOURNAL_NAME

JUDGE_JOB_JOURNAL_NAME

OPERATOR_CONTINUATION_JOURNAL_NAME

READER_PLACEMENT_FIELDS

ROLLOUT_INDEX

STORE_READER_UNAVAILABLE

STORE_SKIP_KEYS

TASK_INDEX

UNOBSERVED_CLAIM_REASON

UNREADABLE_CLAIM_REASON

API

class nemo_gym.eval.recovery.evidence.CollectedEvidence()

Bases: BaseModel

accounted_walltime_s
float | None = None
accounting_reconciled
bool = False
accounting_snapshot
dict[str, str | None] | None = None
accounting_unknown_reason
str | None = None
attempt_extension
int | None = None
attempt_extension_applied
int | None = None
attempt_extension_state
str = 'absent'
benchmark_metric_coverage
tuple[int, int, str | None] | None = None
chain_continuation_note
str | None = None
chain_ended_itself
bool = False
chain_incomplete
bool = False
checkpoint_enabled
bool | None = Field(default=None, strict=True)
collector_fallback_reason
str | None = None
collector_host
str | None = None
counter_states
dict[str, str] | None = None
coverage_allocation_id
str | None = None
coverage_scored
tuple[int, int] | None = None
coverage_writer_unrecorded
bool = False
failures
list[dict[str, Any]]
followup_claims
dict[str, str] = Field(default_factory=dict)
followup_sentinels
dict[str, str] = Field(default_factory=dict)
infra_retries_configured
int | None = None
infra_retries_state
str = 'absent'
infra_retries_used
float | None = None
infra_retry_extension
float | None = None
infra_retry_extension_state
str = 'absent'
job_chain
list[str] = Field(default_factory=list)
judge_job_acknowledgements
list[dict[str, Any]] = Field(default_factory=list)
judge_jobs
list[dict[str, Any]] = Field(default_factory=list)
managed_reader_error
str | None = None
managed_reader_unavailable
bool = False
managed_selection
list[ManagedRecoveryFact] | None = None
managed_store
bool = False
materialized
list[dict[str, Any]] | None
mlflow
dict[str, Any] | None = None
operator_epoch
OperatorEpoch | None = None
quality
dict[str, Any] | None = None
reader_fallback_reason
str | None = None
reader_host
str | None = None
rerun_masked_applied
bool | None = None
resubmit_offset
int = 0
results
list[dict[str, Any]] | None
resume_drift_keys
list[str] | None = None
score
float | None = None
source_digests
dict[str, str | None] | None = None
source_job_identity
tuple[str, int, int] | None = None
task_selection
dict[str, Any] | None = None
unreadable_files
list[str] = Field(default_factory=list)
unresolved_rejudgments
list[str] = Field(default_factory=list)
walltime_state
str = 'absent'
walltime_used_s
float | None = None
class nemo_gym.eval.recovery.evidence.ManagedRecoveryFact()

Bases: BaseModel

Read-only facts from the authoritative Store, not a second retry policy.

attempts
int = Field(ge=0, strict=True)
identity
Identity
max_attempts
int = Field(ge=1, strict=True)
pending
bool
replayable
bool | None = None
rerun_approved
bool = False
reserved
bool
class nemo_gym.eval.recovery.evidence.OperatorEpoch()

Bases: BaseModel

Where the last operator continuation started the chain’s accounting: its chain length and walltime baseline.

chain_length
int = Field(ge=0)
walltime_s
float = Field(ge=0, allow_inf_nan=False)
nemo_gym.eval.recovery.evidence._identity(
row: collections.abc.Mapping[str, typing.Any]
nemo_gym.eval.recovery.evidence._managed_source_digest(
path: pathlib.Path
) -> str | None

Hash one stable file with bounded memory, using the exporter’s existing identity checks.

Two streaming passes must agree on stat identity and bytes: a same-size in-place rewrite within one timestamp tick changes no stat field. This doubles source reads; it does not make the reader atomic against ABA writes.

nemo_gym.eval.recovery.evidence._managed_trial_metadata(
artifacts: pathlib.Path,
source_digests: dict,
quality: dict | None,
results: list[dict],
store: object
) -> tuple[list[dict], bool | None]

Session trial metadata from the bound saved config and the Store’s own inputs, after Store selection.

Returns the results with materialized_task_id on each session row and the effective checkpoint setting, or the results unchanged and None when any fact is missing. Self-contained: the remote Store reader runs this source inside the saved install, on that install’s Store, materialization and checkpoint APIs.

nemo_gym.eval.recovery.evidence.select_managed_evidence(
job_dir: pathlib.Path

Read locally accessible Store history in this installed Gym process; never copy its selection rules.

The plain remote collector remains dependency-free. Call only on the same host as the installed caller, after collection. Both snapshots must include the manifest and journal and remain byte-identical.

nemo_gym.eval.recovery.evidence.COLLECT_SCRIPT = 'ACCOUNTING_UNKNOWN_PREFIX = ' + repr(ACCOUNTING_UNKNOWN_PREFIX) + '\n' + 'PROVI...
nemo_gym.eval.recovery.evidence.EPISODE_DEADLINE_REASON = 'Episode timed out'
nemo_gym.eval.recovery.evidence.Identity = tuple[int, int]
nemo_gym.eval.recovery.evidence.JUDGE_JOB_ACK_JOURNAL_NAME = 'judge-job-acknowledgement-journal.jsonl'
nemo_gym.eval.recovery.evidence.JUDGE_JOB_JOURNAL_NAME = 'judge-job-journal.jsonl'
nemo_gym.eval.recovery.evidence.OPERATOR_CONTINUATION_JOURNAL_NAME = 'operator-continuation-journal.jsonl'
nemo_gym.eval.recovery.evidence.READER_PLACEMENT_FIELDS = ('collector_host', 'collector_fallback_reason', 'reader_host', 'reader_fallback_...
nemo_gym.eval.recovery.evidence.ROLLOUT_INDEX = ROLLOUT_INDEX_KEY_NAME
nemo_gym.eval.recovery.evidence.STORE_READER_UNAVAILABLE = 'the run was written by a rollout store and its Store reader could not run, so n...
nemo_gym.eval.recovery.evidence.STORE_SKIP_KEYS = frozenset({'ng_trajectory', 'ng_agent_observations'})
nemo_gym.eval.recovery.evidence.TASK_INDEX = TASK_INDEX_KEY_NAME
nemo_gym.eval.recovery.evidence.UNOBSERVED_CLAIM_REASON = 'a follow-up submission claim is unobserved'
nemo_gym.eval.recovery.evidence.UNREADABLE_CLAIM_REASON = 'the chain marker or a follow-up claim is unreadable'