nemo_gym.artifacts.exporters.base
nemo_gym.artifacts.exporters.base
Module Contents
Classes
Functions
Data
API
Bases: Exporter
Sink for the run metadata a NeMo Gym eval produces: config, metrics, rollouts.
A backend (W&B, MLflow, …) subclasses this and wraps its own run handle. Lifecycle is
setup -> any number of log_* calls -> teardown. Whether a backend runs at all is decided
by its ExporterConfig in the registry, not here.
Exporters are best-effort telemetry: a failing tracking server must not fail the eval. Call
sites go through the export_* wrappers, which swallow and log backend exceptions. Subclasses
implement the unwrapped _* hooks and may raise freely.
Close the reopened run. A backend that marks a run when it reopens it undoes that mark here, and lets go of the run first, so that nothing more reaches it even when closing fails.
A flag of the raw config read as the validated run config reads it, so “false” or “0” (as an
${oc.env:...} interpolation yields) is false. A value that config would reject does not continue a run.
Log the reopened run’s config, as the run continues. Backends call this first in teardown too.
Remove the marker that names the reopened run, so that a failed replacement setup cannot leave the next
resume reopening that run. A marker that names another run is left alone. Best effort: if the marker can be
neither removed nor replaced by the new run’s, the next resume still reopens the old run.
Record the resolved run config. Secrets are already masked by export_config.
Record scalar metrics. step is the training step, or None for a one-shot eval.
Record rollout results as a table. Rollouts are sanitized dicts, one per rollout.
The rollouts a live run would have uploaded.
The run to reopen: the one recorded next to the output, when this run continues that output.
A resume continues it, as does a judge-failure recovery that appends to it. A fresh run into the same output
starts its rollouts anew, so it starts a new tracking run too, as does a resume that finds nothing to resume
(start_over).
<output stem>_<name>_run_id next to output_jsonl_fpath, or None when the run has no output.
A resumed or resubmitted run writes to the same output, so a backend that records its run id there
reopens that run instead of starting a new one. Deleting the file starts a new run. Only a live run
reads or writes it; a post-run export keeps its own marker (RunArtifacts.export_done_fpath).
Whether the rollouts are uploaded, as a live run would have decided.
An explicit upload_rollouts= on the export command wins; otherwise the run’s own saved setting does.
Send a finished run to the backend: its saved config, aggregate metrics and (optionally) rollouts.
Skipped, without contacting the backend, when the run’s export receipt matches its current artifacts; a
receipt that does not match them, or a marker that is not a version-1 receipt, is a failure and nothing is
sent. Failures come back in the result rather than being swallowed the way live export_* calls are.
Optionally certify an explicitly completed live activation; teardown alone never certifies it.
Build the exporter from the whole resolved config.
This overrides Exporter.from_global_config because these exporters have always been built from the full
config (a live run has no settings model of its own) instead of from a validated model. So only the shared
settings are validated here, and each backend validates its own in its constructor.
Optionally mark that the post-run export run_id supersedes the live run live_run_id it could not
certify. Best effort: a backend logs and ignores any failure, and never changes either run’s data.
Open the backing run. Called once, after config resolution.
Track the output in a new run after all.
Rollout collection calls this when it was asked to resume its output but found nothing to resume and starts it
anew, before it logs any rollout. A no-op unless setup reopened the recorded run.
Close the backing run. Must be safe to call when setup failed or never ran.
What one exporter did with one finished run.
Bases: enum.Enum
Sends a finished run to a destination, or converts it to another format.
An exporter receives the run’s RunArtifacts and decides which of those files it reads. Its settings
come from config_model, validated from the export command’s own config before export is called.
Importing a backend’s SDK belongs inside export, so listing exporters never loads one.
Export the run. Report an expected failure in the result; raise only for a bug.
Build the exporter from the export command’s resolved config, validating it first.
How many planned rows an accepted partial stage omitted: the number recovery’s acceptance gate states.
Why the aggregate metrics file is not a complete result, or None.
A failed /aggregate_metrics request leaves an aggregation_error entry beside the healthy agents’
metrics. A live run keeps those partial metrics and reports the failure, but an export marks the run
done, so it must not send a partial result.
Content identities for a run, independent of its writers, timestamps and directory location.
The backend run a live exporter recorded for this run, or None when there is none or it cannot be read.
The upload settings in config, resolving only those values.
A saved run config keeps its ${...} interpolations unresolved, and resolving all of it would need the
environment the run had. The canonical key wins over the deprecated one, as in the settings model, so a
deprecated value that loses is never evaluated.
Publish the already-exported marker only once it is fully written, so a failed write leaves none.
What is missing for the run to be exported faithfully, or None when it can be.
Pass rollouts_required=False when the exporter does not read the rollouts file. When that file is there,
the aggregate metrics must still not be older than it.