Skip to navigation

nemo_gym.rollouts.provenance

View as MarkdownOpen in Claude

Run metadata kept with a result: allowlisted identifiers, fingerprints of prompts and inputs, and the resolved config with secrets masked.

Module Contents

Classes

NameDescription
RunProvenanceReproducibility metadata; unknown values stay null and credentials are never selected.
ServedBackendWhat one model backend reports serving at /models, or why that is unknown.

Functions

NameDescription
_ask_backendWhat one backend serves; any failure, from an unresolved ${...} URL to a malformed answer, is its own.
_backend_urls-
_is_model_server-
_privateA mapping never searched for identifiers: whatever it holds can be a credential under any name.
_sampling-
build_provenanceSnapshot model, judge, template and sampling settings. The masked config itself is saved separately.
served_modelsAsk each model server’s backends what they serve (GET <base_url>/models), once per run.
write_run_provenanceWrite one provenance artifact next to a run’s materialized inputs before dispatch.

Data

INPUT_SAMPLING_ROW_LIMIT

SERVED_MODEL_BACKEND_LIMIT

SERVED_MODEL_PROBE_WORKERS

SERVED_MODEL_TIMEOUT_S

TOP_LEVEL_PATH

_MODEL_KEYS

_SAMPLING_KEYS

logger

API

class nemo_gym.rollouts.provenance.RunProvenance()

Bases: BaseModel

Reproducibility metadata; unknown values stay null and credentials are never selected.

datasets
list[str] = Field(default_factory=list)
efb_config_ref
str | None = None
error_type
str | None = None
gym_commit
str | None = None
gym_version
str = __version__
input_sampling_parameters
list[dict[str, JsonValue]] = Field(default_factory=list)
input_sampling_row_limit
int = INPUT_SAMPLING_ROW_LIMIT
input_sampling_rows
int = 0
input_sampling_rows_skipped
int = 0
judge_prompt_versions
dict[str, str] = Field(default_factory=dict)
materialized_inputs_sha256
str | None = None
models
dict[str, dict[str, str]] = Field(default_factory=dict)
prompt_config
dict[str, str] | None = None
prompt_config_paths
dict[str, str] = Field(default_factory=dict)
sampling_parameters
dict[str, dict[str, JsonValue]] = Field(default_factory=dict)
schema_version
int = 1
seed
int | None = None
served_models
dict[str, list[ServedBackend]] = Field(default_factory=dict)
served_models_unknown_reason
str | None = None
status
Literal['available', 'unavailable'] = 'available'
template_sha256
dict[str, str] = Field(default_factory=dict)
class nemo_gym.rollouts.provenance.ServedBackend()

Bases: BaseModel

What one model backend reports serving at /models, or why that is unknown.

id
str | None = None
root
str | None = None
unknown_reason
str | None = None
nemo_gym.rollouts.provenance._ask_backend(
base_url: str,
api_key: object,
configured: set[str]

What one backend serves; any failure, from an unresolved ${...} URL to a malformed answer, is its own.

nemo_gym.rollouts.provenance._backend_urls(
server: collections.abc.Mapping
) -> list[str]
nemo_gym.rollouts.provenance._is_model_server(
node: object
) -> bool
nemo_gym.rollouts.provenance._private(
key: object
) -> bool

A mapping never searched for identifiers: whatever it holds can be a credential under any name.

nemo_gym.rollouts.provenance._sampling(
config: collections.abc.Mapping
) -> dict[str, pydantic.JsonValue]
nemo_gym.rollouts.provenance.build_provenance(
config: collections.abc.Mapping[str, object],
materialized_fpath: pathlib.Path | None = None,
prompt_config: str | None = None

Snapshot model, judge, template and sampling settings. The masked config itself is saved separately.

nemo_gym.rollouts.provenance.served_models(
config: collections.abc.Mapping[str, object]

Ask each model server’s backends what they serve (GET <base_url>/models), once per run.

root is the checkpoint a vLLM backend loaded; hosted APIs report no root, which stays unknown. Nothing is taken from the configured names, so a claimed model can be checked against the served one.

nemo_gym.rollouts.provenance.write_run_provenance(
output_fpath: pathlib.Path,
config: collections.abc.Mapping[str, object],
prompt_config: str | None = None,
input_jsonl_fpath: str | None = None,
model_server_config: collections.abc.Callable[[], collections.abc.Mapping[str, object]] | None = None
) -> pathlib.Path

Write one provenance artifact next to a run’s materialized inputs before dispatch.

model_server_config returns the config that names the run’s model servers when config does not: a --no-serve collection sees only its own config, and the model servers’ entries come from the head server.

nemo_gym.rollouts.provenance.INPUT_SAMPLING_ROW_LIMIT = 100
nemo_gym.rollouts.provenance.SERVED_MODEL_BACKEND_LIMIT = 8
nemo_gym.rollouts.provenance.SERVED_MODEL_PROBE_WORKERS = 32
nemo_gym.rollouts.provenance.SERVED_MODEL_TIMEOUT_S = 5.0
nemo_gym.rollouts.provenance.TOP_LEVEL_PATH = '<top-level>'
nemo_gym.rollouts.provenance._MODEL_KEYS = frozenset({'model', 'openai_model', 'model_name', 'served_model_name', 'checkpoi...
nemo_gym.rollouts.provenance._SAMPLING_KEYS = frozenset({'temperature', 'top_p', 'top_k', 'min_p', 'max_tokens', 'max_output_t...
nemo_gym.rollouts.provenance.logger = logging.getLogger(__name__)